Learn with trees and ensembles
Model nonlinear patterns, compare forests, and tune a model without using test labels. This module has 3 lessons and takes about 1h 15m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 13 in the Learning LabControl a tree before it memorizes
Compare shallow and deep decision trees on synthetic wear data.
A decision tree splits inputs using questions such as 'Is speed below 1000?' For regression, each final group predicts a number based on its training examples. Trees can capture bends and interactions without assuming one straight line.
A deep tree can memorize noise and achieve tiny training error while making worse new predictions. max_depth limits the number of split levels; min_samples_leaf requires enough examples in each final group. Compare validation errors when choosing these settings.
See it in action
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
# Synthetic independent machines with a nonlinear speed-to-wear relationship.
rng = np.random.default_rng(42)
X = rng.uniform(0, 10, size=(60, 1))
y = (X[:, 0] - 5) ** 2 + rng.normal(0, 2, size=60)
X_train, X_valid, y_train, y_valid = train_test_split(
X, y, test_size=0.3, random_state=42
)
for depth in [2, 4, None]:
model = DecisionTreeRegressor(max_depth=depth, random_state=42)
model.fit(X_train, y_train)
train_mae = mean_absolute_error(y_train, model.predict(X_train))
valid_mae = mean_absolute_error(y_valid, model.predict(X_valid))
print("Depth:", depth, "Train MAE:", round(train_mae, 2),
"Validation MAE:", round(valid_mae, 2))A tree needs limits and validation, even when its training fit looks perfect.
Combine trees and inspect their signals
Train a small random forest and test whether a feature helps on validation data.
A random forest combines many trees built with randomness in examples and candidate features. Averaging their predictions can reduce the instability of one tree. More trees cost more computation, so begin with a modest number.
Permutation importance measures how validation error changes when one feature is shuffled. If error rises, the fitted model relied on that feature in this dataset. This is not a causal claim. Correlated features can hide each other's importance, and a weak model can give misleading importance estimates.
See it in action
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance
from sklearn.metrics import mean_absolute_error
# Synthetic independent sites: temperature, occupancy, unrelated noise.
rng = np.random.default_rng(42)
X = np.column_stack([rng.uniform(20, 35, 80), rng.uniform(0, 100, 80), rng.normal(size=80)])
y = 2 * (X[:, 0] - 20) + 0.4 * X[:, 1] + rng.normal(0, 2, 80)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, random_state=42)
forest = RandomForestRegressor(n_estimators=40, min_samples_leaf=2, random_state=42, n_jobs=1)
forest.fit(X_train, y_train)
print("Validation MAE:", round(mean_absolute_error(y_valid, forest.predict(X_valid)), 2))
importance = permutation_importance(
forest, X_valid, y_valid, scoring="neg_mean_absolute_error", n_repeats=5, random_state=42
)
for name, value in zip(["temperature", "occupancy", "noise"], importance.importances_mean):
print(name, "MAE increase after shuffling:", round(value, 2))Combine trees, then check what helps on unseen data without claiming causation.
Tune, test, and write a model card
Select tree settings with cross-validation and evaluate the selected model on untouched test data.
Hyperparameters are settings chosen before fitting, such as tree depth. GridSearchCV tries a small set of combinations, scores each with cross-validation, and refits the best combination on the provided training data. It should never receive the final test labels.
After selection, measure test performance once and record the intended use, dataset origin, metric, and limitations. A strong score on invented independent machines does not establish performance on real equipment. The short model card is your starting point for a reproducible research report.
See it in action
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split, GridSearchCV, KFold
from sklearn.metrics import mean_absolute_error
# Synthetic independent machine observations.
rng = np.random.default_rng(42)
X = rng.uniform(0, 10, size=(60, 2))
y = X[:, 0] ** 2 + 2 * X[:, 1] + rng.normal(0, 3, 60)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
cv = KFold(n_splits=3, shuffle=True, random_state=42)
search = GridSearchCV(
DecisionTreeRegressor(random_state=42),
{"max_depth": [2, 4, 6], "min_samples_leaf": [1, 3]},
cv=cv, scoring="neg_mean_absolute_error", n_jobs=1
)
search.fit(X_train, y_train)
test_mae = mean_absolute_error(y_test, search.predict(X_test))
print("Chosen settings:", search.best_params_)
print("Test MAE:", round(test_mae, 3))
print("Model card: synthetic wear prediction; independent random split.")
print("Limitation: not validated on real machines or future operating conditions.")Select settings with validation, test once, and document what the result supports.
Free, open and run by a nonprofit.
This page is part of Python to AI, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.