Do research that others can trust
Run reproducible experiments, monitor changing data, and assemble a capstone evidence report. This module has 3 lessons and takes about 1h 30m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 20 in the Learning LabMake an experiment reproducible
Record a dataset fingerprint, split, settings, baseline, and held-out result.
A useful research result lets another person see what was tested and repeat the analysis. Record where the data came from, which rows were used, how the model was configured, and which metric was calculated.
This synthetic calibration experiment fixes the random seed, uses a fixed split, and compares a linear model with a training-mean baseline on held-out rows. A SHA-256 fingerprint identifies the exact generated numeric array used in this run.
A seed alone is not a complete reproducibility plan. Also save the code, dependency versions, preprocessing rules, and access conditions for any real data. Numerical results may differ slightly across libraries and platforms, so use sensible tolerances when comparing runs.
See it in action
import hashlib
import json
import platform
import numpy as np
seed = 20
rng = np.random.default_rng(seed)
x = rng.uniform(0, 5, 80)
y = 3 * x + 2 + rng.normal(0, 0.4, len(x))
data = np.column_stack([x, y])
fingerprint = hashlib.sha256(data.tobytes()).hexdigest()
split = 60
X_train = np.column_stack([np.ones(split), x[:split]])
weights = np.linalg.lstsq(X_train, y[:split], rcond=None)[0]
predictions = np.column_stack([np.ones(len(x) - split), x[split:]]) @ weights
result = {
"dataset": "synthetic calibration v1", "data_sha256": fingerprint,
"seed": seed, "train_rows": [0, split - 1], "test_rows": [split, len(x) - 1],
"model": "linear least squares with intercept",
"test_mae": float(np.mean(abs(y[split:] - predictions))),
"baseline_mae": float(np.mean(abs(y[split:] - y[:split].mean()))),
"python": platform.python_version(), "numpy": np.__version__,
}
print(json.dumps(result, indent=2))
Make the experiment traceable from the reported metric back to the exact procedure.
Monitor change after deployment
Separate input-distribution alerts from measured prediction errors.
A model trained in one season may receive different inputs later. Data drift means the input distribution has changed. It is a reason to investigate; it does not by itself prove that prediction quality has become worse.
The example compares recent sensor means with a reference period, using reference standard deviation as a scale. It then measures recent MAE only when actual outcomes are available. These two checks answer different questions.
The alert thresholds are teaching choices. A real monitoring plan needs expected seasonal changes, minimum sample sizes, delayed outcomes, responsible reviewers, and a clear response. Log alerts for review instead of automatically replacing a model whenever one mean moves.
See it in action
import numpy as np
rng = np.random.default_rng(21)
reference = rng.normal(20, 2, 100)
recent = rng.normal(25, 2, 30)
shift = abs(recent.mean() - reference.mean()) / reference.std()
print("Mean shift in reference SD units:", round(float(shift), 2))
print("Review input shift:", bool(shift > 1.5))
# Later, true outcomes arrive for these predictions.
predictions = np.array([40, 42, 44, 46], dtype=float)
actual = np.array([41, 46, 48, 50], dtype=float)
mae = np.mean(abs(actual - predictions))
print("Recent MAE:", float(mae))
print("Review performance:", bool(mae > 3.0))
Monitor data and outcomes separately, with clear rules for human review.
Build a capstone evidence report
Compare a candidate with a baseline and report uncertainty, limitations, and next steps.
Finish with a research question that can be tested: does a candidate reduce absolute prediction error against a baseline on the same held-out equipment readings? Start with a small, reproducible comparison before adding model complexity.
The example calculates each sample's improvement and bootstraps paired samples to form a descriptive 95% interval for mean improvement. Keeping each candidate and baseline error paired preserves the comparison on the same observation. This ordinary bootstrap assumes independent observations; time series need a method that respects dependence.
Your capstone should include a problem statement, data provenance, leakage checks, baseline, reproducible code, held-out metrics, subgroup analysis where relevant, limitations, and a monitoring plan. These synthetic results demonstrate a workflow, not real-world validation. Extend the course through reviewed projects and research practice.
See it in action
import json
import numpy as np
actual = np.array([10, 12, 15, 18, 20, 23, 25, 28], dtype=float)
baseline = np.array([8, 10, 12, 15, 18, 20, 22, 25], dtype=float)
candidate = np.array([9, 13, 14, 17, 21, 22, 24, 27], dtype=float)
baseline_error = abs(actual - baseline)
candidate_error = abs(actual - candidate)
improvement = baseline_error - candidate_error
rng = np.random.default_rng(20)
indices = rng.integers(0, len(actual), size=(2000, len(actual)))
bootstrap_means = improvement[indices].mean(axis=1)
low, high = np.quantile(bootstrap_means, [0.025, 0.975])
report = {
"question": "Does the candidate reduce held-out calibration MAE?",
"dataset": "eight synthetic independent observations",
"baseline_mae": float(baseline_error.mean()),
"candidate_mae": float(candidate_error.mean()),
"mean_improvement": float(improvement.mean()),
"bootstrap_interval_95": [float(low), float(high)],
"limitations": ["Very small synthetic sample", "No field or external validation"],
"next_step": "Test the frozen procedure on independently collected permitted data",
}
print(json.dumps(report, indent=2))
A strong research project makes its evidence, assumptions, and limits easy to inspect.
Free, open and run by a nonprofit.
This page is part of Python to AI, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.