Evaluate the whole workflow
Measure errors, review load, and regressions with fixed examples. This module has 3 lessons and takes about 1h 50m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 16 in the Learning LabCalculate exact-match accuracy
Return exact-match accuracy rounded to two decimals, or None for an empty expected list or mismatched lengths.
An evaluation set pairs inputs with expected outcomes. Keep a held-out set separate from examples used to improve the prompt. Otherwise the result may overstate generalisation.
Exact-match accuracy divides matching predictions by the number of expected labels. Reject unequal list lengths. An empty set has no accuracy result, represented here by None.
Accuracy alone can hide costly mistakes and poor performance on rare cases. Inspect errors by category and workflow stage.
See it in action
def solve(expected, predicted):
if not expected or len(expected) != len(predicted):
return None
return round(sum(a == b for a, b in zip(expected, predicted)) / len(expected), 2)
print(solve(["repair","training","other"], ["repair","other","other"]))
Accuracy alone can hide costly mistakes and poor performance on rare cases. Inspect errors by category and workflow stage.
Measure false automatic routes
Return automatic, wrong, and review counts for rows with expected and route fields. Only repair and training are automatic routes.
An automatic mistake can be more costly than a review request. Count wrong automatic predictions separately from cases sent to review.
Each row has expected and route fields. Only repair and training routes count as automatic. A mismatch with expected counts as a wrong automatic route.
Review is not necessarily failure. It is a planned way to contain uncertainty, with a measurable time cost.
See it in action
def solve(rows):
automatic = [row for row in rows if row["route"] in ("repair", "training")]
return {"automatic":len(automatic), "wrong":sum(row["route"] != row["expected"] for row in automatic), "review":len(rows)-len(automatic)}
print(solve([{"expected":"repair","route":"repair"},{"expected":"other","route":"training"},{"expected":"training","route":"review"}]))
Review is not necessarily failure. It is a planned way to contain uncertainty, with a measurable time cost.
Practical: Compare a candidate with a baseline
For valid accuracy fractions and a non-negative case count, return pilot only with at least one case, candidate accuracy at least 0.9, and no drop from baseline.
A regression check compares the current workflow with a proposed change on the same examples. Keep the source data and scoring rule fixed.
The fictional release policy requires at least 90 percent accuracy and no drop from the baseline. This is a teaching threshold; actual release criteria depend on the cost of each failure.
Passing a small test set is evidence for a pilot, not proof of production reliability. Add realistic cases and operational monitoring.
Your experiment plan
Use the browser editor or the notebook. All records and AI responses are fictional. No network calls, credentials, or real messages are used.
- Create a labelled evaluation table with normal, ambiguous, and hostile inputs.
- Run the example and predict its result before changing it.
- Complete the function in Practice and run the checks. Add your own normal, empty, and failure cases.
- Record baseline and candidate results on the same held-out cases.
- Save the notebook with your observations and explain one limitation of the simulation.
What to hand in
- An evaluation report with accuracy, wrong routes, coverage, and release decision.
- A test table with inputs, expected outputs, actual outputs, and one revision.
Reflect on your results
What important cases are missing from your test set?
The automatic exercise checks the core code behaviour. Review observations and written deliverables yourself or with a mentor.
See it in action
def solve(baseline, candidate, cases):
if cases <= 0:
return "hold"
return "pilot" if candidate >= 0.9 and candidate >= baseline else "hold"
print(solve(0.9, 0.95, 20))
Passing a small test set is evidence for a pilot, not proof of production reliability. Add realistic cases and operational monitoring.
Free, open and run by a nonprofit.
This page is part of AI Automation, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.