Validate AI output
Treat generated labels and JSON as proposals that must pass checks. This module has 3 lessons and takes about 1h 50m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 07 in the Learning LabAccept only known labels
Return a normalised allowed label, or review for any other value.
A model may return a label outside your allowed list. It may also return explanations where you expected a short category. Treat the result as a proposal.
Trim and lowercase a string, then compare it with repair, training, and other. Any other value returns review. Do not guess a close label.
A valid label can still be wrong for the message. Schema validation and accuracy evaluation are separate tasks.
See it in action
def solve(value):
if not isinstance(value, str):
return "review"
label = value.strip().lower()
return label if label in {"repair", "training", "other"} else "review"
print(solve(" REPAIR "))
A valid label can still be wrong for the message. Schema validation and accuracy evaluation are separate tasks.
Check a structured prediction
Return True only for exactly label and score keys, an allowed label, and an int or float score between 0 and 1. Reject booleans.
A structured prediction contains a label and a score. A score reported by a model is not automatically a calibrated probability. This exercise checks its shape only.
Require exactly label and score keys, a known label, and a numeric score from zero to one. Python bool is a subclass of int, so use an exact type check to reject True as a score.
Use held-out examples to decide how a score threshold behaves. Never assume a score of 0.9 means 90 percent accuracy.
See it in action
def solve(value):
if not isinstance(value, dict) or set(value) != {"label", "score"}:
return False
return value["label"] in ("repair", "training", "other") and type(value["score"]) in (int, float) and 0 <= value["score"] <= 1
print(solve({"label":"repair","score":0.8}))
Use held-out examples to decide how a score threshold behaves. Never assume a score of 0.9 means 90 percent accuracy.
Practical: Parse, validate, then route
Parse JSON containing exactly one label field. Return repair or training only for those exact labels, and review for all other cases.
Connect the steps in order: parse the response, validate the fields, and choose a route. A malformed response should take a visible review path.
This lesson accepts exactly one label field in a JSON object. Valid repair and training labels go to those queues. The valid other label and every invalid response go to review.
Keep parsing failures separate in operational logs, even when several failures share the same user-facing review route.
Your experiment plan
Use the browser editor or the notebook. All records and AI responses are fictional. No network calls, credentials, or real messages are used.
- Prepare valid, malformed, extra-field, and unknown-label response fixtures.
- Run the example and predict its result before changing it.
- Complete the function in Practice and run the checks. Add your own normal, empty, and failure cases.
- Replace one valid label with a list and confirm the route stays review.
- Save the notebook with your observations and explain one limitation of the simulation.
What to hand in
- A response adapter and a failure classification table.
- A test table with inputs, expected outputs, actual outputs, and one revision.
Reflect on your results
Which failures should trigger a prompt change, and which need code changes?
The automatic exercise checks the core code behaviour. Review observations and written deliverables yourself or with a mentor.
See it in action
def solve(raw):
import json
try:
value = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return "review"
if not isinstance(value, dict) or set(value) != {"label"}:
return "review"
label = value["label"]
return label if label in ("repair", "training") else "review"
print(solve('{"label":"repair"}'))
Keep parsing failures separate in operational logs, even when several failures share the same user-facing review route.
Free, open and run by a nonprofit.
This page is part of AI Automation, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.