Test a support triage assistant.
Compare keyword rules with simulated model predictions and measure review workload.
Which routing policy gives a useful balance of coverage and wrong automatic routes?
A fictional help desk wants suggestions for repair and training queues. The supplied model labels and scores are invented fixtures, not the result of a real model. Use them to build a repeatable evaluation and review policy.
Build it, step by step
- 01
Write a prompt specification and label definitions before reading the simulated predictions.
- 02
Implement a keyword baseline and compare it with expected labels.
- 03
Validate the simulated labels and numeric scores.
- 04
Compare at least two thresholds using accuracy, automatic errors, and review coverage.
- 05
Add ambiguous and hostile messages. Keep a separate held-out set and state that no real model was evaluated.
Your project checklist
- Unknown or invalid predictions go to review.
- Baseline and candidate use the same labelled examples.
- The report includes counts as well as rates.
- A score is not described as a calibrated probability.
- Fictional predictions and sample limitations are clearly labelled.
Free, open and run by a nonprofit.
This page is part of AI Automation, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.