Build grounded AI workflows
Retrieve useful passages, answer with evidence, and evaluate an offline RAG prototype. This module has 3 lessons and takes about 1h 30m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 18 in the Learning LabRetrieve passages from a small library
Rank maintenance passages with a transparent TF-IDF-style retrieval score.
Retrieval-augmented generation, often called RAG, first finds relevant source material and then supplies it to an answer system. Build and test retrieval before adding a generator. If the needed passage is missing, fluent wording cannot repair the evidence gap.
The offline retriever below combines word counts with inverse document frequency. A word found in fewer passages gets more weight. Cosine similarity then ranks the passage vectors against a query vector.
All passages are fictional teaching notes and the system uses exact word overlap. It may miss synonyms or return a weak match for an unrelated query. The next lessons add refusal and evaluation instead of pretending this prototype is a full generative model.
See it in action
import re
import numpy as np
passages = [
{"id": "P1", "text": "Inspect pump vibration during the weekly maintenance review."},
{"id": "P2", "text": "Record solar panel output with the energy meter each morning."},
{"id": "P3", "text": "Log battery temperature and battery voltage before the inspection."},
]
def words(text):
return re.findall(r"[a-z]+", text.lower())
vocabulary = sorted({w for p in passages for w in words(p["text"])})
counts = np.array([[words(p["text"]).count(w) for w in vocabulary] for p in passages])
document_frequency = (counts > 0).sum(axis=0)
idf = np.log((1 + len(passages)) / (1 + document_frequency)) + 1
vectors = counts * idf
query = "battery temperature inspection"
q = np.array([words(query).count(w) for w in vocabulary]) * idf
denominators = np.linalg.norm(vectors, axis=1) * np.linalg.norm(q)
scores = np.divide(vectors @ q, denominators,
out=np.zeros(len(passages)), where=denominators > 0)
for index in np.argsort(-scores):
print(passages[index]["id"], round(float(scores[index]), 3), passages[index]["text"])
Make source retrieval inspectable before building an answer layer.
Answer with sources or abstain
Build an offline answer function that cites exact evidence and declines unsupported questions.
An answer workflow needs a clear contract: use the supplied evidence, attach its source, and decline when the evidence does not answer the question. Here a simple overlap rule selects a passage, and the answer returns that exact passage.
This is an extractive prototype of the retrieval and evidence controls around RAG. It does not generate new language or call an AI service. Returning an exact passage makes the source relationship easy to inspect, but matching a keyword still does not prove the passage answers the whole question.
Retrieved documents should be treated as data, even if their text contains commands. A production generator needs separate trusted instructions, access controls, and tests for misleading sources. Use a validated threshold instead of assuming the demonstration threshold is universal.
See it in action
import re
sources = [
{"id": "E1", "text": "Solar output readings are recorded at 09:00 in the training log."},
{"id": "E2", "text": "Battery inspection notes are reviewed every Friday in this fictional lab."},
]
stopwords = {"the", "are", "is", "in", "at", "what", "when", "how"}
def tokens(text):
return set(re.findall(r"[a-z0-9]+", text.lower())) - stopwords
def answer(question):
query = tokens(question)
ranked = sorted(sources, key=lambda s: len(query & tokens(s["text"])), reverse=True)
best = ranked[0]
overlap = len(query & tokens(best["text"]))
if overlap < 2:
return "The supplied sources do not support an answer."
return f"{best['text']} [{best['id']}]"
print(answer("When are solar output readings recorded?"))
print(answer("What is the emergency replacement cost?"))
A useful answer states what its sources support and can decline when support is missing.
Evaluate retrieval and grounded answers
Measure retrieval coverage, source support, and correct abstention separately.
Build a tiny test set before changing a retrieval system. For a fictional clinic operations handbook, label which passage IDs answer each question. Also include questions the handbook cannot answer.
Recall at k asks how many relevant source IDs appear among the first k retrieved results. The example checks answer support by testing whether an extractive answer appears inside its cited source. It separately checks whether unsupported questions receive no answer.
Exact substring checks only suit this extractive prototype. Generated answers need claim-level review for meaning, unsupported details, and whether they answer the question. Keep retrieval quality, answer correctness, and refusal behaviour separate so one score does not hide another failure.
See it in action
sources = {
"H1": "The fictional clinic opens at 08:00.",
"H2": "Appointment counts are recorded daily.",
"H3": "The supply log is reviewed on Monday.",
}
cases = [
{"relevant": {"H1"}, "retrieved": ["H1", "H3"],
"answer": "The fictional clinic opens at 08:00.", "citation": "H1"},
{"relevant": {"H2"}, "retrieved": ["H3", "H2"],
"answer": "Appointment counts are recorded daily.", "citation": "H2"},
{"relevant": set(), "retrieved": ["H3"], "answer": "", "citation": ""},
]
recalls, supported, abstentions = [], [], []
for case in cases:
if case["relevant"]:
recalls.append(len(case["relevant"] & set(case["retrieved"][:2])) / len(case["relevant"]))
else:
abstentions.append(case["answer"] == "")
if case["answer"]:
text = sources.get(case["citation"], "")
supported.append(case["answer"] in text)
print("Recall at 2:", sum(recalls) / len(recalls))
print("Extractive source support:", sum(supported) / len(supported))
print("Unsupported-question abstention:", sum(abstentions) / len(abstentions))
Use separate tests for finding evidence, using evidence, and recognising missing evidence.
Free, open and run by a nonprofit.
This page is part of Python to AI, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.