Ten modules, ten pieces. This capstone assembles them into one working system on a new corpus, so you cannot lean on anything you tuned earlier: the support knowledge base of a fictional smart-thermostat company, Thermio, plus a small sales table. Then it runs the whole loop of the course:
build → answer → trace → judge → find what failed → fix the documentation → re-index → re-measure
Everything is offline and deterministic: a stand-in generator and a stand-in supervisor take the place of the language models, exactly as in the earlier modules, so the numbers you see are produced by the code you can read. Swapping in real models changes the quality of the two stand-ins, not the architecture.
You will learn
- How the pieces of modules 01 to 10 fit into one pipeline, with a router that sends a question to passage retrieval, to SQL, or to a refusal
- Choosing the retrieval weight and the refusal threshold on a dev set, then evaluating on a separate battery
- A typed answer that cannot exist without a citation (Pydantic)
- A supervisor that judges every answer, and whether to trust it (we measure its agreement with the references)
- Turning failures into a documentation fix, re-indexing, and proving the improvement with confidence intervals
- A checklist for putting a RAG system in front of users
1. The system we are building
question ──► router ──┬─ passage ─► retrieval (BM25 + embeddings, RRF) ─► generator ─┐
├─ sql ─────► text-to-SQL ─► guard ─► DuckDB ───────────────────────┤
└─ refuse (nothing relevant found) ─────────────────────────────────┤
▼
typed Answer (citations required) ─► trace log (tokens, cost, latency)
│
supervisor judges every answer ─► daily summary
│
retrieval-miss analysis ─► documentation fix ─► re-index ─► re-measure
| Component | Built in module |
|---|---|
| chunking by document structure, BM25, hit@k and MRR | 01 |
| embeddings and hybrid retrieval with Reciprocal Rank Fusion | 02 |
| typed output with Pydantic | 04 |
| a pipeline as explicit steps with a router (LangGraph expresses the same thing as a graph) | 05 |
| answers from data with a read-only SQL guard | 06 |
| question battery, grading, bootstrap confidence intervals | 07 |
| the supervisor, the loop that fixes documentation | 08 |
| traces, tokens, cost | 09 |
| refusal, security habits, cost model | 10 |
Picture: the improvement loop
The whole capstone is this loop, run once. Boxes 1 to 5 build, answer, observe and diagnose; only box 6 changes anything, and what it changes is the documentation. Box 8 measures the result on the same battery, with an interval, so the gain is not taken on faith.
# diagram() is a small SVG helper defined in the downloadable notebook
diagram(820, 345, [
("box", 45, 50, 160, 80, "1 Build\n25 passages, hybrid\nweight chosen on dev", "grey"),
("arrow", 205, 90, 235, 90, ""),
("box", 235, 50, 160, 80, "2 Answer\nrouter: passage, sql\nor refuse; typed", "green"),
("arrow", 395, 90, 425, 90, ""),
("box", 425, 50, 160, 80, "3 Trace\ntokens, cost, latency\nper request", "blue"),
("arrow", 585, 90, 615, 90, ""),
("box", 615, 50, 160, 80, "4 Judge\nsupervisor: do two\nrankers agree?", "violet"),
("arrow", 695, 130, 695, 180, ""),
("box", 615, 180, 160, 80, "5 Find failures\nrank of right passage\nand word gaps", "red"),
("arrow", 615, 220, 585, 220, ""),
("box", 425, 180, 160, 80, "6 Fix the docs\nnot the code, nor\nthe thresholds", "gold"),
("arrow", 425, 220, 395, 220, ""),
("box", 235, 180, 160, 80, "7 Re-index\nsame code, new\nknowledge base", "grey"),
("arrow", 235, 220, 205, 220, ""),
("box", 45, 180, 160, 80, "8 Re-measure\nsame battery,\nbootstrap interval", "violet"),
("arrow", 125, 180, 125, 130, ""),
("text", 135, 160, "next round", 12, "start"),
("note", 45, 280, 730, 50, "Measured here: 18 of 24 correct before (75%), 23 of 24 after, no regressions.\nOn the 20 questions whose reference did not change: gain +0.14, 95% CI [0.00, 0.29]."),
], "The improvement loop: fix the source of truth, then re-measure")
Picture: one real question, end to end
The question is c01 from the battery. Every number in the boxes is printed by a cell below. The SQL and refusal
routes are the two other ways out of the router; the supervisor and the fix are the loop of the previous picture
applied to this one failure.
diagram(840, 470, [
("box", 20, 140, 160, 90, "Question c01\nCan I connect the\nthermostat to a 5 GHz\nnetwork?", "gold"),
("arrow", 180, 185, 210, 185, ""),
("box", 210, 150, 120, 70, "Router\nno sales word,\nso: passage", "blue"),
("arrow", 270, 150, 370, 100, ""),
("box", 370, 40, 190, 70, "SQL path, e.g. s01\none guarded SELECT\non DuckDB: 2465 [1]", "grey"),
("arrow", 330, 185, 370, 185, ""),
("box", 370, 150, 190, 70, "Retrieval\ncosine top 0.35 > 0.25\ntop 3: t07, t01, t02", "blue"),
("arrow", 465, 220, 465, 270, "below 0.25"),
("box", 370, 270, 190, 60, "Refuse path\n'I cannot find that\nin the documentation'", "red"),
("arrow", 560, 185, 600, 185, ""),
("arrow", 560, 75, 660, 150, ""),
("arrow", 560, 300, 600, 215, ""),
("box", 600, 150, 220, 70, "Typed Answer\ncites t07 (Offline):\nrestart the router [1]", "green"),
("arrow", 710, 220, 710, 260, ""),
("box", 600, 260, 220, 60, "Trace\n91 in, 25 out tokens\n$0.0000067, 345 ms", "blue"),
("arrow", 710, 320, 710, 350, ""),
("box", 600, 350, 220, 80, "Supervisor\nretriever picked t07,\nBM25 prefers t01:\nlow confidence", "violet"),
("arrow", 600, 390, 560, 390, ""),
("box", 340, 350, 220, 80, "Documentation fix\nt01 gets: It cannot\nconnect to 5 GHz\nnetworks, only 2.4 GHz", "gold"),
("arrow", 340, 390, 300, 390, ""),
("box", 80, 350, 220, 80, "After re-index\nc01 is now correct\nand nothing else\nbecame wrong", "green"),
], "One real question, end to end")
2. The data: a knowledge base, a table, three sets of questions
- The knowledge base: 25 support passages, each with a section title.
- The sales table: units and revenue by month, product and region, loaded into DuckDB.
- The dev set: 12 questions (8 paraphrases and 4 that use exact terms such as
E45) used only to choose the retrieval weight and the refusal threshold. - The battery: 24 questions with a known answer, used only to evaluate: 16 answerable from passages, 4 answerable from the sales table, and 4 that nothing can answer (including a trap with a plausible error code).
Keeping dev and battery apart is the discipline of module 07: tune on one set, report on the other.
import re, math, json, random, time, itertools
from collections import defaultdict
from collections import Counter
import numpy as np
import duckdb
import matplotlib.pyplot as plt
from typing import Literal
from pydantic import BaseModel, model_validator, ValidationError
from sentence_transformers import SentenceTransformer
KB = [
("t01", "Setup / Wi-Fi", "The Thermio One connects to 2.4 GHz Wi-Fi only. Hold the ring button for 5 seconds until it blinks blue, then pick your network in the app."),
("t02", "Setup / Wiring", "Thermio One works with 24 V heating and cooling systems and needs a C-wire. If your system has no C-wire, install the included power adapter at the furnace."),
("t03", "Setup / Compatibility", "Thermio One is not compatible with 120 V or 240 V line-voltage baseboard heaters. Use Thermio Line for those systems."),
("t04", "Features / Schedules", "Schedules support up to 6 changes per day, with separate weekday and weekend programs. Changes apply within one minute of saving."),
("t05", "Features / Geofencing", "Away mode via geofencing starts when the last phone leaves a 500 m radius, and it sets the temperature back by 4 degrees."),
("t06", "Features / Eco mode", "Eco mode limits heating to 16 degrees and cooling to 28 degrees. The heating limit can be changed in settings between 10 and 18 degrees."),
("t07", "Troubleshooting / Offline", "If the thermostat shows Offline, restart the router, then hold the ring button for 10 seconds to reconnect. Settings are kept."),
("t08", "Troubleshooting / Blank display", "A blank display usually means no power. Check the C-wire and the furnace breaker. The battery backup keeps the display on for about 2 hours."),
("t09", "Troubleshooting / Error E21", "Error E21 means the temperature sensor reading is out of range. Remove the thermostat from its base for 30 seconds and refit it. If E21 returns, contact support for a replacement."),
("t10", "Troubleshooting / Error E45", "Error E45 means the system is short cycling. The thermostat waits 5 minutes between heating starts to protect the compressor."),
("t11", "Troubleshooting / Wrong temperature", "If the temperature reading seems off, avoid direct sun and drafts. A calibration offset of up to 3 degrees can be set in the app."),
("t12", "Warranty / Coverage", "Thermio One has a 3 year limited warranty from the date of purchase. Damage from water or from incorrect wiring is not covered."),
("t13", "Warranty / Returns", "Returns are accepted within 45 days of delivery if the device is undamaged. The refund is issued to the original payment method within 10 business days."),
("t14", "Warranty / Replacement", "A faulty unit under warranty is replaced with a refurbished or new device and shipped within 3 business days of approval."),
("t15", "Account / Password reset", "To reset the password, use the Forgot password link on the sign-in page. The reset email arrives within 5 minutes and the link is valid for 1 hour."),
("t16", "Account / Sharing", "Up to 8 people can share one home in the app. The owner can remove members at any time under Home settings."),
("t17", "Account / Data", "Temperature history is stored for 12 months. Deleting an account removes all history within 30 days."),
("t18", "Energy / Savings report", "The monthly energy report compares runtime with the previous month and with similar homes. Reports need at least 14 days of data."),
("t19", "Energy / Smart utility", "Utility programs can request a temporary setback of up to 2 degrees for at most 4 hours. You can opt out of any event from the app."),
("t20", "Hardware / Battery", "Thermio One uses a built-in rechargeable battery that charges from the C-wire. The battery is not user-replaceable."),
("t21", "Hardware / Sensors", "A remote room sensor (sold separately) measures temperature and occupancy up to 10 m from the thermostat. Up to 6 sensors can be paired."),
("t22", "Hardware / Dimensions", "The thermostat measures 86 mm in diameter and 24 mm in depth and weighs 150 g."),
("t23", "Support / Contact", "Support is available by chat from 08:00 to 20:00 on weekdays. Response time for email is within 24 hours."),
("t24", "Support / Firmware", "Firmware updates install automatically between 02:00 and 04:00 local time. An update takes about 10 minutes and the schedule keeps running."),
("t25", "Support / Privacy", "Thermio does not record audio or video. Location is used only for geofencing and can be switched off in the app."),
]
# sales table: month x product x region (deterministic synthetic numbers)
_r = random.Random(21)
PRICE = {"Thermio One": 129, "Thermio Line": 99, "Thermio Sensor": 39}
sales = []
for m in range(1, 7):
for p in PRICE:
for reg in ("North", "South"):
u = _r.randint(40, 400)
sales.append(dict(month=f"2026-{m:02d}", product=p, region=reg, units=u, revenue_eur=u * PRICE[p]))
con = duckdb.connect()
con.execute("CREATE TABLE sales (month VARCHAR, product VARCHAR, region VARCHAR, units INTEGER, revenue_eur INTEGER)")
con.executemany("INSERT INTO sales VALUES (?, ?, ?, ?, ?)", [tuple(r.values()) for r in sales])
print(len(KB), "passages;", len(sales), "sales rows;", sum(r["units"] for r in sales), "units in total")
print(sales[0])
# references for the SQL questions come from an INDEPENDENT plain-Python calculation, never from the system under test
def tot(key, **where):
return sum(r[key] for r in sales if all(r[k] == v for k, v in where.items()))
exp = dict(
s01=str(tot("units", product="Thermio One")),
s02=max(("North", "South"), key=lambda reg: tot("units", product="Thermio Line", month="2026-03", region=reg)),
s03=str(tot("revenue_eur", month="2026-06")),
s04=str(tot("units", product="Thermio Sensor", region="North")),
)
print("references for the SQL questions:", exp)
# (id, question, gold passage or None, the fact the answer must contain)
BATTERY = [
("c01", "Can I connect the thermostat to a 5 GHz network?", "t01", "2.4 GHz"),
("c02", "My heating system has no common wire, what now?", "t02", "power adapter"),
("c03", "Does it work with electric baseboard heaters?", "t03", "Thermio Line"),
("c04", "How many schedule changes can I set per day?", "t04", "6 changes"),
("c05", "When does the house switch to away mode?", "t05", "500 m"),
("c06", "What is the lowest heating limit in eco mode?", "t06", "16 degrees"),
("c07", "The thermostat says Offline, what should I do?", "t07", "restart the router"),
("c08", "How long does the display stay on when the power is cut?", "t08", "2 hours"),
("c09", "What does error E21 mean?", "t09", "sensor"),
("c10", "Why does the system wait before starting the heating again?", "t10", "5 minutes"),
("c11", "How long is the warranty?", "t12", "3 year"),
("c12", "Within how many days can I return it?", "t13", "45 days"),
("c13", "How do I get a new password?", "t15", "Forgot password"),
("c14", "How many people can share the home?", "t16", "8 people"),
("c15", "How long is the temperature history kept?", "t17", "12 months"),
("c16", "When do firmware updates install?", "t24", "02:00"),
("s01", "How many Thermio One units were sold in total?", "sales", exp["s01"]),
("s02", "Which region sold the most Thermio Line units in March?", "sales", exp["s02"]),
("s03", "What was the total revenue in June?", "sales", exp["s03"]),
("s04", "How many Thermio Sensor units were sold in the North region?", "sales", exp["s04"]),
("u01", "Does the thermostat work with voice assistants?", None, None),
("u02", "What is the price of a Thermio One?", None, None),
("u03", "Who founded the company?", None, None),
("u04", "How do I fix error E99?", None, None),
]
# dev set: used ONLY to choose the retrieval weight and the refusal threshold
DEV = [("d1", "Which wifi bands are supported?", "t01"), ("d2", "How cold can eco mode keep the house in winter?", "t06"),
("d3", "What happens if my account is deleted?", "t17"), ("d4", "Can I get my money back?", "t13"),
("d5", "How do sensors in other rooms work?", "t21"), ("d6", "Is my voice recorded?", "t25"),
("d7", "What if the screen stays dark?", "t08"), ("d8", "How big is the device?", "t22"),
("d9", "What does E45 mean?", "t10"), ("d10", "Do I need a C-wire?", "t02"),
("d11", "Is error E21 serious?", "t09"), ("d12", "firmware update time", "t24")]
OFF_TOPIC = ["What is the capital of France?", "How do I bake sourdough bread?", "Who won the football match yesterday?", "What is the speed of light?"]
print(len(BATTERY), "battery questions,", len(DEV), "dev questions")
3. Retrieval: chunk by structure, then choose the fusion weight by measurement
The knowledge base already has structure: one passage per topic, each with a section title, so the chunk is the passage and the title is part of the searchable text (module 01). Retrieval can combine BM25 and embeddings with RRF (module 02). In module 02 equal weights were not the best; here we measure the BM25 weight on the dev set instead of assuming, and accept whatever it says.
STOP = set("a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you my me".split())
tokens = lambda text: [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
class Index:
# chunks + BM25 + embeddings over a list of (id, section, text) passages
def __init__(self, passages):
self.ids = [p[0] for p in passages]
self.texts = [f"{p[1]}. {p[2]}" for p in passages]
self.tf = [defaultdict(int) for _ in self.texts]
for i, t in enumerate(self.texts):
for w in tokens(t):
self.tf[i][w] += 1
self.len = np.array([sum(d.values()) for d in self.tf], float)
df = defaultdict(int)
for d in self.tf:
for w in d:
df[w] += 1
n = len(self.texts)
self.idf = {w: math.log(1 + (n - c + 0.5) / (c + 0.5)) for w, c in df.items()}
self.emb = embedder.encode(self.texts, normalize_embeddings=True)
def bm25(self, q):
s = np.zeros(len(self.texts))
for w in set(tokens(q)):
if w in self.idf:
for i, d in enumerate(self.tf):
f = d.get(w, 0)
if f:
s[i] += self.idf[w] * f * 2.5 / (f + 1.5 * (0.25 + 0.75 * self.len[i] / self.len.mean()))
return s
def cosine(self, q):
return self.emb @ embedder.encode(q, normalize_embeddings=True)
def fused(self, q, w_bm25):
tot = np.zeros(len(self.texts))
for sc, w in ((self.bm25(q), w_bm25), (self.cosine(q), 1.0)):
for rank, i in enumerate(np.argsort(-sc), 1):
tot[i] += w / (60 + rank)
return tot
index = Index(KB)
def mrr(idx, w):
r = [list(np.argsort(-idx.fused(q, w))).index(idx.ids.index(g)) + 1 for _, q, g in DEV]
return float(np.mean([1 / x for x in r])), float(np.mean([x == 1 for x in r]))
print("BM25 weight (semantic weight = 1) vs the DEV set:")
dev_scores = {}
for w in (0, 0.25, 0.5, 1.0):
dev_scores[w] = mrr(index, w)
print(f" weight {w:<5} MRR {dev_scores[w][0]:.3f} hit@1 {dev_scores[w][1]:.2f}")
W_BM25 = max(dev_scores, key=lambda w: (dev_scores[w][0], -w)) # best MRR; ties go to the smaller (simpler) weight
print("chosen BM25 weight:", W_BM25)
On these dev questions adding BM25 to the fusion did not help: the best weight is whichever the table shows, and with the weight at zero the "hybrid" is semantic-only. That is a legitimate outcome of measuring (in module 02 the identifier questions favoured a small BM25 weight; this knowledge base is small, its passages are distinctive, and most dev questions are paraphrases). It would be wrong to keep a weight just because "hybrid is best practice". BM25 is not wasted, though: it stays in the system as the independent second ranker that the supervisor uses in section 7. Re-run the weight selection whenever the corpus changes.
4. Router and safe SQL
Some questions are answered by a passage, some by a number. A router decides, and it must be conservative:
a question is sent to SQL only if it clearly asks for a quantity in the sales table. The SQL path follows
module 06: the "model" (here a rule-based stand-in) writes one SELECT, a guard validates it, and
DuckDB runs it on a registered view.
MONTHS = {m: f"2026-{i:02d}" for i, m in enumerate(["january", "february", "march", "april", "may", "june"], 1)}
def route(question):
q = question.lower()
return "sql" if re.search(r"\b(sold|sales|revenue)\b", q) else "passage"
def text_to_sql(question):
# stand-in for the model that writes the SELECT: a rule-based translation of the few shapes we support
q = question.lower()
where = []
for p in PRICE:
if p.lower() in q:
where.append(f"product = '{p}'")
for reg in ("north", "south"):
if reg in q:
where.append(f"region = '{reg.title()}'")
for m, key in MONTHS.items():
if m in q:
where.append(f"month = '{key}'")
metric = "revenue_eur" if "revenue" in q else "units"
clause = (" WHERE " + " AND ".join(where)) if where else ""
if re.search(r"which (region|product)", q):
col = "region" if "region" in q else "product"
return f"SELECT {col}, SUM({metric}) AS total FROM sales{clause} GROUP BY {col} ORDER BY total DESC LIMIT 1"
return f"SELECT SUM({metric}) AS total FROM sales{clause}"
FORBIDDEN = re.compile(r"\b(insert|update|delete|drop|alter|create|attach|copy|pragma|install|load|read_\w+)\b", re.I)
def guard(sql):
body = sql.strip().rstrip(";")
if ";" in body or not re.match(r"(?is)^select\b", body):
return False, "only a single SELECT is allowed"
if FORBIDDEN.search(body):
return False, "forbidden keyword"
if set(re.findall(r"(?i)\b(?:from|join)\s+(\w+)", body)) - {"sales"}:
return False, "unknown table"
return True, body
for bad in ["DROP TABLE sales", "SELECT * FROM sales; DELETE FROM sales", "SELECT * FROM read_csv('/etc/passwd')", "SELECT * FROM secrets"]:
print(f"{bad[:44]:46s}-> {guard(bad)}")
print(text_to_sql("Which region sold the most Thermio Line units in March?"))
5. A typed answer that cannot exist without a citation
Every answer leaves the system as a Pydantic object. The type itself refuses an answer that is not a refusal and has no citation, so a bug in the generator cannot silently produce an unsupported claim. The generator is the deterministic stand-in of earlier modules: it returns the sentence of the top passage closest to the question. The refusal rule is calibrated on the dev set and the off-topic questions, never on the battery.
class Citation(BaseModel):
n: int
source: str # a passage id, or "sales" for a database result
class Answer(BaseModel):
question: str
route: Literal["passage", "sql", "refuse"]
text: str
citations: list[Citation] = []
@model_validator(mode="after")
def needs_citation(self):
if self.route != "refuse" and not self.citations:
raise ValueError("an answer that is not a refusal must cite a source")
return self
# ----- refusal threshold, from the dev set (answerable) and off-topic questions (not answerable) -----
dev_top = [index.cosine(q).max() for _, q, _ in DEV]
off_top = [index.cosine(q).max() for q in OFF_TOPIC]
T_COS = (max(off_top) + min(dev_top)) / 2 if max(off_top) < min(dev_top) else max(off_top) + 0.01
print(f"top cosine: dev questions {min(dev_top):.2f} to {max(dev_top):.2f}; off-topic {min(off_top):.2f} to {max(off_top):.2f} -> refuse below {T_COS:.2f}")
REFUSAL = "I cannot find that in the Thermio documentation."
def best_sentence(question, text):
sents = [s for s in re.split(r"(?<=[.!?])\s+", text) if s]
sims = embedder.encode(sents, normalize_embeddings=True) @ embedder.encode(question, normalize_embeddings=True)
return sents[int(np.argmax(sims))]
def answer_question(question, idx, w=None):
# returns (Answer, details); details feed the trace and the supervisor
w = W_BM25 if w is None else w
if route(question) == "sql":
ok, sql = guard(text_to_sql(question))
if ok:
row = con.execute(sql).fetchall()[0]
value = row[0] if len(row) == 1 else row[0]
return Answer(question=question, route="sql", text=f"{value} [1]", citations=[Citation(n=1, source="sales")]), dict(sql=sql, pieces=[], top_cos=1.0)
return Answer(question=question, route="refuse", text=REFUSAL), dict(sql=sql, pieces=[], top_cos=0.0)
cos = idx.cosine(question)
order = list(np.argsort(-idx.fused(question, w)))[:3]
if cos.max() < T_COS:
return Answer(question=question, route="refuse", text=REFUSAL), dict(pieces=[idx.ids[i] for i in order], top_cos=float(cos.max()))
top = order[0]
sentence = best_sentence(question, idx.texts[top].split(". ", 1)[1])
try:
ans = Answer(question=question, route="passage", text=f"{sentence} [1]", citations=[Citation(n=1, source=idx.ids[top])])
except ValidationError:
ans = Answer(question=question, route="refuse", text=REFUSAL)
return ans, dict(pieces=[idx.ids[i] for i in order], top_cos=float(cos.max()))
for q in ["Can I connect the thermostat to a 5 GHz network?", "Which region sold the most Thermio Line units in March?", "What is the capital of Peru?"]:
a, _ = answer_question(q, index)
print(f"{q}\n [{a.route}] {a.text} cites {[c.source for c in a.citations]}")
6. Run the battery, with a trace for every request
Now the system answers the 24 battery questions. Each request writes a trace (module 09): route, the passages it saw, token counts, cost (with assumed prices) and a simulated latency. The grade against the references is computed separately and is what we are trying to improve.
PRICE_IN, PRICE_OUT = 0.035, 0.14 # ASSUMED USD per million tokens for the generation model
TRACE_LOG = []
def grade(ans, gold, fact):
# correct = contains the reference fact; for an unanswerable question (gold None) correct = a refusal
if gold is None:
return int(ans.route == "refuse")
return int(ans.route != "refuse" and fact.lower() in ans.text.lower())
def run_battery(idx, label, battery=None):
r = random.Random(5)
rows = []
for qid, q, gold, fact in (battery or BATTERY):
ans, det = answer_question(q, idx)
n_in = round((len(q.split()) + 60 * (ans.route == "passage")) * 1.3) or 1
n_out = round(len(ans.text.split()) * 1.3)
cost = (n_in * PRICE_IN + n_out * PRICE_OUT) / 1e6
latency = (6 if ans.route != "sql" else 12) + (120 + 8 * n_out + r.random() * 30 if ans.route == "passage" else 0)
TRACE_LOG.append(dict(run=label, id=qid, route=ans.route, tokens_in=n_in, tokens_out=n_out, cost=cost, latency=latency))
rows.append(dict(id=qid, question=q, gold=gold, fact=fact, answer=ans, details=det, correct=grade(ans, gold, fact)))
return rows
before = run_battery(index, "before")
c = np.array([r["correct"] for r in before])
groups = {"passage questions (16)": [r for r in before if r["id"].startswith("c")], "SQL questions (4)": [r for r in before if r["id"].startswith("s")],
"unanswerable (4)": [r for r in before if r["id"].startswith("u")]}
print(f"correct overall: {c.sum()} of {len(c)} ({c.mean():.0%})")
for name, g in groups.items():
print(f" {name:26s}{sum(x['correct'] for x in g)} of {len(g)}")
print("\nwrong answers:")
for r in before:
if not r["correct"]:
print(f" {r['id']} {r['question']}\n -> [{r['answer'].route}] {r['answer'].text} (reference {r['fact']!r})")
7. The supervisor
In production the supervisor is a second model that reads the question, the passages and the answer and returns a verdict (module 08). Here it is a cheap stand-in, and it has the one property the real one also has: it does not know the reference answer. It uses a classic uncertainty signal that needs no model call: do two independent rankers agree? The answer was built from the passage the retriever liked best. A pure BM25 ranker, which works differently from the retriever, is asked which passage it would cite. If the two disagree, or BM25 finds nothing at all, the answer is flagged low confidence and a person (or a stronger model) should look.
- a refusal is justified if nothing in the index is close to the question, and a missed answer if something is;
- a database answer is supported (the guard and the query are the evidence);
- a passage answer is supported if the rankers agree and low confidence otherwise;
- for each flag it names the two candidate passages, which is where the fix belongs.
def supervise(row, idx):
ans, q = row["answer"], row["question"]
near = idx.cosine(q)
j = int(np.argmax(near))
name = lambda k: f"{idx.ids[k]} ({idx.texts[k].split('. ')[0]})"
if ans.route == "refuse":
if near[j] >= T_COS + 0.1: # a passage clearly related exists: the refusal may be a miss
return dict(verdict="missed", missing=name(j))
return dict(verdict="justified refusal", missing="")
if ans.route == "sql":
return dict(verdict="supported", missing="")
b = idx.bm25(q)
cited = idx.ids.index(ans.citations[0].source)
if b.max() > 0 and int(np.argmax(b)) == cited:
return dict(verdict="supported", missing="")
return dict(verdict="low confidence", missing=f"retriever chose {name(cited)}; BM25 prefers " + (name(int(np.argmax(b))) if b.max() > 0 else "nothing (no word matches)"))
for r in before:
r["judge"] = supervise(r, index)
verdicts = Counter(r["judge"]["verdict"] for r in before)
print(", ".join(f"{k}: {v}" for k, v in verdicts.most_common()), "\n")
# how does it behave on the DEV questions, whose answers we can check against the dev references?
dev_rows = [dict(question=q, answer=answer_question(q, index)[0]) for _, q, _ in DEV]
dev_flag = [supervise(r, index)["verdict"] == "low confidence" for r in dev_rows]
dev_wrong = [(r["answer"].citations[0].source if r["answer"].citations else None) != g for r, (_, _, g) in zip(dev_rows, DEV)]
caught = sum(f and w for f, w in zip(dev_flag, dev_wrong))
print(f"on the {len(DEV)} dev questions it flags {sum(dev_flag)}; {sum(dev_wrong)} answers were actually wrong; {caught} of the flags were real errors, {sum(dev_wrong) - caught} wrong answers were not flagged")
# does the supervisor agree with the references on the battery? (it never saw them)
flag_bad = np.array([r["judge"]["verdict"] in ("missed", "low confidence") for r in before])
truth_bad = np.array([not r["correct"] for r in before])
print(f"\n{'':28s}{'not flagged':>14s}{'flagged':>10s}")
for label, t in (("right (against reference)", False), ("wrong (against reference)", True)):
print(f"{label:28s}{int(((truth_bad == t) & ~flag_bad).sum()):>14d}{int(((truth_bad == t) & flag_bad).sum()):>10d}")
print(f"\nflagged {int((flag_bad & truth_bad).sum())} of {truth_bad.sum()} wrong answers (recall), {int((flag_bad & truth_bad).sum())} of {flag_bad.sum()} flags were real errors (precision)")
Read the cross-table honestly. On the battery the triage flagged the wrong answers where the two rankers disagree, and every flag was a real error. It missed the wrong answers where both rankers agreed on the right passage and our generator still picked an incomplete or wrong sentence from it: the judge looks at which passage, not at whether the sentence answers the question. On the paraphrased and short dev questions it over-flags, because BM25 is weak when users do not use the document's words (see the dev line above). Those are the two faces of a cheap judge: it misses some real errors and raises some false alarms. A real LLM judge, reading the passages and the answer, closes much of that gap, at a cost per judged answer that module 10's budget can price. Use this table to decide how much to trust the nightly summary, not as proof that the system is fine.
8. The daily summary
The part of the supervisor that scales is the summary: not 24 verdicts to read, but a page that says what changed and where to look.
tl = [t for t in TRACE_LOG if t["run"] == "before"]
costs, lats = np.array([t["cost"] for t in tl]), np.array([t["latency"] for t in tl])
print(f"requests {len(tl)} cost ${costs.sum():.6f} total, ${costs.mean():.7f} per answer latency p50 {np.median(lats):.0f} ms, p95 {np.percentile(lats, 95):.0f} ms")
print("routes: " + ", ".join(f"{k} {v}" for k, v in Counter(t["route"] for t in tl).most_common()))
print("verdicts: " + ", ".join(f"{k} {v}" for k, v in verdicts.most_common()))
print("\nneeds attention (supervisor view, no references used):")
for r in before:
if r["judge"]["verdict"] in ("missed", "unsupported"):
print(f" {r['id']} [{r['judge']['verdict']}] {r['question']}\n closest passage in the index: {r['judge']['missing']}")
9. Retrieval-miss analysis and a documentation fix
Now the failures that the references exposed. For each wrong passage question we ask the one question that decides the remedy: where was the passage that holds the answer in the ranking, and what words did the question use that the passage does not? Rank 1 with the wrong sentence is a generation problem; a rank beyond the top 3 is a retrieval miss, and the vocabulary gap says how to close it in the documentation.
def analyse(rows, idx):
for r in rows:
if r["correct"] or r["gold"] in (None, "sales"):
continue
order = [idx.ids[i] for i in np.argsort(-idx.fused(r["question"], W_BM25))]
rank = order.index(r["gold"]) + 1
gap = sorted(set(tokens(r["question"])) - set(tokens(idx.texts[idx.ids.index(r["gold"])])))
print(f"{r['id']} {r['question']}\n reference passage {r['gold']} ranked {rank} -> {'retrieval miss' if rank > 3 else 'in the top 3 but not used'}; "
f"question words not in the passage: {gap}")
analyse(before, index)
traps = [r for r in before if r["gold"] is None and not r["correct"]]
for r in traps:
print(f"\n{r['id']} {r['question']}\n answered from {r['answer'].citations[0].source if r['answer'].citations else '-'}: {r['answer'].text} (nothing in the docs answers this)")
# THE FIX, written by reading the analysis above. Everything is a change to the DOCUMENTATION, the source of truth:
# 1) answer the question in the words users use (the answer sentence must contain the fact, in one place)
# 2) the questions the docs never covered get a passage that says so, or says where to go
FIXES = {
"t01": " It cannot connect to 5 GHz networks, only to 2.4 GHz networks.",
"t03": " Electric baseboard heaters are supported by Thermio Line, not by Thermio One.",
"t12": " The warranty lasts 3 years.",
}
NEW = [
("t26", "Troubleshooting / Other error codes", "Only the error codes E21 and E45 are documented. For any other error code, contact support and include the code shown on the display."),
("t27", "Features / Voice assistants", "Thermio does not currently support voice assistants or third-party smart-home platforms."),
("t28", "Support / Pricing", "The price of a Thermio One is not listed in this documentation. See the store page for the current price."),
]
KB_FIXED = [(i, s, t + FIXES.get(i, "")) for i, s, t in KB] + NEW
for i in FIXES:
print(f"{i}: appended {FIXES[i]!r}")
for n in NEW:
print(f"{n[0]}: new passage '{n[1]}'")
A fix to the documentation is a fix for every future user and for every future model, which is why module 08 routes corrections there instead of patching prompts. Note what we did not do: we did not change the code, the thresholds or the retrieval weight. The only change is the knowledge base. Now re-index and re-measure on the same battery.
One honest wrinkle, stated before the numbers so it cannot be hidden in them: three of the four "unanswerable"
questions (u01, u02, u04) were really documentation gaps, and the fix gives each a passage. For those three the
right behaviour is no longer a refusal, so their references change (they now expect the new passage's fact).
Moving a reference is legitimate when the scope of the documentation changes, and dangerous if it is the only thing
that moved, so we report two results: all 24 questions with the updated references, and the 20 questions whose reference did not change.
UPDATED = {"u01": ("t27", "does not currently support"), "u02": ("t28", "store page"), "u04": ("t26", "contact support")}
BATTERY_AFTER = [(i, q, *UPDATED[i]) if i in UPDATED else (i, q, g, f) for i, q, g, f in BATTERY]
index2 = Index(KB_FIXED)
after = run_battery(index2, "after", BATTERY_AFTER)
b_same = np.array([r["correct"] for r in before]) # old answers, old references
b_new = np.array([grade(r["answer"], g, f) for r, (_, _, g, f) in zip(before, BATTERY_AFTER)]) # old answers, UPDATED references
a_ok = np.array([r["correct"] for r in after])
unchanged = np.array([i not in UPDATED for i, *_ in BATTERY])
rng = np.random.default_rng(0)
def boot(b, a, mask):
b, a = b[mask], a[mask]
ix = rng.integers(0, len(a), (5000, len(a)))
ci = lambda x: f"[{np.percentile(x, 2.5):.2f}, {np.percentile(x, 97.5):.2f}]"
return b.mean(), ci(b[ix].mean(1)), a.mean(), ci(a[ix].mean(1)), a.mean() - b.mean(), ci(a[ix].mean(1) - b[ix].mean(1))
print(f"{'':42s}{'before':>8s}{'95% CI':>16s}{'after':>8s}{'95% CI':>16s}{'change':>9s}{'95% CI':>16s}")
for label, b, mask in (("20 questions, references unchanged", b_same, unchanged), ("24 questions, updated references", b_new, np.ones(24, bool))):
r = boot(b, a_ok, mask)
print(f"{label:42s}{r[0]:>8.2f}{r[1]:>16s}{r[2]:>8.2f}{r[3]:>16s}{r[4]:>+9.2f}{r[5]:>16s}")
fixed = [r["id"] for r, bb, aa in zip(after, b_new, a_ok) if aa and not bb]
broke = [r["id"] for r, bb, aa in zip(after, b_new, a_ok) if bb and not aa]
print("\nnewly correct:", fixed, " newly wrong (regressions):", broke)
for r in after:
if not r["correct"]:
print(f" still wrong: {r['id']} {r['question']} -> {r['answer'].text}")
Two things the numbers say and one they cannot. They say which questions the fix repaired (c01, c03,
c11, u01, u04) and, just as important, that it broke nothing: a regression check is half the value of a
fixed battery. They say how wide the uncertainty is: on the 20 questions whose reference did not change, the
interval of the gain runs from 0.00 to 0.29, so even a clear improvement is only just distinguishable from no change,
and an "after" score of 1.00 gives a degenerate interval, which is a sign the battery is now too easy to tell
better systems apart and needs new, harder questions.
The one remaining failure, u02, is a strict-grader artefact: the answer says the price is not listed
in the documentation, which is right, but the reference expected the words "store page". String references are a
blunt instrument (module 07 grades with a judge as well for this reason), and the honest report keeps it
as a failure rather than loosening the grader after seeing the result.
The numbers cannot say the system is good, only that this change did not make it worse on these questions. Real traffic is wider than any battery, which is what the supervisor's nightly summary (section 8) is for: it turns production into the next battery.
10. Before you put a RAG system in front of users
A checklist, each item pointing at the module where you built it:
- Corpus: every document has an owner and a date; chunks follow the document's structure (01)
- Retrieval: the fusion weight chosen on a dev set that is not your test set, and re-chosen when the corpus changes (02)
- Interfaces: model calls go through one routing layer; every input and output has a typed schema (04)
- Flow: the steps are explicit, with a router and a refusal path; no step can loop forever (05)
- Numbers: quantities come from a read-only, guarded query, never from a model's memory (06)
- Quality: a question battery with references from the data, including traps and unanswerable questions, run on every change (07)
- Supervision: every answer is judged, failures are grouped by cause, and fixes go to the documentation (08)
- Observability: a trace per request with tokens, cost, latency and scores; percentiles on a dashboard (09)
- Budget: cost per answer estimated from your own traces; rate limits and a concurrency cap (10)
- Security: retrieved text treated as untrusted; output validated in code; secrets out of URLs; personal data redacted (10)
- Honesty: the system says "I cannot find that" instead of guessing, and the team knows its refusal rate
- Ownership: someone reads the nightly summary, and fixing the documentation is part of their job
Recap: from module to capstone component
| Module | What you built | In this capstone |
|---|---|---|
| 01 | chunking, BM25, hit@k, MRR, refusal | Index.bm25, section chunks, dev-set MRR |
| 02 | embeddings, RRF, weights | Index.cosine, Index.fused, the weight W_BM25 chosen on the dev set (BM25 became the supervisor's second opinion) |
| 03 | LangChain and LlamaIndex | the same retrieval written by hand: what the frameworks abstract |
| 04 | LiteLLM and Pydantic | the typed Answer and its validator; one place to swap the generator |
| 05 | LangGraph | route and the explicit steps of answer_question |
| 06 | safe text-to-SQL | text_to_sql, guard, DuckDB |
| 07 | evaluation | BATTERY, DEV, grading against independent references, bootstrap CIs |
| 08 | the supervisor | supervise, the daily summary, the documentation fix and re-index |
| 09 | observability | TRACE_LOG with tokens, cost and latency |
| 10 | production and security | refusal threshold, cost model, validation and the habits in the checklist |
Exercises
- Replace
best_sentencewith taking the whole top passage as the answer. Which questions change, and what does that say about the line between retrieval and generation? - Set
W_BM25 = 1.0after section 3 and re-run from section 6. Does the supervisor still flag the failures? Does its second opinion get weaker when BM25 also drives retrieval? - Add a fifth unanswerable trap that shares words with a real passage ("What is the warranty on the Thermio Line?"). Which layer should catch it: retrieval, the generator, the supervisor or the documentation?
- Write a new documentation fix for a failure your supervisor flagged but this notebook did not fix, apply it, and report the before and after with the interval. Did the interval change your conclusion?