RAG Systems 1001

Building · Notebook 05 of 12

Agents with LangGraph

A RAG pipeline as a state graph: nodes, conditional routing, parallel branches, tools and checkpoints.

  • LangGraph
  • state graph
  • routing
  • tools
  • checkpoints
⏱ ~90 min 💻 runs in 20 s on a CPU ⬇ Download notebook Open in Colab View on GitHub

So far the RAG pipeline was a straight line: retrieve, generate. Real assistants branch. A question about a number should go to a database, not to the documents. An off-topic question should be refused before it costs a model call. An answer that fails a check should send the system back to try again, once, with a better query. Two retrievers should run at the same time. And a follow-up such as "and who decides that?" only makes sense if the system remembers the previous turn.

A state graph is the clean way to express this. LangGraph models an application as nodes (plain functions that read and update a shared state) connected by edges, some of them conditional. The flow becomes data you can draw, test, pause and resume, instead of if statements buried in a function. A wind-farm analytics agent in this portfolio is exactly such a graph: ingest, forecast, diagnose, then document retrieval and image analysis in parallel, investigate, then either recommend or recommend a retraining (a conditional branch), and finally explain.

Everything here is deterministic and offline: the "model" steps are local stand-ins, so what you see is the graph machinery, which is what carries over to a real model.

You will learn

  1. State, nodes and edges: the three ideas in LangGraph
  2. Conditional routing: send a question to the documents, to a database stub, or to a refusal
  3. Parallel branches: two retrievers at once, joined before the next step, and what that buys in time
  4. A check-and-retry loop: verify the answer, rewrite the query, try once more, then give up honestly
  5. Checkpointing: conversation memory per thread, so a follow-up question works
  6. Measuring what the loop adds, and what it does not

Picture: a graph is state, nodes and edges

Read it from the left. The state is one shared dictionary. A node is a function that reads it and returns the keys it changed. Edges decide what runs next: a plain edge, a fan-out that runs two nodes at the same step, a join that waits for both, and a conditional edge that looks at the state. A checkpointer saves the state after every step.

# diagram() is a small SVG helper defined in the downloadable notebook
diagram(840, 340, [
    ("box", 20, 100, 170, 120, "State (shared)\nquestion, query\nsemantic, lexical\nfused, answer\nhistory, trace", "grey"),
    ("arrow", 190, 160, 240, 160, "use"),
    ("box", 240, 130, 110, 60, "Node A", "blue"),
    ("arrow", 350, 145, 410, 85, "fan-out"),
    ("arrow", 350, 175, 410, 235, "fan-out"),
    ("box", 410, 55, 110, 60, "Node B", "blue"),
    ("box", 410, 205, 110, 60, "Node C", "blue"),
    ("arrow", 520, 85, 580, 145, "join"),
    ("arrow", 520, 235, 580, 175, "join"),
    ("box", 580, 130, 110, 60, "Node D", "blue"),
    ("arrow", 690, 145, 730, 90, "if ok"),
    ("arrow", 690, 175, 730, 230, "if not"),
    ("box", 730, 60, 90, 55, "end", "green"),
    ("box", 730, 200, 90, 60, "retry or\nrefuse", "red"),
    ("note", 20, 270, 800, 50, "Checkpointer: saves the state after every step under a thread_id, so a follow-up question finds the history."),
], "LangGraph in one picture")
notebook output figure

1. The ingredients

We reuse the Harbourview handbook and the retrievers of modules 01 and 02: BM25 (lexical) and MiniLM embeddings (semantic). Each will become a graph node.

# The Harbourview handbook: 22 passages, fictional, written for this course
PASSAGES = [
    ["p01", "Safety / Access", "Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer vessel. Certificates are checked at the harbour gate and expire after two years."],
    ["p02", "Safety / Weather limits", "Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later."],
    ["p03", "Safety / Lone working", "Lone working inside a nacelle is forbidden. Every task inside a turbine needs at least two people, and both must carry a personal locator beacon."],
    ["p04", "Turbines / Specification", "Each of the 40 turbines is rated at 8.0 MW with a rotor diameter of 164 m. Cut-in wind speed is 3 m/s, rated speed is 12.5 m/s and cut-out speed is 25 m/s."],
    ["p05", "Turbines / Naming", "Turbines are named HV01 to HV40, numbered from the western edge of the array. The offshore substation is OSS1, in the centre."],
    ["p06", "Maintenance / Schedule", "Every turbine gets a planned service once a year between April and September. The service takes two technicians about two days, plus weather delays."],
    ["p07", "Maintenance / Gearbox oil", "Gearbox oil is sampled every six months. If the particle count exceeds ISO 4406 code 18/16/13, a filter change is scheduled within 14 days."],
    ["p08", "Maintenance / Blade inspection", "Blades are inspected by drone every 12 months and after any lightning alarm. Leading-edge erosion above grade 3 is repaired in the next planned service."],
    ["p09", "Faults / Alarm levels", "Alarms have three levels. Level 1 is informational. Level 2 needs a remote check within 4 hours. Level 3 stops the turbine and needs a site visit within 24 hours when the weather allows."],
    ["p10", "Faults / Pitch system", "A pitch system fault (code PF-203) usually means a low accumulator pressure. Reset it remotely once. If it returns within 48 hours, send a crew to replace the hydraulic accumulator."],
    ["p11", "Faults / Yaw misalignment", "A yaw error above 8 degrees for more than 10 minutes raises alarm YW-110. Persistent errors point to a faulty wind vane: the vane is recalibrated, then replaced if the error remains."],
    ["p12", "Faults / Grid loss", "On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid operator confirms voltage within 5% of nominal for 10 minutes."],
    ["p13", "Environment / Marine mammals", "Piling and heavy cable work pause when a marine mammal is sighted within 500 m. Work resumes after 30 minutes without a sighting."],
    ["p14", "Environment / Bird collision", "Between October and March, turbines HV01 to HV08 are curtailed at night during the peak migration periods announced by the ornithology team."],
    ["p15", "Environment / Waste", "Used oil, filters and composite offcuts are brought ashore and logged in the waste register. Nothing is discharged at sea."],
    ["p16", "Logistics / Spare parts", "The harbour warehouse keeps the critical spares: two pitch accumulators, four wind vanes and a spare converter module. Reorder when stock falls below one unit."],
    ["p17", "Logistics / Helicopter", "A helicopter is used only for medical evacuation and for urgent Level 3 visits when vessels cannot sail. Booking goes through the duty coordinator."],
    ["p18", "Contacts / Duty coordinator", "The duty coordinator is reachable 24 hours a day on the control room line. Out-of-hours safety incidents are reported there first."],
    ["p19", "Reporting / Incidents", "Any injury or near miss is reported within one hour and a written report is filed within 24 hours. Environmental incidents follow the same deadlines."],
    ["p20", "Reporting / Performance", "Availability is reported monthly. The target is 97% time-based availability, counting weather downtime as available."],
    ["p21", "Grid / Export", "Power is exported at 66 kV through two cables to the onshore substation. The export limit agreed with the grid operator is 300 MW."],
    ["p22", "Training / Induction", "New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer."],
]

# (id, question, gold passage, fact the answer must contain); None = the handbook does not say
QUESTIONS = [
    ["q01", "What wave height stops the crew boats from sailing?", "p02", "1.5 m"],
    ["q02", "How often is the gearbox oil sampled?", "p07", "six months"],
    ["q03", "Who may work alone inside a nacelle?", "p03", "forbidden"],
    ["q04", "What is the rotor diameter of a turbine?", "p04", "164 m"],
    ["q05", "What do I do if the pitch fault comes back after a reset?", "p10", "replace the hydraulic accumulator"],
    ["q06", "When must a Level 3 alarm be attended on site?", "p09", "24 hours"],
    ["q07", "How soon must an injury be reported?", "p19", "one hour"],
    ["q08", "What is the largest amount of power the farm is allowed to send ashore?", "p21", "300 MW"],
    ["q09", "Which turbines are switched off at night in the autumn to protect birds?", "p14", "HV01 to HV08"],
    ["q10", "How many spare wind vanes are kept in stock?", "p16", "four wind vanes"],
    ["q11", "What qualification do I need before getting on the boat?", "p01", "GWO Basic Safety Training"],
    ["q12", "When can the turbines restart after a blackout?", "p12", "10 minutes"],
    ["q13", "When can a helicopter be used?", "p17", "medical evacuation"],
    ["q14", "What happens when a dolphin is seen near the piling work?", "p13", "pause"],
    ["q15", "What is the price of electricity per MWh?", None, None],
    ["q16", "Who is the chief executive of the operating company?", None, None],
]
import re, time, math, operator, warnings
from typing import Annotated, TypedDict
import numpy as np
import matplotlib.pyplot as plt
warnings.filterwarnings("ignore")
from sentence_transformers import SentenceTransformer

ids = [p[0] for p in PASSAGES]
chunks = [f"{section}. {text}" for _, section, text in PASSAGES]
section_of = {pid: section for pid, section, _ in PASSAGES}
minilm = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
E = minilm.encode(chunks, normalize_embeddings=True)

STOP = set("a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you".split())
def tokens(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

class BM25:
    def __init__(self, docs, k1=1.5, b=0.75):
        self.k1, self.b = k1, b
        self.tf = [{} for _ in docs]
        for i, d in enumerate(docs):
            for t in tokens(d):
                self.tf[i][t] = self.tf[i].get(t, 0) + 1
        self.len = np.array([sum(tf.values()) for tf in self.tf], dtype=float); self.avg = self.len.mean()
        df = {}
        for tf in self.tf:
            for t in tf:
                df[t] = df.get(t, 0) + 1
        self.idf = {t: math.log(1 + (len(docs) - c + 0.5) / (c + 0.5)) for t, c in df.items()}
    def scores(self, query):
        s = np.zeros(len(self.tf))
        for t in set(tokens(query)):
            if t in self.idf:
                for i, tf in enumerate(self.tf):
                    f = tf.get(t, 0)
                    if f:
                        s[i] += self.idf[t] * f * (self.k1 + 1) / (f + self.k1 * (1 - self.b + self.b * self.len[i] / self.avg))
        return s
bm25 = BM25(chunks)

PARAPHRASES = [
    ("r01", "Are boats cancelled when the sea is too rough?", "p02"),
    ("r02", "Can I go up the tower alone to fix something?", "p03"),
    ("r03", "How big are the blades?", "p04"),
    ("r04", "Who do I call in the middle of the night about an accident?", "p18"),
    ("r05", "What if the oil in the gearbox gets dirty?", "p07"),
    ("r06", "Do you stop work when whales or seals swim close?", "p13"),
    ("r07", "How much spare equipment is kept on shore?", "p16"),
    ("r08", "Is the farm allowed to produce at full output?", "p21"),
    ("r09", "What should new colleagues do before their first trip?", "p22"),
    ("r10", "How should rubbish and old oil be handled?", "p15"),
]
ANSWERABLE = [(q[0], q[1], q[2]) for q in QUESTIONS if q[2]] + PARAPHRASES     # 24 questions with a gold passage
UNANSWERABLE = [("u01", q[1], None) for q in QUESTIONS if not q[2]] + [
    ("u03", "What is the best pasta recipe?", None), ("u04", "How do I reset my email password?", None)]
print(len(ANSWERABLE), "answerable and", len(UNANSWERABLE), "unanswerable questions")
Loading weights:   0%|          | 0/103 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 103/103 [00:00<00:00, 3181.27it/s]

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
24 answerable and 4 unanswerable questions

2. State, nodes and edges

The state is a typed dictionary that every node can read and partly update. A node is a function from the state to a partial state (only the keys it changes). Where several nodes may write the same key, a reducer says how to combine them: operator.add on a list means "append". Our state carries the question, a working query, the two retrievers' results, the fused ranking, the answer and a trace list in which every node records that it ran (which also lets us test the graph's path).

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver

class State(TypedDict, total=False):
    question: str
    query: str                                   # what the retrievers actually search (may be rewritten)
    route: str                                   # "doc" | "data"
    semantic: list                               # [(passage id, cosine)] best first
    lexical: list                                # [(passage id, bm25 score)] best first
    fused: list                                  # [(passage id, rrf score)]
    answer: str
    citations: list
    status: str                                  # "ok" | "retry" | "refused" | "unsupported"
    attempts: int
    max_retries: int
    history: Annotated[list, operator.add]       # questions asked in this conversation (reducer: append)
    trace: Annotated[list, operator.add]         # names of the nodes that ran, in order

The nodes

Each node is a small, separately testable function. They map one-to-one onto the stages of the course:

Node What it does Real-world counterpart
contextualize adds the previous question to a short follow-up an LLM "condense question" step
classify routes: documents or database an LLM or classifier router
semantic, lexical the two retrievers of module 02 vector search, keyword search
fuse Reciprocal Rank Fusion of the two rankings the hybrid step of module 02
generate extractive answer with a citation (module 01) the LLM call of module 04
check is the answer supported by its citation? a validator, later a judge (modules 07, 08)
rewrite rewrites the query after a failed check an LLM query rewriter
LATENCY = 0.0          # seconds each retriever "takes" (set above 0 to mimic a remote search service)
DATA_WORDS = re.compile(r"\b(last|this|next) (month|week|year)\b|\baverage\b|\btotal\b|\btrend\b|\bhow many\b.*\b(were|was|last|today)\b", re.I)
FOLLOW_UP = re.compile(r"^(and|what about|so)\b|\b(that|it|those|they|there)\b", re.I)

def contextualize(s):
    history = s.get("history", [])
    q = s["question"]
    if history and len(tokens(q)) <= 6 and FOLLOW_UP.search(q):
        q = f"{q} (previous question: {history[-1]})"
    return {"query": q, "attempts": 0, "history": [s["question"]], "trace": ["contextualize"]}

def classify(s):
    return {"route": "data" if DATA_WORDS.search(s["question"]) else "doc", "trace": ["classify"]}

def semantic(s):
    time.sleep(LATENCY)
    sims = E @ minilm.encode(s["query"], normalize_embeddings=True)
    return {"semantic": [(ids[i], float(sims[i])) for i in np.argsort(-sims)], "trace": ["semantic"]}

def lexical(s):
    time.sleep(LATENCY)
    sc = bm25.scores(s["query"])
    return {"lexical": [(ids[i], float(sc[i])) for i in np.argsort(-sc)], "trace": ["lexical"]}

def fuse(s, k=60, weights=(1.0, 0.25)):
    total = {pid: 0.0 for pid in ids}
    for ranking, w in zip((s["semantic"], s["lexical"]), weights):
        for rank, (pid, _) in enumerate(ranking, start=1):
            total[pid] += w / (k + rank)
    return {"fused": sorted(total.items(), key=lambda kv: -kv[1]), "trace": ["fuse"]}

def sql_stub(s):
    return {"answer": "That needs figures from a database; module 06 builds the safe text-to-SQL step.", "citations": [], "status": "ok", "trace": ["sql_stub"]}

def generate(s):
    pid = s["fused"][0][0]
    text = chunks[ids.index(pid)].split(". ", 1)[1]
    asked, searched = set(tokens(s["question"])), set(tokens(s["query"]))
    # the sentence closest to what the user typed; the (possibly rewritten) query only breaks ties
    best = max(re.split(r"(?<=[.!?])\s+", text), key=lambda x: (len(asked & set(tokens(x))), len(searched & set(tokens(x)))))
    return {"answer": f"{best} [{pid}]", "citations": [pid], "trace": ["generate"]}

def check(s):
    # supported = the cited sentence shares at least one content word with the ORIGINAL question
    sentence = s["answer"].rsplit(" [", 1)[0]
    supported = bool(set(tokens(s["question"])) & set(tokens(sentence)))
    if supported:
        return {"status": "ok", "trace": ["check"]}
    if s["attempts"] < s["max_retries"]:
        return {"status": "retry", "trace": ["check"]}
    return {"status": "unsupported", "answer": "I cannot find that in the handbook.", "citations": [], "trace": ["check"]}

SYNONYMS = {"boat": "vessel", "boats": "vessels", "qualification": "certificate training", "blackout": "loss of grid connection",
            "dolphin": "marine mammal", "whales": "marine mammal", "seals": "marine mammal", "alone": "lone working",
            "rough": "wave height", "rubbish": "waste", "accident": "injury incident", "dirty": "particle count",
            "trip": "transfer", "tower": "turbine nacelle", "colleagues": "staff", "produce": "export", "equipment": "spares", "blades": "blade rotor"}
def rewrite(s):
    # a hand-written stand-in for an LLM query rewriter: expand words the handbook words differently
    q = " ".join(SYNONYMS.get(w, w) for w in re.findall(r"[a-z0-9]+", s["query"].lower()))
    return {"query": q, "attempts": s["attempts"] + 1, "trace": ["rewrite"]}

def refuse(s):
    return {"answer": "I cannot find that in the handbook.", "citations": [], "status": "refused", "trace": ["refuse"]}

3. Wiring the graph

Edges say what runs next. Three kinds appear:

  • a plain edge, add_edge(a, b): always go from a to b;
  • a fan-out: two edges leaving one node (or a conditional edge returning a list of nodes) start those nodes in the same step, in parallel;
  • a conditional edge, add_conditional_edges(a, fn): fn looks at the state and names the next node.

A node with edges from two parents (add_edge([a, b], c)) waits for both: that is the join.

MIN_COSINE = 0.2       # below this the best passage is unrelated: refuse before generating (see the calibration in module 02)

def route_after_classify(s):
    return ["semantic", "lexical"] if s["route"] == "doc" else "sql_stub"      # a list = run both in parallel

def route_after_fuse(s):
    best_cosine = s["semantic"][0][1]
    return "generate" if best_cosine >= MIN_COSINE else "refuse"

def route_after_check(s):
    return "rewrite" if s["status"] == "retry" else END

def build_graph(checkpointer=None):
    g = StateGraph(State)
    for name, fn in [("contextualize", contextualize), ("classify", classify), ("semantic", semantic), ("lexical", lexical),
                     ("fuse", fuse), ("sql_stub", sql_stub), ("generate", generate), ("check", check), ("rewrite", rewrite), ("refuse", refuse)]:
        g.add_node(name, fn)
    g.add_edge(START, "contextualize")
    g.add_edge("contextualize", "classify")
    g.add_conditional_edges("classify", route_after_classify, ["semantic", "lexical", "sql_stub"])
    g.add_edge(["semantic", "lexical"], "fuse")                         # join: wait for both retrievers
    g.add_conditional_edges("fuse", route_after_fuse, ["generate", "refuse"])
    g.add_edge("generate", "check")
    g.add_conditional_edges("check", route_after_check, ["rewrite", END])
    g.add_edge("rewrite", "semantic")                                   # the loop: search again with the new query
    g.add_edge("rewrite", "lexical")
    g.add_edge("sql_stub", END)
    g.add_edge("refuse", END)
    return g.compile(checkpointer=checkpointer)

app = build_graph()

def outcome(out, gold):
    refused = not out["citations"]
    if gold is None:
        return "correct refusal" if refused else "invented answer"
    if refused:
        return "refused (answerable)"
    return "correct" if out["citations"][0] == gold else "wrong passage"

def ask(question, max_retries=1, graph=None, config=None):
    return (graph or app).invoke({"question": question, "max_retries": max_retries}, config)

out = ask("What wave height stops the crew boats from sailing?")
print(out["answer"]); print("path:", " -> ".join(out["trace"]))
Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. [p02]
path: contextualize -> classify -> lexical -> semantic -> fuse -> generate -> check

The trace shows the real path: semantic and lexical appear together because they ran in the same step. Here is the graph itself, read from the compiled object (not drawn by hand), as Mermaid text you can paste into any Mermaid renderer, and as a picture. Dashed edges are conditional.

graph = app.get_graph()
print(graph.draw_mermaid().split("---")[-1].strip()[:900], "...")
graph TD;
	__start__([<p>__start__</p>]):::first
	contextualize(contextualize)
	classify(classify)
	semantic(semantic)
	lexical(lexical)
	fuse(fuse)
	sql_stub(sql_stub)
	generate(generate)
	check(check)
	rewrite(rewrite)
	refuse(refuse)
	__end__([<p>__end__</p>]):::last
	__start__ --> contextualize;
	check -.-> __end__;
	check -.-> rewrite;
	classify -.-> lexical;
	classify -.-> semantic;
	classify -.-> sql_stub;
	contextualize --> classify;
	fuse -.-> generate;
	fuse -.-> refuse;
	generate --> check;
	lexical --> fuse;
	rewrite --> lexical;
	rewrite --> semantic;
	semantic --> fuse;
	refuse --> __end__;
	sql_stub --> __end__;
	classDef default fill:#f2f0ff,line-height:1.2
	classDef first fill-opacity:0
	classDef last fill:#bfb6fc ...
import networkx as nx
layer = {"__start__": 0, "contextualize": 1, "classify": 2, "sql_stub": 3, "semantic": 3, "lexical": 3, "fuse": 4,
         "refuse": 5, "generate": 5, "check": 6, "rewrite": 7, "__end__": 8}
G = nx.DiGraph()
for e in graph.edges:
    G.add_edge(e.source, e.target, conditional=e.conditional)
rows = {}
pos = {}
for node in G.nodes:
    rows.setdefault(layer[node], []).append(node)
for x, nodes in rows.items():
    for j, node in enumerate(sorted(nodes)):
        pos[node] = (x, -(j - (len(nodes) - 1) / 2) * 1.2)
fig, ax = plt.subplots(figsize=(11, 4.2))
nx.draw_networkx_nodes(G, pos, node_color="#cfe8f3", node_size=1900, edgecolors="#2a7ab0", ax=ax)
nx.draw_networkx_labels(G, pos, labels={n: n.strip("_") for n in G.nodes}, font_size=7, ax=ax)
solid = [(u, v) for u, v, d in G.edges(data=True) if not d["conditional"]]
dashed = [(u, v) for u, v, d in G.edges(data=True) if d["conditional"]]
nx.draw_networkx_edges(G, pos, edgelist=solid, ax=ax, node_size=1900, arrowsize=12, connectionstyle="arc3,rad=0.08")
nx.draw_networkx_edges(G, pos, edgelist=dashed, ax=ax, node_size=1900, arrowsize=12, style="dashed", edge_color="#c0392b", connectionstyle="arc3,rad=0.15")
ax.axis("off"); plt.tight_layout(); plt.show()
print(len(G.nodes), "nodes,", len(solid), "plain edges,", len(dashed), "conditional edges")
notebook output figure
12 nodes, 9 plain edges, 7 conditional edges

4. Routing: documents, data or refusal

Three questions, three paths. The first goes to both retrievers, the second is a question about figures that the handbook cannot hold (it goes to the SQL stub), the third is off topic and is refused by the gate after fusion without calling the generator.

Picture: the path of one document question

The next cell prints the real path for three questions; this picture is the first of them, "How soon must an injury be reported?". Both retrievers run in the same step, the gate checks the best cosine against MIN_COSINE = 0.2, and the check node ends the run with a supported answer. The red box lists the other exits: the data question goes to sql_stub, an off-topic one is refused after fuse.

diagram(860, 400, [
    ("box", 20, 150, 120, 60, "Question\nInjury report", "gold"),
    ("arrow", 140, 180, 175, 180, ""),
    ("box", 175, 150, 120, 60, "contextualize\nclassify: doc", "blue"),
    ("arrow", 295, 165, 345, 100, "parallel"),
    ("arrow", 295, 195, 345, 260, "parallel"),
    ("box", 345, 70, 120, 55, "lexical\nBM25", "blue"),
    ("box", 345, 235, 120, 55, "semantic\nMiniLM", "blue"),
    ("arrow", 465, 100, 515, 165, ""),
    ("arrow", 465, 262, 515, 195, ""),
    ("box", 515, 150, 100, 60, "fuse\nRRF", "blue"),
    ("arrow", 615, 180, 650, 180, "gate"),
    ("box", 650, 150, 110, 60, "generate\nbest sentence", "green"),
    ("arrow", 705, 210, 705, 255, ""),
    ("box", 640, 255, 130, 60, "check\nsupported: ok", "violet"),
    ("arrow", 705, 315, 705, 345, ""),
    ("box", 690, 345, 160, 45, "Answer [p19]", "gold"),
    ("box", 20, 330, 320, 55, "Other exits: data question to sql_stub;\noff-topic to refuse (cosine below 0.2)", "red"),
    ("arrow", 660, 315, 610, 345, "retry"),
    ("box", 470, 330, 140, 45, "rewrite query", "grey"),
    ("arrow", 470, 345, 420, 290, "search again"),
], "Data flow through the compiled graph (real trace)")
notebook output figure
for q in ["How soon must an injury be reported?",
          "What was the availability last month?",
          "What is the best pasta recipe?"]:
    out = ask(q)
    print(f"Q: {q}\n   path: {' -> '.join(out['trace'])}\n   A: {out['answer']}\n")
Q: How soon must an injury be reported?
   path: contextualize -> classify -> lexical -> semantic -> fuse -> generate -> check
   A: Any injury or near miss is reported within one hour and a written report is filed within 24 hours. [p19]

Q: What was the availability last month?
   path: contextualize -> classify -> sql_stub
   A: That needs figures from a database; module 06 builds the safe text-to-SQL step.

Q: What is the best pasta recipe?
   path: contextualize -> classify -> lexical -> semantic -> fuse -> refuse
   A: I cannot find that in the handbook.

5. Parallel branches: what they buy

Our two retrievers take milliseconds, so running them in parallel gains nothing visible. The reason to do it is the real case: a vector database over the network, or a model call, each taking a few hundred milliseconds. Set LATENCY to simulate that and compare the graph (the two branches in the same step) with running the same two functions one after the other.

LATENCY = 0.3
q = "How soon must an injury be reported?"
t0 = time.perf_counter(); ask(q); t_graph = time.perf_counter() - t0

t0 = time.perf_counter()
s = {"question": q, "query": q}
s.update(semantic(s)); s.update(lexical(s)); s.update(fuse(s))
t_seq = time.perf_counter() - t0
LATENCY = 0.0
print(f"two retrievers of {0.3} s each: graph (parallel) {t_graph:.2f} s   sequential {t_seq:.2f} s   ->  {t_seq / t_graph:.1f}x")
two retrievers of 0.3 s each: graph (parallel) 0.35 s   sequential 0.63 s   ->  1.8x

The graph's wall time is about the slower of the two branches, not their sum. (The latency is simulated with sleep; with real network calls the effect is the same, and it grows with the number of parallel branches.) That is why the wind-farm agent runs document retrieval and image analysis side by side.

6. The check-and-retry loop

check asks whether the answer is supported: does the sentence we are about to return share content words with the question? If not, and we have retries left, the graph goes to rewrite and searches again with the new query; after the retry budget is spent it refuses instead of returning an unsupported answer. Watch it on a question phrased the way a person would ask it:

# find a question that the loop rescues: refused or wrong with no retry, correct with one retry
rescued = next(q for _, q, gold in ANSWERABLE
               if outcome(ask(q, max_retries=0), gold) != "correct" and outcome(ask(q, max_retries=1), gold) == "correct")
for retries in (0, 1):
    out = ask(rescued, max_retries=retries)
    print(f"max_retries={retries}: {' -> '.join(out['trace'])}\n   query used: {out['query']!r}\n   A: {out['answer']}\n")
max_retries=0: contextualize -> classify -> lexical -> semantic -> fuse -> generate -> check
   query used: 'What qualification do I need before getting on the boat?'
   A: I cannot find that in the handbook.

max_retries=1: contextualize -> classify -> lexical -> semantic -> fuse -> generate -> check -> rewrite -> lexical -> semantic -> fuse -> generate -> check
   query used: 'what certificate training do i need before getting on the vessel'
   A: Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer vessel. [p01]

The question was picked by the code (the first one in the battery that the loop turns from a failure into a correct answer). With no retry budget the system fails a question the handbook can answer; with one retry the rewritten query finds the right passage. Now the honest measurement over the whole battery: the 24 answerable questions and the 4 that cannot be answered, with no retry, with one retry, and with the retrieval gate and check removed (always answer from the top passage).

def run_battery(max_retries):
    counts = {}
    for _, q, gold in ANSWERABLE + UNANSWERABLE:
        label = outcome(ask(q, max_retries=max_retries), gold)
        counts[label] = counts.get(label, 0) + 1
    return counts

def run_no_guards():
    counts = {}
    for _, q, gold in ANSWERABLE + UNANSWERABLE:
        s = {"question": q, "query": q}
        s.update(semantic(s)); s.update(lexical(s)); s.update(fuse(s)); s.update(generate(s))
        counts[outcome(s, gold)] = counts.get(outcome(s, gold), 0) + 1
    return counts

labels = ["correct", "wrong passage", "refused (answerable)", "correct refusal", "invented answer"]
results = {"no guards (always answer)": run_no_guards(), "gate + check, 0 retries": run_battery(0), "gate + check, 1 retry": run_battery(1)}
print(f"{'':28s}" + "".join(f"{l:>22s}" for l in labels))
for name, c in results.items():
    print(f"{name:28s}" + "".join(f"{c.get(l, 0):22d}" for l in labels))
                                           correct         wrong passage  refused (answerable)       correct refusal       invented answer
no guards (always answer)                       19                     5                     0                     0                     4
gate + check, 0 retries                         17                     4                     3                     4                     0
gate + check, 1 retry                           18                     4                     2                     4                     0

Read the table with three cautions. First, the guards do their job on the questions that should be refused: without them every unanswerable question gets an invented answer. Second, guards are not free: the answerable questions that were already right without them are not all right with them (the check refuses a few correct answers whose wording shares no word with the question), so the "correct" column goes down from 19 to 17 before the retry brings it back to 18. You trade a few wrongful refusals for no invented answers; for a safety handbook that is usually the right trade, but it is a trade, and the table is how you see it. Third, the retry rewriter above is a hand-written synonym table made after looking at these very questions, so the gain it shows here is an upper bound that demonstrates the mechanism, not an estimate of what a real rewriter achieves on new questions. Module 07 shows how to measure that properly on questions kept apart from the tuning. Notice also what neither guard can do: when the top passage is the wrong one but shares a word with the question, the check passes and a wrong passage is returned. Cheap checks catch gross failures; subtle ones need a judge.

7. Checkpointing: memory per conversation

A graph compiled with a checkpointer saves its state after every step under a thread_id. Invoke it again with the same thread and the saved state comes back, so the history list keeps growing and the contextualize node can resolve a follow-up. A different thread starts fresh. MemorySaver keeps this in RAM; LangGraph also has SQLite and Postgres checkpointers for durable conversations and for pausing a graph to wait for a person.

chat = build_graph(checkpointer=MemorySaver())
alice = {"configurable": {"thread_id": "alice"}}
bob = {"configurable": {"thread_id": "bob"}}

follow_up = "And who decides that?"
r1 = ask("What wave height stops the crew boats from sailing?", graph=chat, config=alice)
print("alice 1:", r1["answer"])
r2 = ask(follow_up, graph=chat, config=alice)
print("alice 2:", follow_up, "->", r2["answer"], "\n         query used:", repr(r2["query"]))
r3 = ask(follow_up, graph=chat, config=bob)
print("bob   1:", follow_up, "->", r3["answer"])
print("\nsaved history of thread alice:", chat.get_state(alice).values["history"])
print("saved history of thread bob  :", chat.get_state(bob).values["history"])
alice 1: Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. [p02]
alice 2: And who decides that? -> The duty coordinator decides at 06:00 and may cancel later. [p02] 
         query used: 'And who decides that? (previous question: What wave height stops the crew boats from sailing?)'
bob   1: And who decides that? -> I cannot find that in the handbook.

saved history of thread alice: ['What wave height stops the crew boats from sailing?', 'And who decides that?']
saved history of thread bob  : ['And who decides that?']

Alice's follow-up works because the thread remembers her first question; the same words from Bob, who has no history, are searched as-is and find nothing relevant. Per-conversation state is also a privacy boundary: never let two users share a thread_id.

8. When a graph is worth it

A graph earns its keep when the flow has branches, loops, parallel work or state that must survive: the wind-farm agent's two-way recommend/retrain decision and its parallel retrieval and vision steps are exactly that. For a straight retrieve-then-generate service, a plain function is simpler and just as testable, and the previous modules' pipeline needs no framework at all. Two practical benefits are independent of how complex your flow is: the trace (you can assert in a test that a question took the path you expect) and the explicit state (every intermediate value is inspectable, which is what the observability module of this course builds on).

Exercises

  1. Add "data" questions of your own to DATA_WORDS and confirm with trace that they skip both retrievers.
  2. Set max_retries=3. Does the measured table change? Why does a second rewrite of the same text add nothing, and what would a rewriter need to do differently on the second attempt?
  3. Add a node log_unanswered between refuse and END that appends the question to a list. This is the seed of module 08's improvement loop: what would you do with that list?
  4. Remove contextualize from the graph and repeat the Alice follow-up. What does the checkpointer still save, and why does the follow-up now fail even though the history is there?