Free course · 12 hands-on notebooks · runs on any laptop
RAG Systems 1001
How to build retrieval-augmented generation systems you can trust: find the right passages, answer only from them, measure every answer, and let a second model judge the answers so the system keeps improving. Eleven notebooks with real, executed results you can read here or run yourself.
- 12notebooks
- ~18 hof study
- 4parts
- 100%runnable code
Measured, not claimed
Every comparison is run on a question battery with known answers: lexical against semantic against hybrid retrieval, framework against framework, answers graded by rules and by a judge model. When a technique does not help, the notebook says so.
A system that improves itself
A supervisor model grades every answer for correctness, faithfulness and citations, finds retrieval misses and turns user corrections into documentation fixes: the continuous-improvement loop, built step by step.
Honest about the tools
LangChain, LlamaIndex, LangGraph, LiteLLM, Pydantic and Langfuse are shown where they earned their place in real projects. LangSmith, GraphRAG and Neo4j were not needed, and the course says why.
Syllabus
Four parts, 12 notebooks. Read the explanations, run the cells, then try the exercises at the end of each notebook. Your progress is remembered in this browser.
Foundations
Chunking, lexical and semantic retrieval, hybrid search and grounded answers with citations.
From a Question to a Grounded Answer
The anatomy of RAG, chunking measured, BM25 from scratch, hit@k and MRR, answers with citations, and refusing when the documents are silent.
- chunking
- BM25
- hit@k & MRR
- citations
- refusal
Semantic and Hybrid Search
Embeddings and cosine similarity on a CPU, lexical against semantic against hybrid on paraphrases and exact codes, Reciprocal Rank Fusion and its weights, and when a vector index pays.
- embeddings
- semantic search
- RRF
- FAISS
- calibration
Building
Frameworks, model routing and typed output, agents as state graphs and answers from data.
LangChain and LlamaIndex
The same corpus through two frameworks: loaders, splitters, vector stores and retrievers, what each abstracts and costs, and when to skip both.
- LangChain
- LlamaIndex
- FAISS
- retrievers
- framework choice
Model Routing and Typed Output
One interface to many models with LiteLLM, Pydantic schemas for requests, responses and structured model output, validation and retries.
- LiteLLM
- Pydantic
- structured output
- validation
- fallbacks
Agents with LangGraph
A RAG pipeline as a state graph: nodes, conditional routing, parallel branches, tools and checkpoints.
- LangGraph
- state graph
- routing
- tools
- checkpoints
Answers from Data: Safe Text-to-SQL
When the answer is a number, not a passage: a model writes one read-only SELECT, a guard validates it, DuckDB runs it and the result is cited like a passage.
- text-to-SQL
- DuckDB
- SQL guard
- citations
- read-only
Quality
Evaluating a RAG system properly and a supervisor model that drives continuous improvement.
Evaluating a RAG System
A question battery with references taken from the data, rule-based grading, a judge model, faithfulness and citation checks, sizing an evaluation and before/after fixes.
- question battery
- grading
- LLM-as-judge
- faithfulness
- before/after
The Supervisor Model
A second model judges every answer, a bank keeps the validated ones, user remarks are judged and become documentation fixes: the continuous-improvement loop.
- supervisor
- answer bank
- user feedback
- doc fixes
- improvement loop
Production
Observability, serverless production on a budget, graphs, and a supervised assistant built end to end.
Observability: Langfuse and LangSmith
Traces, token and cost accounting, scores attached to traces and dashboards; why one tracing stack was enough.
- Langfuse
- LangSmith
- tracing
- cost
- scores
Production RAG on a Budget, and GraphRAG
A serverless search endpoint, memory limits, cost per question, security and rate limits; and GraphRAG and Neo4j: when a knowledge graph pays and why we did not use one.
- serverless
- cost per question
- security
- GraphRAG
- Neo4j
Capstone: A Supervised RAG Assistant
Build the whole system end to end on a new corpus, evaluate it, run the supervisor and fix what it finds.
- end to end
- evaluation
- supervisor
- fixes
A Real Corpus: Nine State of the Union Addresses
Repeat the method on real speeches: chunk size, metadata, refusing what was never said, and fixes on the query side, with honest intervals.
- real documents
- metadata filter
- refusal
- held-out set
Run it yourself
Each lesson page shows the notebook with its saved outputs, so the whole course can be read in the browser. To experiment, download a notebook or open it in Google Colab from the lesson page, or clone the repository and run everything locally :
git clone https://github.com/fborbon/academy.git
cd academy/rag-systems
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
jupyter lab
What you need
- Python: comfortable with functions, lists and dictionaries (the Intro module of Python 1001 covers them).
- Maths: school level; the notebooks explain vectors, cosine similarity and rank metrics as they go.
- Hardware: any laptop from the last decade. Everything runs on a CPU in seconds to minutes. No API keys: where a hosted model would be used, the notebook runs a local stand-in and marks the optional real call.
Data and credits
- The Harbourview handbook and its questions are fictional, written for this course.
- Module 11 uses a second fictional corpus, a smart-thermostat knowledge base (Thermio), also written for this course.
- Module 12 uses nine State of the Union addresses (1950 to 2006) from the
state_unioncorpus distributed with NLTK, which reproduces the texts of The American Presidency Project. They are works of the United States federal government, in the public domain. - Embedding model: all-MiniLM-L6-v2 (Apache 2.0), downloaded once from Hugging Face.