Module 01 ended with BM25 failing on questions that mean the same as a passage but use different words. The fix is semantic search: turn every text into a vector of numbers (an embedding) such that texts with similar meaning land close together, then search by distance. This module builds it, measures it against BM25 on three kinds of question, and shows a result that surprises most newcomers: neither wins everywhere, and a naive 50/50 combination does not automatically win either. Combining them (hybrid retrieval) is what the supervised chat assistant built in this project uses, but the weights matter, and you only find that out by measuring.
You will learn
- What an embedding is and how similarity search works (cosine similarity)
- Running a small embedding model on a laptop CPU
- Where semantic search beats BM25 (paraphrases) and where it loses (codes and exact names)
- Reciprocal Rank Fusion, the simple, robust way to merge two rankings
- Calibrating when to refuse with embedding scores
- When a vector index (FAISS and friends) starts to matter, measured
1. The corpus and the BM25 baseline
The same Harbourview handbook as module 01 (copied here so the notebook stands alone), and the same BM25 we wrote there. Besides the 14 answerable questions of module 01 we add two new sets that no method was tuned on:
- Paraphrases: ten questions written the way a colleague would actually ask them, sharing few words with the handbook ("Are boats cancelled when the sea is too rough?").
- Identifiers: five questions about exact codes and names (
PF-203,YW-110,OSS1...).
# The Harbourview handbook: 22 passages, fictional, written for this course
PASSAGES = [
["p01", "Safety / Access", "Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer vessel. Certificates are checked at the harbour gate and expire after two years."],
["p02", "Safety / Weather limits", "Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later."],
["p03", "Safety / Lone working", "Lone working inside a nacelle is forbidden. Every task inside a turbine needs at least two people, and both must carry a personal locator beacon."],
["p04", "Turbines / Specification", "Each of the 40 turbines is rated at 8.0 MW with a rotor diameter of 164 m. Cut-in wind speed is 3 m/s, rated speed is 12.5 m/s and cut-out speed is 25 m/s."],
["p05", "Turbines / Naming", "Turbines are named HV01 to HV40, numbered from the western edge of the array. The offshore substation is OSS1, in the centre."],
["p06", "Maintenance / Schedule", "Every turbine gets a planned service once a year between April and September. The service takes two technicians about two days, plus weather delays."],
["p07", "Maintenance / Gearbox oil", "Gearbox oil is sampled every six months. If the particle count exceeds ISO 4406 code 18/16/13, a filter change is scheduled within 14 days."],
["p08", "Maintenance / Blade inspection", "Blades are inspected by drone every 12 months and after any lightning alarm. Leading-edge erosion above grade 3 is repaired in the next planned service."],
["p09", "Faults / Alarm levels", "Alarms have three levels. Level 1 is informational. Level 2 needs a remote check within 4 hours. Level 3 stops the turbine and needs a site visit within 24 hours when the weather allows."],
["p10", "Faults / Pitch system", "A pitch system fault (code PF-203) usually means a low accumulator pressure. Reset it remotely once. If it returns within 48 hours, send a crew to replace the hydraulic accumulator."],
["p11", "Faults / Yaw misalignment", "A yaw error above 8 degrees for more than 10 minutes raises alarm YW-110. Persistent errors point to a faulty wind vane: the vane is recalibrated, then replaced if the error remains."],
["p12", "Faults / Grid loss", "On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid operator confirms voltage within 5% of nominal for 10 minutes."],
["p13", "Environment / Marine mammals", "Piling and heavy cable work pause when a marine mammal is sighted within 500 m. Work resumes after 30 minutes without a sighting."],
["p14", "Environment / Bird collision", "Between October and March, turbines HV01 to HV08 are curtailed at night during the peak migration periods announced by the ornithology team."],
["p15", "Environment / Waste", "Used oil, filters and composite offcuts are brought ashore and logged in the waste register. Nothing is discharged at sea."],
["p16", "Logistics / Spare parts", "The harbour warehouse keeps the critical spares: two pitch accumulators, four wind vanes and a spare converter module. Reorder when stock falls below one unit."],
["p17", "Logistics / Helicopter", "A helicopter is used only for medical evacuation and for urgent Level 3 visits when vessels cannot sail. Booking goes through the duty coordinator."],
["p18", "Contacts / Duty coordinator", "The duty coordinator is reachable 24 hours a day on the control room line. Out-of-hours safety incidents are reported there first."],
["p19", "Reporting / Incidents", "Any injury or near miss is reported within one hour and a written report is filed within 24 hours. Environmental incidents follow the same deadlines."],
["p20", "Reporting / Performance", "Availability is reported monthly. The target is 97% time-based availability, counting weather downtime as available."],
["p21", "Grid / Export", "Power is exported at 66 kV through two cables to the onshore substation. The export limit agreed with the grid operator is 300 MW."],
["p22", "Training / Induction", "New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer."],
]
# (id, question, gold passage, fact the answer must contain); None = the handbook does not say
QUESTIONS = [
["q01", "What wave height stops the crew boats from sailing?", "p02", "1.5 m"],
["q02", "How often is the gearbox oil sampled?", "p07", "six months"],
["q03", "Who may work alone inside a nacelle?", "p03", "forbidden"],
["q04", "What is the rotor diameter of a turbine?", "p04", "164 m"],
["q05", "What do I do if the pitch fault comes back after a reset?", "p10", "replace the hydraulic accumulator"],
["q06", "When must a Level 3 alarm be attended on site?", "p09", "24 hours"],
["q07", "How soon must an injury be reported?", "p19", "one hour"],
["q08", "What is the largest amount of power the farm is allowed to send ashore?", "p21", "300 MW"],
["q09", "Which turbines are switched off at night in the autumn to protect birds?", "p14", "HV01 to HV08"],
["q10", "How many spare wind vanes are kept in stock?", "p16", "four wind vanes"],
["q11", "What qualification do I need before getting on the boat?", "p01", "GWO Basic Safety Training"],
["q12", "When can the turbines restart after a blackout?", "p12", "10 minutes"],
["q13", "When can a helicopter be used?", "p17", "medical evacuation"],
["q14", "What happens when a dolphin is seen near the piling work?", "p13", "pause"],
["q15", "What is the price of electricity per MWh?", None, None],
["q16", "Who is the chief executive of the operating company?", None, None],
]
import re, math, time
import numpy as np
import matplotlib.pyplot as plt
PARAPHRASES = [
("r01", "Are boats cancelled when the sea is too rough?", "p02"),
("r02", "Can I go up the tower alone to fix something?", "p03"),
("r03", "How big are the blades?", "p04"),
("r04", "Who do I call in the middle of the night about an accident?", "p18"),
("r05", "What if the oil in the gearbox gets dirty?", "p07"),
("r06", "Do you stop work when whales or seals swim close?", "p13"),
("r07", "How much spare equipment is kept on shore?", "p16"),
("r08", "Is the farm allowed to produce at full output?", "p21"),
("r09", "What should new colleagues do before their first trip?", "p22"),
("r10", "How should rubbish and old oil be handled?", "p15"),
]
IDENTIFIERS = [
("e01", "What does code PF-203 mean?", "p10"),
("e02", "alarm YW-110", "p11"),
("e03", "ISO 4406 18/16/13", "p07"),
("e04", "Where is OSS1?", "p05"),
("e05", "66 kV cables", "p21"),
]
SETS = {
"original (14)": [(q[0], q[1], q[2]) for q in QUESTIONS if q[2]],
"paraphrases (10)": PARAPHRASES,
"identifiers (5)": IDENTIFIERS,
}
ids = [p[0] for p in PASSAGES]
chunks = [f"{section}. {text}" for _, section, text in PASSAGES]
STOP = set("a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you".split())
def tokens(text):
return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
class BM25:
def __init__(self, docs, k1=1.5, b=0.75):
self.k1, self.b = k1, b
self.tf = [{} for _ in docs]
for i, d in enumerate(docs):
for t in tokens(d):
self.tf[i][t] = self.tf[i].get(t, 0) + 1
self.len = np.array([sum(tf.values()) for tf in self.tf], dtype=float)
self.avg = self.len.mean()
df = {}
for tf in self.tf:
for t in tf:
df[t] = df.get(t, 0) + 1
n = len(docs)
self.idf = {t: math.log(1 + (n - c + 0.5) / (c + 0.5)) for t, c in df.items()}
def scores(self, query):
s = np.zeros(len(self.tf))
for t in set(tokens(query)):
if t in self.idf:
for i, tf in enumerate(self.tf):
f = tf.get(t, 0)
if f:
s[i] += self.idf[t] * f * (self.k1 + 1) / (f + self.k1 * (1 - self.b + self.b * self.len[i] / self.avg))
return s
bm25 = BM25(chunks)
print(len(PASSAGES), "passages;", {k: len(v) for k, v in SETS.items()})
2. Embeddings: meaning as geometry
An embedding model reads a text and outputs a fixed-length vector (here 384 numbers). It was trained so that sentences that mean the same thing get vectors that point the same way. The standard similarity is the cosine of the angle between two vectors: 1 means the same direction, 0 unrelated.
We use all-MiniLM-L6-v2, a 22-million-parameter model that runs comfortably on a CPU and is small enough
to ship inside an application. (Larger hosted models such as Amazon Titan Text Embeddings, used in other
projects of this portfolio, give better quality at a per-token price; the code around them is identical.)
The first run downloads about 90 MB.
Picture: meaning as geometry
The same question meets two passages. A keyword search finds no word in common with the right one, so it ranks it far down. An embedding model turns each text into 384 numbers, and the question lands much closer to the passage that answers it than to an unrelated one. The cosine values are printed by the cell below.
# diagram() is a small SVG helper defined in the downloadable notebook
diagram(800, 330, [
("box", 20, 112, 200, 64, "Question\nAre boats cancelled when\nthe sea is too rough?", "gold"),
("arrow", 220, 144, 256, 144, ""),
("box", 256, 118, 150, 52, "all-MiniLM-L6-v2\ntext to 384 numbers", "blue"),
("arrow", 406, 138, 470, 76, ""),
("text", 436, 100, "cosine 0.45", 12, "end"),
("arrow", 406, 150, 470, 212, ""),
("text", 436, 204, "cosine 0.07", 12, "end"),
("box", 470, 44, 310, 62, "[p02] Safety / Weather limits\nCrew transfer vessels do not sail...", "green"),
("box", 470, 182, 310, 62, "[p07] Maintenance / Gearbox oil\nGearbox oil is sampled every six months", "grey"),
("note", 20, 262, 370, 52, "Keyword view (BM25): the question shares no word\nwith p02, so p02 ranks 8th of 22."),
("note", 410, 262, 370, 52, "Embedding view: same meaning, same direction,\nso p02 ranks 1st of 22."),
], "Same meaning, different words")
from sentence_transformers import SentenceTransformer
t0 = time.time()
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
print(f"model loaded in {time.time() - t0:.1f} s, {sum(p.numel() for p in model.parameters()) / 1e6:.0f} M parameters, dimension {model.get_sentence_embedding_dimension()}")
t0 = time.time()
E = model.encode(chunks, normalize_embeddings=True) # unit length, so a dot product IS the cosine
print(f"embedded {len(chunks)} passages in {time.time() - t0:.2f} s -> matrix {E.shape}, {E.astype(np.float16).nbytes / 1024:.0f} KiB as float16")
def semantic_scores(query):
return E @ model.encode(query, normalize_embeddings=True)
for a, b in [("Are boats cancelled when the sea is too rough?", "Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m"),
("Are boats cancelled when the sea is too rough?", "Gearbox oil is sampled every six months"),
("What does code PF-203 mean?", "A pitch system fault (code PF-203) usually means a low accumulator pressure")]:
va, vb = model.encode([a, b], normalize_embeddings=True)
print(f"cosine {va @ vb:5.2f} {a!r} vs {b[:60]!r}")
The paraphrase and the passage that answers it score clearly higher than the paraphrase and an unrelated passage, although they share almost no words. That is the whole idea. Now the fair test.
3. Lexical against semantic, on three kinds of question
We run the same evaluation as module 01 (hit@1, hit@3, MRR), now on three question sets.
def ranks_of(scorer, questions):
out = []
for _, question, gold in questions:
order = [ids[i] for i in np.argsort(-scorer(question))]
out.append(order.index(gold) + 1)
return np.array(out)
def report(scorers):
print(f"{'method':22s}" + "".join(f"{name:>34s}" for name in SETS))
print(f"{'':22s}" + f"{'hit@1 hit@3 MRR':>34s}" * len(SETS))
for label, scorer in scorers.items():
row = f"{label:22s}"
for questions in SETS.values():
r = ranks_of(scorer, questions)
row += f"{(r == 1).mean():>14.2f}{(r <= 3).mean():>8.2f}{(1 / r).mean():>8.2f}"
print(row)
report({"BM25 (lexical)": bm25.scores, "MiniLM (semantic)": semantic_scores})
Look at the columns, not the average. Semantic search should win on the paraphrases: that is what it was
built for. But look at the identifiers: a model that learned meaning has no special reason to treat
PF-203 or YW-110 as exact keys, whereas BM25 treats a rare token as the strongest possible signal. Let us
see which questions each one misses.
for name, questions in list(SETS.items())[1:]:
print(name)
for (qid, question, gold), rb, rs in zip(questions, ranks_of(bm25.scores, questions), ranks_of(semantic_scores, questions)):
if rb > 1 or rs > 1:
print(f" {qid} BM25 rank {rb:2d} | semantic rank {rs:2d} {question}")
4. Hybrid retrieval with Reciprocal Rank Fusion
Each method fails on different questions, so ask both and merge the two rankings. The trap is that the scores are not comparable: BM25 scores run from 0 to maybe 15, cosines from 0 to 1. Normalising and adding them is fragile. Reciprocal Rank Fusion (RRF) sidesteps scores entirely and uses only the ranks:
$$\text{RRF}(d)=\sum_{\text{rankers}}\frac{1}{k+\text{rank}(d)}\qquad(k=60\text{ is the standard constant})$$
A passage near the top of either list gets a high fused score, and one near the top of both gets the highest. It needs no tuning, no training and no score calibration, which is why it is the default hybrid recipe in most search systems.
Picture: how two rankings become one
The first paraphrase question goes to both rankers. Each one reports the rank of the right passage (p02), and RRF turns the two ranks into one score, with a weight for BM25. The four boxes at the bottom are the measured result of changing that weight, on the ten paraphrase questions of the table you will see below.
diagram(800, 340, [
("box", 20, 76, 190, 66, "Question\nAre boats cancelled when\nthe sea is too rough?", "gold"),
("arrow", 210, 98, 270, 66, ""),
("arrow", 210, 122, 270, 154, ""),
("box", 270, 40, 200, 54, "BM25 ranker\nright passage p02: rank 8", "blue"),
("box", 270, 128, 200, 54, "Semantic ranker\nright passage p02: rank 1", "blue"),
("arrow", 470, 67, 540, 100, ""),
("text", 484, 60, "w / (60+8)", 12, "start"),
("arrow", 470, 155, 540, 122, ""),
("text", 484, 178, "1 / (60+1)", 12, "start"),
("box", 540, 70, 240, 76, "Reciprocal Rank Fusion\nscore = w / (60 + rank)\nsummed over both rankers", "violet"),
("text", 400, 214, "Paraphrase questions, hit@1, as the BM25 weight w changes", 13, "middle"),
("box", 20, 228, 180, 62, "w = 0\nhit@1 0.90\n(semantic only)", "green"),
("box", 220, 228, 180, 62, "w = 0.25\nhit@1 0.60", "gold"),
("box", 420, 228, 180, 62, "w = 0.5\nhit@1 0.50", "red"),
("box", 620, 228, 160, 62, "w = 1 (equal)\nhit@1 0.50", "red"),
("note", 20, 302, 760, 30, "Both rankers vote with their own ranks, so the weak ranker on this kind of question can drag the fusion down."),
], "Data flow of hybrid retrieval")
def rrf(scorers, k=60, weights=None):
weights = weights or [1.0] * len(scorers)
def fused(query):
total = np.zeros(len(ids))
for scorer, w in zip(scorers, weights):
order = np.argsort(-scorer(query))
for rank, i in enumerate(order, start=1):
total[i] += w / (k + rank)
return total
return fused
hybrid = rrf([bm25.scores, semantic_scores])
report({"BM25 (lexical)": bm25.scores, "MiniLM (semantic)": semantic_scores, "Hybrid (RRF)": hybrid})
Read this table carefully, because it does not say what the textbooks say. On this corpus the plain 50/50 hybrid fixes the identifier misses of semantic search, but on the paraphrases it is dragged down by the weaker ranker: BM25 puts the wrong passages high on those questions and equal-weight fusion believes it. Fusion helps when both rankers are decent and wrong in different places; it does not rescue a ranker that is simply bad on a whole kind of question. The knob that matters is the weight. Sweep it:
print("weight of BM25 relative to the semantic ranker (semantic weight = 1):")
report({f"BM25 weight {w:g}": rrf([bm25.scores, semantic_scores], weights=[w, 1.0]) for w in (0, 0.25, 0.5, 1.0)})
Lowering the BM25 weight trades one weakness for the other: at 0.25 the identifier questions are all right again and the paraphrase MRR recovers part of the way (0.78, against 0.66 at equal weight and 0.95 for the semantic ranker alone), but no setting here is best on every set. With only 29 questions the differences between close numbers are within noise, so do not copy a weight from this table: tune it on your own questions (module 07 shows how to build and size that evaluation) and re-check it whenever the corpus or the embedding model changes.
5. Refusing with embedding scores
With BM25 a score of zero meant "no shared word". Cosine scores never reach zero (everything is a little similar to everything), so the refusal threshold has to be read from data again, using the two unanswerable questions plus some off-topic ones.
off_topic = ["What is the price of electricity per MWh?", "Who is the chief executive of the operating company?",
"What is the best pasta recipe?", "How do I reset my email password?", "Who won the football match yesterday?"]
best_off = [semantic_scores(q).max() for q in off_topic]
best_on = [semantic_scores(q[1]).max() for qs in SETS.values() for q in qs]
fig, ax = plt.subplots(figsize=(8, 2.8))
ax.scatter(best_on, np.ones(len(best_on)), s=60, color="#2a7ab0", label="answerable")
ax.scatter(best_off, np.zeros(len(best_off)), s=60, color="#c0392b", label="not in the handbook")
ax.set_yticks([0, 1], ["off topic", "answerable"]); ax.set_xlabel("cosine similarity of the best passage"); ax.set_ylim(-.6, 1.6); ax.grid(alpha=.3)
plt.tight_layout(); plt.show()
print(f"answerable: lowest {min(best_on):.2f} off-topic: highest {max(best_off):.2f}")
The two clusters overlap here (the lowest answerable score is below the highest off-topic one), so no single threshold is right: raising it refuses real questions, lowering it answers nonsense. A common production answer is to not decide here: keep a permissive threshold, let the generator see the passages, and have the generator (and later the supervisor of module 08) say "the documents do not contain this". Retrieval proposes; the answer stage and the evaluator dispose.
6. When do you need a vector index?
We compared the query vector with all 22 passage vectors, a brute-force search. It is exact and, at this size, instantaneous. Libraries such as FAISS (used by another project of this portfolio), pgvector or managed vector databases exist for corpora where brute force becomes too slow, by trading a little accuracy for a much faster approximate nearest-neighbour search. Measure before you adopt one:
rng = np.random.default_rng(0)
dim = E.shape[1]
for n in (1_000, 10_000, 100_000):
M = rng.standard_normal((n, dim)).astype(np.float32)
M /= np.linalg.norm(M, axis=1, keepdims=True)
q = M[0]
t0 = time.perf_counter()
for _ in range(20):
top = np.argpartition(-(M @ q), 5)[:5]
ms = (time.perf_counter() - t0) / 20 * 1000
print(f"{n:>9,d} vectors: {ms:7.2f} ms per brute-force query, {M.nbytes / 1e6:6.0f} MB in float32")
Brute force scales linearly: ten times the vectors, ten times the time. A few thousand chunks is
milliseconds, and an application serving a few hundred thousand chunks on one CPU is still within an
interactive budget. The decision to adopt an approximate index is therefore driven by corpus size and
query volume, not by fashion; below about a hundred thousand chunks a NumPy matrix shipped as a single
file (as float16 it is half the size) is simpler to deploy, test and keep correct. The same logic made
two of this portfolio's retrieval components choose in-process FAISS or plain BM25 over an always-on
database server.
Exercises
- Add five paraphrases of your own and one identifier question. Where does each method fail?
- Change the RRF constant
kto 5 and to 500. How sensitive is the hybrid? - Give BM25 a weight of 2 in
rrf. Which set gains, which loses, and how would you know in production which direction to turn the knob without a labelled battery? - Embed
section + textversustextalone. Does putting the section title in the embedded text help?