{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fbef30d1",
   "metadata": {},
   "source": [
    "# 01 · De la pregunta a una respuesta fundamentada\n",
    "\n",
    "Un modelo de lenguaje responde con lo que memorizó durante su entrenamiento. Nunca ha leído **tus** documentos,\n",
    "no puede saber qué cambió la semana pasada y, cuando no sabe, a menudo se inventa algo que suena bien.\n",
    "La **generación aumentada por recuperación (RAG, por sus siglas en inglés)** lo arregla dividiendo el trabajo en dos:\n",
    "primero *encontrar* los pasajes que importan y después pedirle al modelo que responda *usando solo esos pasajes* y que\n",
    "diga de dónde sale cada afirmación.\n",
    "\n",
    "Este primer módulo construye el sistema RAG completo más pequeño posible, con Python casi puro y NumPy, y mide cada\n",
    "paso. No se llama a ningún modelo ni hace falta ninguna clave: el objetivo es ver las piezas en movimiento y dónde\n",
    "fallan, antes de que los frameworks las escondan.\n",
    "\n",
    "**Aprenderás**\n",
    "1. La anatomía de un sistema RAG y de qué es responsable cada parte\n",
    "2. El troceado: por qué *dónde* cortas los documentos decide si la respuesta llega a poder encontrarse, medido\n",
    "3. Recuperación léxica con BM25, escrita desde cero\n",
    "4. Cómo medir la recuperación con honestidad: hit@k y rango recíproco medio (MRR) sobre una batería de preguntas\n",
    "5. Respuestas fundamentadas con citas, y negarse a responder cuando los documentos callan\n",
    "6. Los modos de fallo que corrigen los módulos siguientes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61be0c2f",
   "metadata": {},
   "source": [
    "## 1. La anatomía de un sistema RAG\n",
    "\n",
    "```\n",
    "  documentos ──► trocear ──► indexar ────────┐              (se hace una vez, sin conexión)\n",
    "                                             ▼\n",
    "  pregunta ──► recuperar los k mejores ──► armar el prompt ──► generar ──► respuesta + citas\n",
    "                    ▲                                              │\n",
    "                    └──────────── evaluar y mejorar ◄──────────────┘   (se hace de forma continua)\n",
    "```\n",
    "\n",
    "| Etapa | Pregunta a la que responde | Fallo típico |\n",
    "|---|---|---|\n",
    "| **Trocear** | ¿Cuál es la unidad de texto que podemos encontrar y citar? | El dato queda cortado por la mitad, o enterrado en un trozo enorme |\n",
    "| **Indexar / recuperar** | ¿Qué trozos son relevantes para esta pregunta? | El trozo correcto existe pero no está entre los *k* primeros |\n",
    "| **Prompt** | ¿Qué ve el modelo? | Demasiados trozos, ruido, contradicciones |\n",
    "| **Generar** | ¿Puede responder *solo* con lo que se le dio? | Añade datos de memoria, o ignora los trozos |\n",
    "| **Evaluar** | ¿Es correcta y, si no lo es, por qué? | Nadie mira, así que nada mejora |\n",
    "\n",
    "El resto del curso son estas cinco etapas, en este orden, y termina con un sistema que se evalúa a sí mismo y\n",
    "devuelve sus propios errores a la documentación."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92f214d2",
   "metadata": {},
   "source": [
    "### Imagen: las cinco etapas y dónde falla cada una\n",
    "\n",
    "Lee la imagen de izquierda a derecha. La fila de arriba es el camino de una pregunta. Las cajas rojas de debajo\n",
    "son el fallo típico de cada etapa, y por eso el curso mide cada etapa por separado en lugar de juzgar solo la\n",
    "respuesta final."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8fb18b2e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.054437Z",
     "iopub.status.busy": "2026-11-17T21:17:51.053990Z",
     "iopub.status.idle": "2026-11-17T21:17:51.096979Z",
     "shell.execute_reply": "2026-11-17T21:17:51.094826Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 800 300\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"800\" height=\"300\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"400.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Un sistema RAG en cinco etapas</text><rect x=\"24\" y=\"48\" width=\"132\" height=\"62\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"90.0\" y=\"75.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Trozo</text><text x=\"90.0\" y=\"92.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">corta el texto</text><line x1=\"156\" y1=\"79\" x2=\"172\" y2=\"79\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"172\" y=\"48\" width=\"132\" height=\"62\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"238.0\" y=\"75.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Recuperar</text><text x=\"238.0\" y=\"92.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">los k mejores</text><line x1=\"304\" y1=\"79\" x2=\"320\" y2=\"79\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"320\" y=\"48\" width=\"132\" height=\"62\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"386.0\" y=\"75.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Prompt</text><text x=\"386.0\" y=\"92.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">lo que ve el modelo</text><line x1=\"452\" y1=\"79\" x2=\"468\" y2=\"79\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"468\" y=\"48\" width=\"132\" height=\"62\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"534.0\" y=\"75.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Generar</text><text x=\"534.0\" y=\"92.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">responde con [1]</text><line x1=\"600\" y1=\"79\" x2=\"616\" y2=\"79\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"616\" y=\"48\" width=\"160\" height=\"62\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"696.0\" y=\"75.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Evaluar</text><text x=\"696.0\" y=\"92.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">¿acierta? ¿por qué?</text><text x=\"24\" y=\"138\" text-anchor=\"start\" font-size=\"12\" fill=\"#333\">Fallo típico de cada etapa</text><rect x=\"24\" y=\"150\" width=\"132\" height=\"66\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"90.0\" y=\"170.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">El dato queda</text><text x=\"90.0\" y=\"187.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">partido o</text><text x=\"90.0\" y=\"204.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">enterrado</text><rect x=\"172\" y=\"150\" width=\"132\" height=\"66\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"238.0\" y=\"170.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">El trozo correcto</text><text x=\"238.0\" y=\"187.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">no está entre</text><text x=\"238.0\" y=\"204.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">los k mejores</text><rect x=\"320\" y=\"150\" width=\"132\" height=\"66\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"386.0\" y=\"179.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Ruido, demasiados</text><text x=\"386.0\" y=\"196.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">trozos, conflictos</text><rect x=\"468\" y=\"150\" width=\"132\" height=\"66\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"534.0\" y=\"170.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Añade datos de</text><text x=\"534.0\" y=\"187.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">memoria, ignora</text><text x=\"534.0\" y=\"204.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">los trozos</text><rect x=\"616\" y=\"150\" width=\"160\" height=\"66\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"696.0\" y=\"170.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Nadie mira,</text><text x=\"696.0\" y=\"187.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">así que nada</text><text x=\"696.0\" y=\"204.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">mejora</text><rect x=\"24\" y=\"236\" width=\"752\" height=\"46\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/><text x=\"34\" y=\"256\" font-size=\"12\" fill=\"#444\">Las dos primeras etapas se ejecutan una vez, sin conexión. Las otras tres, con cada pregunta.</text><text x=\"34\" y=\"272\" font-size=\"12\" fill=\"#444\">La evaluación devuelve lo que encuentra a los trozos, a la recuperación y al prompt.</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import SVG, display\n",
    "import html as _h\n",
    "\n",
    "_COL = {  # fill, stroke  (light cards with dark text: readable in light and dark themes)\n",
    "    \"gold\": (\"#fff4c2\", \"#b89d28\"), \"blue\": (\"#dbeafe\", \"#3b82f6\"), \"green\": (\"#dcfce7\", \"#16a34a\"),\n",
    "    \"red\": (\"#fee2e2\", \"#dc2626\"), \"grey\": (\"#f1f1f4\", \"#8a8aa0\"), \"violet\": (\"#ede9fe\", \"#7c3aed\"),\n",
    "}\n",
    "\n",
    "def diagram(width, height, items, title=None):\n",
    "    # items: (\"box\", x, y, w, h, \"text\\nmore text\", colour) | (\"arrow\", x1, y1, x2, y2, \"label\") |\n",
    "    #        (\"text\", x, y, \"text\", size, \"start|middle|end\") | (\"note\", x, y, w, h, \"text\")\n",
    "    out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 {width} {height}\" width=\"100%\" '\n",
    "           f'font-family=\"Helvetica, Arial, sans-serif\">',\n",
    "           '<defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" '\n",
    "           'orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs>',\n",
    "           f'<rect width=\"{width}\" height=\"{height}\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/>']\n",
    "    if title:\n",
    "        out.append(f'<text x=\"{width / 2}\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">{_h.escape(title)}</text>')\n",
    "    for it in items:\n",
    "        kind = it[0]\n",
    "        if kind == \"box\":\n",
    "            _, x, y, w, h, text, col = it\n",
    "            fill, stroke = _COL[col]\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"9\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"1.6\"/>')\n",
    "            lines = text.split(\"\\n\")\n",
    "            y0 = y + h / 2 - (len(lines) - 1) * 8.5 + 4.5\n",
    "            for k, ln in enumerate(lines):\n",
    "                bold = ' font-weight=\"700\"' if k == 0 and len(lines) > 1 else \"\"\n",
    "                out.append(f'<text x=\"{x + w / 2}\" y=\"{y0 + k * 17}\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\"{bold}>{_h.escape(ln)}</text>')\n",
    "        elif kind == \"arrow\":\n",
    "            _, x1, y1, x2, y2, label = it\n",
    "            out.append(f'<line x1=\"{x1}\" y1=\"{y1}\" x2=\"{x2}\" y2=\"{y2}\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/>')\n",
    "            if label:\n",
    "                out.append(f'<text x=\"{(x1 + x2) / 2}\" y=\"{(y1 + y2) / 2 - 6}\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">{_h.escape(label)}</text>')\n",
    "        elif kind == \"text\":\n",
    "            _, x, y, text, size, anchor = it\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x}\" y=\"{y + k * (size + 4)}\" text-anchor=\"{anchor}\" font-size=\"{size}\" fill=\"#333\">{_h.escape(ln)}</text>')\n",
    "        elif kind == \"note\":\n",
    "            _, x, y, w, h, text = it\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/>')\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x + 10}\" y=\"{y + 20 + k * 16}\" font-size=\"12\" fill=\"#444\">{_h.escape(ln)}</text>')\n",
    "    out.append(\"</svg>\")\n",
    "    display(SVG(\"\".join(out)))\n",
    "\n",
    "\n",
    "diagram(800, 300, [\n",
    "    (\"box\", 24, 48, 132, 62, \"Trozo\\ncorta el texto\", \"blue\"),\n",
    "    (\"arrow\", 156, 79, 172, 79, \"\"),\n",
    "    (\"box\", 172, 48, 132, 62, \"Recuperar\\nlos k mejores\", \"blue\"),\n",
    "    (\"arrow\", 304, 79, 320, 79, \"\"),\n",
    "    (\"box\", 320, 48, 132, 62, \"Prompt\\nlo que ve el modelo\", \"blue\"),\n",
    "    (\"arrow\", 452, 79, 468, 79, \"\"),\n",
    "    (\"box\", 468, 48, 132, 62, \"Generar\\nresponde con [1]\", \"green\"),\n",
    "    (\"arrow\", 600, 79, 616, 79, \"\"),\n",
    "    (\"box\", 616, 48, 160, 62, \"Evaluar\\n¿acierta? ¿por qué?\", \"violet\"),\n",
    "    (\"text\", 24, 138, \"Fallo típico de cada etapa\", 12, \"start\"),\n",
    "    (\"box\", 24, 150, 132, 66, \"El dato queda\\npartido o\\nenterrado\", \"red\"),\n",
    "    (\"box\", 172, 150, 132, 66, \"El trozo correcto\\nno está entre\\nlos k mejores\", \"red\"),\n",
    "    (\"box\", 320, 150, 132, 66, \"Ruido, demasiados\\ntrozos, conflictos\", \"red\"),\n",
    "    (\"box\", 468, 150, 132, 66, \"Añade datos de\\nmemoria, ignora\\nlos trozos\", \"red\"),\n",
    "    (\"box\", 616, 150, 160, 66, \"Nadie mira,\\nasí que nada\\nmejora\", \"red\"),\n",
    "    (\"note\", 24, 236, 752, 46, \"Las dos primeras etapas se ejecutan una vez, sin conexión. Las otras tres, con cada pregunta.\\nLa evaluación devuelve lo que encuentra a los trozos, a la recuperación y al prompt.\"),\n",
    "], \"Un sistema RAG en cinco etapas\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d71fd677",
   "metadata": {},
   "source": [
    "## 2. El corpus\n",
    "\n",
    "Todos los notebooks de este curso usan el mismo conjunto pequeño y ficticio de documentos: el manual de operaciones de\n",
    "un parque eólico marino, *Harbourview*. Se escribió para el curso, así que no tiene licencia, ni contenido confidencial,\n",
    "ni ninguna empresa real. Tiene 22 pasajes cortos y cada uno contiene varios datos comprobables (un límite de oleaje,\n",
    "un plazo, un número de piezas). También tenemos 16 preguntas con respuesta conocida, dos de las cuales el manual no\n",
    "puede responder a propósito.\n",
    "\n",
    "Un corpus pequeño es una ventaja aquí: puedes leerlo entero, así que cuando la recuperación falla puedes ver *por qué*.\n",
    "El corpus se queda en inglés a propósito: son los datos, no la explicación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e5c00883",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.102687Z",
     "iopub.status.busy": "2026-11-17T21:17:51.102250Z",
     "iopub.status.idle": "2026-11-17T21:17:51.120500Z",
     "shell.execute_reply": "2026-11-17T21:17:51.118267Z"
    }
   },
   "outputs": [],
   "source": [
    "# El manual de Harbourview: 22 pasajes, ficticios, escritos para este curso\n",
    "PASSAGES = [\n",
    "    [\"p01\", \"Safety / Access\", \"Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer vessel. Certificates are checked at the harbour gate and expire after two years.\"],\n",
    "    [\"p02\", \"Safety / Weather limits\", \"Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later.\"],\n",
    "    [\"p03\", \"Safety / Lone working\", \"Lone working inside a nacelle is forbidden. Every task inside a turbine needs at least two people, and both must carry a personal locator beacon.\"],\n",
    "    [\"p04\", \"Turbines / Specification\", \"Each of the 40 turbines is rated at 8.0 MW with a rotor diameter of 164 m. Cut-in wind speed is 3 m/s, rated speed is 12.5 m/s and cut-out speed is 25 m/s.\"],\n",
    "    [\"p05\", \"Turbines / Naming\", \"Turbines are named HV01 to HV40, numbered from the western edge of the array. The offshore substation is OSS1, in the centre.\"],\n",
    "    [\"p06\", \"Maintenance / Schedule\", \"Every turbine gets a planned service once a year between April and September. The service takes two technicians about two days, plus weather delays.\"],\n",
    "    [\"p07\", \"Maintenance / Gearbox oil\", \"Gearbox oil is sampled every six months. If the particle count exceeds ISO 4406 code 18/16/13, a filter change is scheduled within 14 days.\"],\n",
    "    [\"p08\", \"Maintenance / Blade inspection\", \"Blades are inspected by drone every 12 months and after any lightning alarm. Leading-edge erosion above grade 3 is repaired in the next planned service.\"],\n",
    "    [\"p09\", \"Faults / Alarm levels\", \"Alarms have three levels. Level 1 is informational. Level 2 needs a remote check within 4 hours. Level 3 stops the turbine and needs a site visit within 24 hours when the weather allows.\"],\n",
    "    [\"p10\", \"Faults / Pitch system\", \"A pitch system fault (code PF-203) usually means a low accumulator pressure. Reset it remotely once. If it returns within 48 hours, send a crew to replace the hydraulic accumulator.\"],\n",
    "    [\"p11\", \"Faults / Yaw misalignment\", \"A yaw error above 8 degrees for more than 10 minutes raises alarm YW-110. Persistent errors point to a faulty wind vane: the vane is recalibrated, then replaced if the error remains.\"],\n",
    "    [\"p12\", \"Faults / Grid loss\", \"On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid operator confirms voltage within 5% of nominal for 10 minutes.\"],\n",
    "    [\"p13\", \"Environment / Marine mammals\", \"Piling and heavy cable work pause when a marine mammal is sighted within 500 m. Work resumes after 30 minutes without a sighting.\"],\n",
    "    [\"p14\", \"Environment / Bird collision\", \"Between October and March, turbines HV01 to HV08 are curtailed at night during the peak migration periods announced by the ornithology team.\"],\n",
    "    [\"p15\", \"Environment / Waste\", \"Used oil, filters and composite offcuts are brought ashore and logged in the waste register. Nothing is discharged at sea.\"],\n",
    "    [\"p16\", \"Logistics / Spare parts\", \"The harbour warehouse keeps the critical spares: two pitch accumulators, four wind vanes and a spare converter module. Reorder when stock falls below one unit.\"],\n",
    "    [\"p17\", \"Logistics / Helicopter\", \"A helicopter is used only for medical evacuation and for urgent Level 3 visits when vessels cannot sail. Booking goes through the duty coordinator.\"],\n",
    "    [\"p18\", \"Contacts / Duty coordinator\", \"The duty coordinator is reachable 24 hours a day on the control room line. Out-of-hours safety incidents are reported there first.\"],\n",
    "    [\"p19\", \"Reporting / Incidents\", \"Any injury or near miss is reported within one hour and a written report is filed within 24 hours. Environmental incidents follow the same deadlines.\"],\n",
    "    [\"p20\", \"Reporting / Performance\", \"Availability is reported monthly. The target is 97% time-based availability, counting weather downtime as available.\"],\n",
    "    [\"p21\", \"Grid / Export\", \"Power is exported at 66 kV through two cables to the onshore substation. The export limit agreed with the grid operator is 300 MW.\"],\n",
    "    [\"p22\", \"Training / Induction\", \"New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer.\"],\n",
    "]\n",
    "\n",
    "# (id, pregunta, pasaje correcto, dato que debe contener la respuesta); None = el manual no lo dice\n",
    "QUESTIONS = [\n",
    "    [\"q01\", \"What wave height stops the crew boats from sailing?\", \"p02\", \"1.5 m\"],\n",
    "    [\"q02\", \"How often is the gearbox oil sampled?\", \"p07\", \"six months\"],\n",
    "    [\"q03\", \"Who may work alone inside a nacelle?\", \"p03\", \"forbidden\"],\n",
    "    [\"q04\", \"What is the rotor diameter of a turbine?\", \"p04\", \"164 m\"],\n",
    "    [\"q05\", \"What do I do if the pitch fault comes back after a reset?\", \"p10\", \"replace the hydraulic accumulator\"],\n",
    "    [\"q06\", \"When must a Level 3 alarm be attended on site?\", \"p09\", \"24 hours\"],\n",
    "    [\"q07\", \"How soon must an injury be reported?\", \"p19\", \"one hour\"],\n",
    "    [\"q08\", \"What is the largest amount of power the farm is allowed to send ashore?\", \"p21\", \"300 MW\"],\n",
    "    [\"q09\", \"Which turbines are switched off at night in the autumn to protect birds?\", \"p14\", \"HV01 to HV08\"],\n",
    "    [\"q10\", \"How many spare wind vanes are kept in stock?\", \"p16\", \"four wind vanes\"],\n",
    "    [\"q11\", \"What qualification do I need before getting on the boat?\", \"p01\", \"GWO Basic Safety Training\"],\n",
    "    [\"q12\", \"When can the turbines restart after a blackout?\", \"p12\", \"10 minutes\"],\n",
    "    [\"q13\", \"When can a helicopter be used?\", \"p17\", \"medical evacuation\"],\n",
    "    [\"q14\", \"What happens when a dolphin is seen near the piling work?\", \"p13\", \"pause\"],\n",
    "    [\"q15\", \"What is the price of electricity per MWh?\", None, None],\n",
    "    [\"q16\", \"Who is the chief executive of the operating company?\", None, None],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "62f54790",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.125059Z",
     "iopub.status.busy": "2026-11-17T21:17:51.124725Z",
     "iopub.status.idle": "2026-11-17T21:17:51.924086Z",
     "shell.execute_reply": "2026-11-17T21:17:51.921809Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22 passages, 560 words in total\n",
      "16 questions, 2 of them unanswerable by design\n",
      "\n",
      "[p01] Safety / Access\n",
      "    Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer\n",
      "    vessel. Certificates are checked at the harbour gate and expire after two years.\n",
      "\n",
      "[p02] Safety / Weather limits\n",
      "    Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed\n",
      "    at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later.\n",
      "\n",
      "[p03] Safety / Lone working\n",
      "    Lone working inside a nacelle is forbidden. Every task inside a turbine needs at least two people,\n",
      "    and both must carry a personal locator beacon.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patito/.local/lib/python3.12/site-packages/matplotlib/projections/__init__.py:63: UserWarning: Unable to import Axes3D. This may be due to multiple versions of Matplotlib being installed (e.g. as a system package and as a pip package). As a result, the 3D projection is not available.\n",
      "  warnings.warn(\"Unable to import Axes3D. This may be due to multiple versions of \"\n"
     ]
    }
   ],
   "source": [
    "import re, math, textwrap\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(len(PASSAGES), \"passages,\", sum(len(t.split()) for _, _, t in PASSAGES), \"words in total\")\n",
    "print(len(QUESTIONS), \"questions,\", sum(q[2] is None for q in QUESTIONS), \"of them unanswerable by design\\n\")\n",
    "for pid, section, text in PASSAGES[:3]:\n",
    "    print(f\"[{pid}] {section}\\n    {textwrap.fill(text, 100, subsequent_indent='    ')}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c05a585",
   "metadata": {},
   "source": [
    "## 3. Troceado: dónde cortas decide qué se puede encontrar\n",
    "\n",
    "El manual llega como un único texto largo, no como 22 pasajes ordenados. El divisor más simple corta cada *N*\n",
    "caracteres. Es rápido y no necesita entender el documento, y así empiezan muchos primeros prototipos. Su defecto es que\n",
    "un corte puede caer en mitad de la frase que contiene la respuesta.\n",
    "\n",
    "Podemos medirlo. Para cada pregunta con respuesta conocemos el dato que debe contener (por ejemplo `1.5 m`). Un troceado\n",
    "es *bueno para esa pregunta* cuando la **frase completa que enuncia el dato** sobrevive dentro de un único trozo: un\n",
    "trozo que contiene `1.5 m` pero perdió *\"...do not sail when the wave height exceeds\"* no sirve como evidencia.\n",
    "(Comprobar solo el número favorecería a todas las estrategias.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6eb8ff4e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.928881Z",
     "iopub.status.busy": "2026-11-17T21:17:51.928423Z",
     "iopub.status.idle": "2026-11-17T21:17:51.946825Z",
     "shell.execute_reply": "2026-11-17T21:17:51.945025Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3862 characters in the handbook\n",
      "strategy                                      chunks  mean chars  sentences kept whole\n",
      "fixed 200 chars                                   20         193                   71%\n",
      "fixed 200 chars, overlap 50                       26         197                   86%\n",
      "fixed 400 chars                                   10         386                   86%\n",
      "2 sentences                                       23         166                  100%\n",
      "by section (the document's own structure)         22         174                  100%\n"
     ]
    }
   ],
   "source": [
    "handbook = \"\\n\\n\".join(f\"{section}\\n{text}\" for _, section, text in PASSAGES)\n",
    "print(len(handbook), \"characters in the handbook\")\n",
    "\n",
    "def fixed_chunks(text, size, overlap=0):\n",
    "    step = size - overlap\n",
    "    return [text[i:i + size] for i in range(0, max(len(text) - overlap, 1), step)]\n",
    "\n",
    "def section_chunks(text):\n",
    "    return [block for block in text.split(\"\\n\\n\") if block.strip()]\n",
    "\n",
    "def sentence_chunks(text, per_chunk=2):\n",
    "    sentences = re.split(r\"(?<=[.!?])\\s+\", text.replace(\"\\n\\n\", \" \"))\n",
    "    return [\" \".join(sentences[i:i + per_chunk]) for i in range(0, len(sentences), per_chunk)]\n",
    "\n",
    "all_sentences = [s for _, _, t in PASSAGES for s in re.split(r\"(?<=[.!?])\\s+\", t)]\n",
    "# la frase completa que enuncia cada dato de respuesta\n",
    "fact_sentences = [next(s for s in all_sentences if q[3] in s) for q in QUESTIONS if q[3]]\n",
    "\n",
    "def fact_survival(chunks):\n",
    "    # proporción de frases de dato que aparecen enteras dentro de un único trozo\n",
    "    return sum(any(s in c for c in chunks) for s in fact_sentences) / len(fact_sentences)\n",
    "\n",
    "strategies = {\n",
    "    \"fixed 200 chars\":               fixed_chunks(handbook, 200),\n",
    "    \"fixed 200 chars, overlap 50\":   fixed_chunks(handbook, 200, 50),\n",
    "    \"fixed 400 chars\":               fixed_chunks(handbook, 400),\n",
    "    \"2 sentences\":                   sentence_chunks(handbook, 2),\n",
    "    \"by section (the document's own structure)\": section_chunks(handbook),\n",
    "}\n",
    "print(f\"{'strategy':44s}{'chunks':>8s}{'mean chars':>12s}{'sentences kept whole':>22s}\")\n",
    "for name, ch in strategies.items():\n",
    "    print(f\"{name:44s}{len(ch):8d}{np.mean([len(c) for c in ch]):12.0f}{fact_survival(ch):22.0%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3877a431",
   "metadata": {},
   "source": [
    "Los cortes ciegos de tamaño fijo rompen una parte de las frases de respuesta, y el solapamiento rescata parte de ellas\n",
    "a cambio de más texto repetido. Cortar siguiendo la estructura propia del documento conserva cada frase entera **y**\n",
    "hace que cada trozo sea una unidad que una persona reconocería (una sección), lo que además te da una cita natural:\n",
    "*Safety / Weather limits*. (Lee las cifras exactas en la tabla de arriba; dependen de dónde caigan los cortes.)\n",
    "\n",
    "**Regla práctica:** usa la estructura que ya existe (encabezados, secciones, filas, párrafos) antes de recurrir a un\n",
    "número de caracteres. Cuando no hay estructura, usa ventanas con conciencia de frase y con solapamiento. El tamaño es un\n",
    "compromiso: los trozos pequeños recuperan con precisión pero pierden contexto; los grandes conservan el contexto pero\n",
    "diluyen la coincidencia y cuestan más tokens de prompt. Mantenemos la sección como trozo durante el resto del curso."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9744291",
   "metadata": {},
   "source": [
    "## 4. Recuperación léxica con BM25\n",
    "\n",
    "La idea de recuperación buena más antigua es también la línea base simple más fuerte: **ordenar los trozos según las\n",
    "palabras de la pregunta que contienen**, dando más peso a las palabras raras (*accumulator*) que a las comunes (*the*).\n",
    "**BM25** es la forma estándar de hacerlo: una puntuación de frecuencia de término que se satura (la décima repetición de\n",
    "una palabra aporta poco) y se normaliza por la longitud del trozo, multiplicada por un peso de frecuencia inversa de\n",
    "documento (IDF).\n",
    "\n",
    "$$\\text{score}(q,d)=\\sum_{t\\in q}\\text{IDF}(t)\\cdot\\frac{f(t,d)\\,(k_1+1)}{f(t,d)+k_1\\,(1-b+b\\,|d|/\\overline{|d|})}$$\n",
    "\n",
    "Aquí tienes una implementación completa. Los buscadores han funcionado con esta fórmula durante décadas, y es lo que\n",
    "usó la búsqueda de Social Pulse de este proyecto para su recuperación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ff80d004",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.952560Z",
     "iopub.status.busy": "2026-11-17T21:17:51.952175Z",
     "iopub.status.idle": "2026-11-17T21:17:51.970719Z",
     "shell.execute_reply": "2026-11-17T21:17:51.969149Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What wave height stops the crew boats from sailing?\n",
      "   6.30  [p02] Safety / Weather limits. Crew transfer vessels do not sail when the significant wave heigh...\n",
      "   2.34  [p09] Faults / Alarm levels. Alarms have three levels. Level 1 is informational. Level 2 needs a...\n",
      "   1.81  [p01] Safety / Access. Technicians must hold a valid GWO Basic Safety Training certificate befor...\n"
     ]
    }
   ],
   "source": [
    "STOP = set(\"a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you\".split())\n",
    "\n",
    "def tokens(text):\n",
    "    return [w for w in re.findall(r\"[a-z0-9]+\", text.lower()) if w not in STOP]\n",
    "\n",
    "class BM25:\n",
    "    def __init__(self, docs, k1=1.5, b=0.75):\n",
    "        self.k1, self.b = k1, b\n",
    "        self.tf = [{} for _ in docs]\n",
    "        for i, d in enumerate(docs):\n",
    "            for t in tokens(d):\n",
    "                self.tf[i][t] = self.tf[i].get(t, 0) + 1\n",
    "        self.len = np.array([sum(tf.values()) for tf in self.tf], dtype=float)\n",
    "        self.avg = self.len.mean()\n",
    "        df = {}\n",
    "        for tf in self.tf:\n",
    "            for t in tf:\n",
    "                df[t] = df.get(t, 0) + 1\n",
    "        n = len(docs)\n",
    "        self.idf = {t: math.log(1 + (n - c + 0.5) / (c + 0.5)) for t, c in df.items()}\n",
    "\n",
    "    def scores(self, query):\n",
    "        s = np.zeros(len(self.tf))\n",
    "        for t in set(tokens(query)):\n",
    "            if t not in self.idf:\n",
    "                continue\n",
    "            for i, tf in enumerate(self.tf):\n",
    "                f = tf.get(t, 0)\n",
    "                if f:\n",
    "                    s[i] += self.idf[t] * f * (self.k1 + 1) / (f + self.k1 * (1 - self.b + self.b * self.len[i] / self.avg))\n",
    "        return s\n",
    "\n",
    "chunks = [f\"{section}. {text}\" for _, section, text in PASSAGES]    # el título de la sección también es buscable\n",
    "ids = [p[0] for p in PASSAGES]\n",
    "bm25 = BM25(chunks)\n",
    "\n",
    "def search(index, query, k=3):\n",
    "    s = index.scores(query)\n",
    "    order = np.argsort(-s)[:k]\n",
    "    return [(ids[i], float(s[i])) for i in order]\n",
    "\n",
    "q = \"What wave height stops the crew boats from sailing?\"\n",
    "print(q)\n",
    "for pid, score in search(bm25, q):\n",
    "    print(f\"  {score:5.2f}  [{pid}] {chunks[ids.index(pid)][:90]}...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec5e6091",
   "metadata": {},
   "source": [
    "## 5. Medir la recuperación con honestidad\n",
    "\n",
    "\"Parece que funciona\" no es una medición. Tenemos 14 preguntas con respuesta y un pasaje correcto conocido, así que\n",
    "podemos calcular:\n",
    "\n",
    "* **hit@k**: la proporción de preguntas cuyo pasaje correcto está entre los *k* primeros resultados. hit@1 es el\n",
    "  criterio estricto; hit@3 es lo que importa cuando el generador recibe tres pasajes.\n",
    "* **MRR** (rango recíproco medio): el promedio de 1/posición del pasaje correcto (1 si es el primero, 0.5 si es el\n",
    "  segundo, 0 si no está en la lista). Premia poner el pasaje correcto más arriba, no solo en algún lugar.\n",
    "\n",
    "Informa siempre estas cifras sobre preguntas con las que el sistema **no** se ajustó."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "618de4b4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.975330Z",
     "iopub.status.busy": "2026-11-17T21:17:51.975018Z",
     "iopub.status.idle": "2026-11-17T21:17:51.986357Z",
     "shell.execute_reply": "2026-11-17T21:17:51.984806Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  q08 gold p21 ranked 2: What is the largest amount of power the farm is allowed to send ashore?\n",
      "  q11 gold p01 ranked 2: What qualification do I need before getting on the boat?\n",
      "{'hit1': 0.857, 'hit3': 1.0, 'mrr': 0.929}\n"
     ]
    }
   ],
   "source": [
    "answerable = [q for q in QUESTIONS if q[2]]\n",
    "\n",
    "def evaluate(index, k=3, verbose=False):\n",
    "    ranks = []\n",
    "    for qid, question, gold, _ in answerable:\n",
    "        order = [pid for pid, _ in search(index, question, k=len(ids))]\n",
    "        rank = order.index(gold) + 1\n",
    "        ranks.append(rank)\n",
    "        if verbose and rank > 1:\n",
    "            print(f\"  {qid} gold {gold} ranked {rank}: {question}\")\n",
    "    ranks = np.array(ranks)\n",
    "    return dict(hit1=(ranks == 1).mean(), hit3=(ranks <= 3).mean(), mrr=(1 / ranks).mean()), ranks\n",
    "\n",
    "metrics, ranks = evaluate(bm25, verbose=True)\n",
    "print({k: round(float(v), 3) for k, v in metrics.items()})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53d3e0a3",
   "metadata": {},
   "source": [
    "BM25 pone primero el pasaje correcto en la mayoría de las preguntas sin ajustes, y por eso es una línea base que\n",
    "conviene ejecutar siempre antes de algo más sofisticado. Ahora lee las que no puso primero junto a su pasaje correcto:\n",
    "*\"send ashore\"* frente a *\"exported ... to the onshore substation\"*, y *\"qualification\"* frente a *\"certificate\"*. El\n",
    "autor y quien pregunta usaron palabras distintas para la misma idea.\n",
    "\n",
    "Esa es la debilidad que define a la búsqueda léxica: **empareja palabras, no significado.** Si quien pregunta usa\n",
    "palabras distintas a las del autor, el pasaje correcto puede perder frente a uno que solo comparte una palabra común.\n",
    "Cuenta las palabras que cada pregunta comparte con su pasaje correcto para verlo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eebb09fa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:51.991325Z",
     "iopub.status.busy": "2026-11-17T21:17:51.991010Z",
     "iopub.status.idle": "2026-11-17T21:17:52.001581Z",
     "shell.execute_reply": "2026-11-17T21:17:51.999700Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q01: 3 shared words with its gold passage ['crew', 'height', 'wave']\n",
      "q02: 3 shared words with its gold passage ['gearbox', 'oil', 'sampled']\n",
      "q03: 2 shared words with its gold passage ['inside', 'nacelle']\n",
      "q04: 2 shared words with its gold passage ['diameter', 'rotor']\n",
      "q05: 3 shared words with its gold passage ['fault', 'pitch', 'reset']\n",
      "q06: 4 shared words with its gold passage ['3', 'alarm', 'level', 'site']\n",
      "q07: 2 shared words with its gold passage ['injury', 'reported']\n",
      "q08: 1 shared words with its gold passage ['power']   <-- missed\n",
      "q09: 2 shared words with its gold passage ['night', 'turbines']\n",
      "q10: 4 shared words with its gold passage ['spare', 'stock', 'vanes', 'wind']\n",
      "q11: 1 shared words with its gold passage ['before']   <-- missed\n",
      "q12: 2 shared words with its gold passage ['restart', 'turbines']\n",
      "q13: 2 shared words with its gold passage ['helicopter', 'used']\n",
      "q14: 2 shared words with its gold passage ['piling', 'work']\n"
     ]
    }
   ],
   "source": [
    "for qid, question, gold, _ in answerable:\n",
    "    q_terms = set(tokens(question))\n",
    "    p_terms = set(tokens(chunks[ids.index(gold)]))\n",
    "    overlap = q_terms & p_terms\n",
    "    top = search(bm25, question, k=1)[0][0]\n",
    "    flag = \"\" if top == gold else \"   <-- missed\"\n",
    "    print(f\"{qid}: {len(overlap)} shared words with its gold passage {sorted(overlap)}{flag}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a84de933",
   "metadata": {},
   "source": [
    "### Imagen: una pregunta a través del sistema que hemos construido\n",
    "\n",
    "Este es el camino de la primera pregunta de la batería por el código de las secciones 4 a 6, con los números que\n",
    "imprime el notebook. Sigue las flechas: BM25 puntúa cada trozo, se conservan los tres primeros, la comprobación de\n",
    "rechazo compara la mejor puntuación con el umbral y solo entonces se elige una frase y se cita."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5c944fce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:52.005824Z",
     "iopub.status.busy": "2026-11-17T21:17:52.005516Z",
     "iopub.status.idle": "2026-11-17T21:17:52.019350Z",
     "shell.execute_reply": "2026-11-17T21:17:52.017649Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 800 372\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"800\" height=\"372\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"400.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Flujo de datos de una pregunta</text><rect x=\"20\" y=\"48\" width=\"210\" height=\"70\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"125.0\" y=\"70.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Pregunta</text><text x=\"125.0\" y=\"87.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">What wave height stops</text><text x=\"125.0\" y=\"104.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">the crew boats from sailing?</text><line x1=\"230\" y1=\"83\" x2=\"270\" y2=\"83\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"270\" y=\"48\" width=\"150\" height=\"70\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"345.0\" y=\"70.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">BM25</text><text x=\"345.0\" y=\"87.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">puntúa los 22</text><text x=\"345.0\" y=\"104.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">trozos</text><line x1=\"420\" y1=\"83\" x2=\"460\" y2=\"83\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"440.0\" y=\"77.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">top 3</text><rect x=\"460\" y=\"40\" width=\"320\" height=\"30\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"620.0\" y=\"59.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">6.30   [p02] Safety / Weather limits</text><rect x=\"460\" y=\"76\" width=\"320\" height=\"30\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"620.0\" y=\"95.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">2.34   [p09] Faults / Alarm levels</text><rect x=\"460\" y=\"112\" width=\"320\" height=\"30\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"620.0\" y=\"131.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">1.81   [p01] Safety / Access</text><line x1=\"620\" y1=\"142\" x2=\"620\" y2=\"188\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"500\" y=\"188\" width=\"280\" height=\"70\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"640.0\" y=\"210.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Comprobación de rechazo</text><text x=\"640.0\" y=\"227.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">la mejor puntuación 6.30 supera</text><text x=\"640.0\" y=\"244.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">el umbral 1.0: responder</text><line x1=\"500\" y1=\"223\" x2=\"440\" y2=\"223\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"230\" y=\"188\" width=\"210\" height=\"70\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"335.0\" y=\"210.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Generador extractivo</text><text x=\"335.0\" y=\"227.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">frase de p02 que comparte</text><text x=\"335.0\" y=\"244.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">más palabras</text><line x1=\"335\" y1=\"258\" x2=\"335\" y2=\"290\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"335.0\" y=\"268.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">citar [1]</text><rect x=\"20\" y=\"176\" width=\"190\" height=\"82\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/><text x=\"30\" y=\"196\" font-size=\"12\" fill=\"#444\">Por debajo del umbral</text><text x=\"30\" y=\"212\" font-size=\"12\" fill=\"#444\">el sistema se niega</text><text x=\"30\" y=\"228\" font-size=\"12\" fill=\"#444\">en vez de adivinar</text><text x=\"30\" y=\"244\" font-size=\"12\" fill=\"#444\">(sección 6).</text><rect x=\"20\" y=\"290\" width=\"760\" height=\"66\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"400.0\" y=\"310.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Respuesta con su cita</text><text x=\"400.0\" y=\"327.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the</text><text x=\"400.0\" y=\"344.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">wind speed at the harbour exceeds 20 m/s. [1]   (fuente: p02, Safety / Weather limits)</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diagram(800, 372, [\n",
    "    (\"box\", 20, 48, 210, 70, \"Pregunta\\nWhat wave height stops\\nthe crew boats from sailing?\", \"gold\"),\n",
    "    (\"arrow\", 230, 83, 270, 83, \"\"),\n",
    "    (\"box\", 270, 48, 150, 70, \"BM25\\npuntúa los 22\\ntrozos\", \"blue\"),\n",
    "    (\"arrow\", 420, 83, 460, 83, \"top 3\"),\n",
    "    (\"box\", 460, 40, 320, 30, \"6.30   [p02] Safety / Weather limits\", \"grey\"),\n",
    "    (\"box\", 460, 76, 320, 30, \"2.34   [p09] Faults / Alarm levels\", \"grey\"),\n",
    "    (\"box\", 460, 112, 320, 30, \"1.81   [p01] Safety / Access\", \"grey\"),\n",
    "    (\"arrow\", 620, 142, 620, 188, \"\"),\n",
    "    (\"box\", 500, 188, 280, 70, \"Comprobación de rechazo\\nla mejor puntuación 6.30 supera\\nel umbral 1.0: responder\", \"violet\"),\n",
    "    (\"arrow\", 500, 223, 440, 223, \"\"),\n",
    "    (\"box\", 230, 188, 210, 70, \"Generador extractivo\\nfrase de p02 que comparte\\nmás palabras\", \"green\"),\n",
    "    (\"arrow\", 335, 258, 335, 290, \"citar [1]\"),\n",
    "    (\"note\", 20, 176, 190, 82, \"Por debajo del umbral\\nel sistema se niega\\nen vez de adivinar\\n(sección 6).\"),\n",
    "    (\"box\", 20, 290, 760, 66, \"Respuesta con su cita\\nCrew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the\\nwind speed at the harbour exceeds 20 m/s. [1]   (fuente: p02, Safety / Weather limits)\", \"gold\"),\n",
    "], \"Flujo de datos de una pregunta\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65324d8",
   "metadata": {},
   "source": [
    "## 6. Generar una respuesta fundamentada, o negarse\n",
    "\n",
    "El generador es la parte que, en producción, es un modelo de lenguaje: *responde la pregunta usando solo estos pasajes\n",
    "numerados, cítalos como [1], [2] y di que no lo encuentras si la respuesta no está.* Los notebooks 03 y 04 conectan un\n",
    "modelo real. Aquí usamos un sustituto **extractivo** deliberadamente simple: devuelve la frase del pasaje principal que\n",
    "comparte más palabras con la pregunta. Basta para mostrar los dos comportamientos que más importan:\n",
    "\n",
    "1. **Citas**: toda respuesta apunta al pasaje del que salió, para que un lector pueda comprobarla.\n",
    "2. **Rechazo**: cuando la mejor puntuación es demasiado baja, di *\"no está en el manual\"* en lugar de adivinar. Un\n",
    "   sistema RAG que nunca se niega responderá la pregunta sobre el director ejecutivo con el pasaje menos malo.\n",
    "\n",
    "¿Dónde ponemos el umbral de rechazo? A partir de los datos, no a ojo: compara la puntuación máxima en las preguntas que\n",
    "el manual puede responder con la de las que no."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ad7ec1e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:52.023727Z",
     "iopub.status.busy": "2026-11-17T21:17:52.023426Z",
     "iopub.status.idle": "2026-11-17T21:17:52.243866Z",
     "shell.execute_reply": "2026-11-17T21:17:52.241896Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAENCAYAAACSMPPWAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAMwJJREFUeJzt3Xl4VOX99/HPZA9ZWEJCEgj7FhDCEpBFJQgKKAiWKlKqwSI+WBQpi1JBAZWlFhVERbEClgrYKogKRdmCFiVQJAiyhCVAgEAQgSQsATL38we/TB2ykHBCZkber+uaS+as33N/yXg+nHMmNmOMEQAAAABY4OXqAgAAAAB4PoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAQAAAMAyggUAAAAAywgWAAAAACzzcXUBcC92u11Hjx5VSEiIbDabq8sBAACACxljlJ2drejoaHl5FX9NgmABJ0ePHlVMTIyrywAAAIAbSU9PV40aNYpdhmABJyEhIZKu/OUJDQ0tl33a7XadOHFC4eHh10zCcA/0zPPQM89E3zwPPfM89Kx4WVlZiomJcZwjFodgASf5tz+FhoaWa7C4cOGCQkND+YH2EPTM89Azz0TfPA898zz0rGRKcos8owcAAADAMoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAQAAAMAyggUAAAAAywgWAAAAACwjWAAAAACwjGABAAAAwDKCBQAAAADLCBYAAAAALCNYAAAAALCMYAEAAADAMoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAQAAAMAyggUAAAAAywgWAAAAACwjWAAAAACwjGABAAAAwDKCBQAAAADLCBYAAAAALCNYAAAAALCMYAEAAADAMoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAQAAAMAyggUAAAAAywgWAAAAACwjWAAAAACwjGABAAAAwDKCBQAAAADLCBYAAAAALCNYAAAAALCMYAEAAADAMoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAQAAAMAyggUAAAAAywgWAAAAACwjWAAAAACwjGABAAAAwDKCBQAAAADLCBYA3IIxxtUllJon1lyWrvf4C1svf9rNPqY3wrXGtKRjXtRy9Kxo7jw27lyb5P71uZI7j42PqwsAcHMyxuiH9FNauCFNq3ce0/mLlxXo56MusZHq366OmsdUls1mc3WZTjyx5rJ0vcdf2HoBvt5qXauKjIz+e+Bn5V62O5YP8PVW1yZR+t1NMKY3wrX61KxGJW07fPqafSxqO3c2rqY2datqU9pJrbkJfw6Kkz9mX27aq09+/K/OXcxzm7Fx988vd6/PlTxpbGzGnWMPyl1WVpYqVqyoM2fOKDQ0tFz2abfblZmZqYiICHl5cRHNE1jt2aU8u8YvSdHSLeny9rIpz/6/j6H8971bxmji/S3k6+0efyc8seZfclXPiluvODZJRnLrMS0Ppe1bSfoUU6WC0n8+V2wfx93XXC9/9kOxfcvvUWHr34w9yx/7z7ekq3FladcpKT8uu3ps3P3zy9X1ufN5iKvHRirduaF7jR6c2Gw2ffrpp0XOP3DggGw2m1JSUsqtJsAqY4zGL0nRZynpklTghCX//Wcp6Rq/JMUtLvl6Ys1l6XqP/1rrFbvP//vvr3VMb4SS9in953PFzv8sJV2/mblWS7cU37erp/7afw6Kc/XYXz1krhwbd//8cvf6XMkTx4ZgAaBc/ZB+Sku3pOtan3/GSEu3pGvb4VPlU1gxPLHmsnS9x1/S9UqzTRStLMZbujLm+eHjete/2Xrmzp8R7lyb5P71uZInjg3Bopzl5eXJbrdfe0HgV2rhhjR5e5XsXlBvL5sWbki7wRVdmyfWXJau9/hLs15Jt4mildV4l4WbrWfu/BnhzrVJ7l+fK3ni2HhksFixYoVuu+02VapUSWFhYerZs6f27dsn6X+3By1evFidO3dWhQoVFBcXp++++86x/sGDB9WrVy9VrlxZQUFBatq0qZYvXy5Jio+P17Rp0xzL9unTR76+vsrJyZEkHT58WDabTXv37pUk5ebmatSoUapevbqCgoJ06623KikpybH+vHnzVKlSJX322Wdq0qSJ/P39dejQIW3atEl33XWXqlatqooVK6pTp076/vvvCxxrRkaGevToocDAQNWtW1cff/xxsWOzfft29ejRQ8HBwapWrZoefvhh/fTTT0Uun5ubq6ysLKeXdOV+w/J8GWPKfZ+8XNOzNTszZOxGXtI1X8ZutHpHhsuP1RNrdoeelWY9Tx1Td+pbWY13Wbxutp5dPfY2Nxobd//8cpf63PE8xF3Gxm4v+T+Ie+S3Qp09e1YjRoxQ8+bNlZOToxdeeEH333+/07MGY8eO1bRp09SgQQONHTtW/fv31969e+Xj46OhQ4fq4sWL+vrrrxUUFKQdO3YoODhYktSpUyclJSVp1KhRMsbom2++UaVKlfSf//xH3bt317p161S9enXVr19fkvTkk09qx44dWrRokaKjo7VkyRJ1795d27ZtU4MGDSRJ586d01/+8hf97W9/U1hYmCIiIrR//34lJiZq5syZMsbo1Vdf1T333KM9e/YoJCTEcRzPP/+8pk6dqhkzZmj+/Pl66KGHtG3bNsXGxhYYl9OnT+vOO+/UY489ptdff13nz5/Xs88+qwcffFBr1qwpdCynTJmiiRMnFph+4sQJXbhw4bp7VBp2u11nzpyRMcbtHppC4a63Z8YY1ayQJ1Oh5PuyKU/Hjx936TepeFrNhSnvnh07dqzU611rm+42puWhpH27nj7daDdLz64ee5uk6kH/N6+IdcprbNz988td6nPH8xB3GRtJys7OLvGyHhks+vbt6/R+zpw5Cg8PdwoIo0aN0r333itJmjhxopo2baq9e/eqcePGOnTokPr27atmzZpJkurWrevYVkJCgt5//33l5eVp+/bt8vPzU79+/ZSUlKTu3bsrKSlJnTp1kiQdOnRIc+fO1aFDhxQdHe3Y74oVKzR37lxNnjxZknTp0iW9/fbbiouLc+znzjvvdDqG2bNnq1KlSlq3bp169uzpmP7AAw/osccekyS99NJLWrlypWbOnKm33367wLi8+eabatmypWO/+WMTExOj1NRUNWzYsMA6f/7znzVixAjH+6ysLMXExCg8PLxcvxXKZrMpPDzcbX6gUTwrPTt0zlvnL+aVePlAP29Vq1attCWWKU+s+Wrl3bPIyMhSr3etbbrbmJaH0vStLMe7LNxMPfvl2Od3ac/p/30r1NXKc2zc/fPLHepz1/MQdxgbSQoICCjxsh4ZLPbs2aMXXnhBycnJ+umnnxyXaA4dOqQmTZpIkpo3b+5YPioqSpKUmZmpxo0ba9iwYXriiSf01VdfqWvXrurbt69j+dtvv13Z2dnasmWLvv32W3Xq1EkJCQmaOnWqJGndunUaPXq0JGnbtm3Ky8srcMKem5ursLAwx3s/Pz+neiTp+PHjGjdunJKSkpSZmam8vDydO3dOhw4dclquffv2Bd4X9S1QW7du1dq1ax3h6pf27dtXaLDw9/eXv79/geleXl7l+sNls9nKfZ+w5np7dmdslJZvO1Kibwjy9rKpS5Mol/+98MSaC1PePSvNeiXd5s2opH0rq/EuCzdbz64ee6MroaKwYFHeY+Pun1/uUp87noe4y9iUZpvuM3ql0KtXL/3888967733lJycrOTkZEnSxYsXHcv4+vo6/px/SSg/gDz22GPav3+/Hn74YW3btk3x8fGaOXOmJKlSpUqKi4tTUlKS1q1bp4SEBN1xxx3asmWLUlNTtWfPHscVi5ycHHl7e2vz5s1KSUlxvHbu3KkZM2Y49h8YGFjgslRiYqJSUlI0Y8YMffvtt0pJSVFYWJjTMZRWTk6OevXq5VRLSkqK9uzZozvuuOO6twuUpf7t6pT4xCfPbtS/XZ0bXNG1eWLNZel6j78065V0myhaWY13WbjZeubOnxHuXJvk/vW5kieOjccFi5MnT2r37t0aN26cunTpotjYWJ06Vfqv14qJidGQIUO0ePFijRw5Uu+9955jXqdOnbR27Vp9/fXXSkhIUJUqVRQbG6tJkyYpKirK8S//LVu2VF5enjIzM1W/fn2nV2RkZLH7X79+vYYNG6Z77rlHTZs2lb+/f6EPWW/YsKHA+8Ker5CkVq1a6ccff1Tt2rUL1BMUFFTaIQJuiOYxldW7ZYyudQuozXbll6M1q1G5fAorhifWXJau9/hLul5ptomilcV4S1fGPKbK9T+scTP2zJ0/I9y5Nsn963MlTxwbjwsWlStXVlhYmGbPnq29e/dqzZo1Ts8IlMTw4cP15ZdfKi0tTd9//73Wrl3rdLKekJCgL7/8Uj4+PmrcuLFj2ocffui4WiFJDRs21IABA/TII49o8eLFSktL08aNGzVlyhQtW7as2BoaNGig+fPna+fOnUpOTtaAAQMUGBhYYLl//etfmjNnjlJTUzV+/Hht3LhRTz75ZKHbHDp0qH7++Wf1799fmzZt0r59+/Tll1/q0UcfVV6e+9x3i5ubzWbTxPtb6L4WMZJU4Kv08t/f1+LKbxJ1hwc/PbHmsnS9x3+t9Yrd5//999c6pjdCSfuUHxqK6+Pipzqrd8vi+3b11F/7z0Fxrh77q4fMlWPj7p9f7l6fK3ni2HjcMxZeXl5atGiRhg0bpltuuUWNGjXSG2+8oYSEhBJvIy8vT0OHDtXhw4cVGhqq7t276/XXX3fMv/3222W3251CREJCgmbMmFFgP3PnztXLL7+skSNH6siRI6pataratWvn9AB2Yd5//309/vjjatWqlWJiYjR58mSNGjWqwHITJ07UokWL9Mc//lFRUVFauHCh4zmSq0VHR2v9+vV69tlndffddys3N1e1atVS9+7d3eqeQcDX20uT+rbUQ7fW1sINaVq145jOX7ysQD8fdW0Sqf7t6qhZjcpu8SGZzxNrLkvXe/xFrRfg66342lVkJG1KO6ncy/+7Gz3A11t3NY3S737lY3ojlKRPt1SvpO1HTl+zj0Vtp0tsNbWtW1Ub005q9U32c1Cc/LHv17aWVmzcqYNnT+vcxTy3GBt3//xy9/pcydPGxmbc4fd/w21kZWWpYsWKOnPmTLl+K1RmZqYiIiIIQB7iRvTMGOM2H4wl5Uk1u1PPClsvf5onjWl5KIu+XWtMSzrmRS1Hz5z9smc2m81tx8bd+1ae9XnaeUh5964054buP3oAbgru/D+4onhizWXpeo+/sPV+efsUyta1xrSkY17UcvSsaO48Nu5cm+T+9bmSO48NwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEs4BbsdrurSwAAAIAFPq4uADcnu92u418s1sEP3tOFYxm6XLehUvenKiAySrUSB6taz9/Iy4vcCwAA4CkIFih3ly+c0+aH++rCkfQrE2z/CxC5GUeUOnWCDs1/X63nfyKfgAouqhIAAACl4bb/JDxw4ED16dPH5du4XhMmTFCLFi3KvZaEhAQNHz78hu/netntdudQUYQLR9K1+eG+3CIFAADgIco9WFx9wl2UGTNmaN68eTe8HpSv418svmaoyHfhSLqOL/v0xhYEAACAMuG2VywqVqyoSpUquboMlLGDH7xXyuVn36BKAAAAUJZKFSwSEhI0bNgwPfPMM6pSpYoiIyM1YcIEp2UOHTqk3r17Kzg4WKGhoXrwwQd1/PhxSdK8efM0ceJEbd26VTabTTabrcirElffOlSSfRdl2rRpioqKUlhYmIYOHapLly455s2fP1/x8fEKCQlRZGSkfve73ykzM9MxPykpSTabTatXr1Z8fLwqVKigDh06aPfu3U77mDp1qqpVq6aQkBANGjRIFy5cKLSWiRMnKjw8XKGhoRoyZIguXrzomJebm6thw4YpIiJCAQEBuu2227Rp0yan9detW6e2bdvK399fUVFRGjNmjC5fvlzksS9btkwVK1bUhx9+WOj83NxcZWVlOb2kK7cs3YjXhWMZMjavQl62QqdfyDh6w2rhZe1ljHF5Dbzo2c3wom+e96JnnveiZ8W/SqrUD29/8MEHGjFihJKTk/Xdd99p4MCB6tixo+666y7Z7XZHqFi3bp0uX76soUOHql+/fkpKSlK/fv20fft2rVixQqtWrZJ05cpEWey7KGvXrlVUVJTWrl2rvXv3ql+/fmrRooUGDx4sSbp06ZJeeuklNWrUSJmZmRoxYoQGDhyo5cuXO21n7NixevXVVxUeHq4hQ4boD3/4g9avXy9J+uc//6kJEyborbfe0m233ab58+frjTfeUN26dZ22sXr1agUEBCgpKUkHDhzQo48+qrCwME2aNEmS9Mwzz+iTTz7RBx98oFq1aumVV15Rt27dtHfvXlWpUkVHjhzRPffco4EDB+rvf/+7du3apcGDBysgIKDQkLVgwQINGTJECxYsUM+ePQsdnylTpmjixIkFpp84caLIcHS97Ha7LtdtWGC6sdmUF1ldkmQzpsD8Y8eO8Q1RbsZut+vMmTMyxtAbD0HPPBN98zz0zPPQs+JlZ2eXeFmbMYWcyRUhISFBeXl5+uabbxzT2rZtqzvvvFNTp07VypUr1aNHD6WlpSkmJkaStGPHDjVt2lQbN25UmzZtNGHCBH366adKSUkpdl8DBw7U6dOn9emnn5Zo30VtIykpSfv27ZO3t7ck6cEHH5SXl5cWLVpU6Dr//e9/1aZNG2VnZys4OFhJSUnq3LmzVq1apS5dukiSli9frnvvvVfnz59XQECAOnTooJYtW+qtt95ybKddu3a6cOGC4zgHDhyozz//XOnp6apQ4co3Hb3zzjsaPXq0zpw5o/Pnz6ty5cqaN2+efve730m6Enpq166t4cOHa/To0Ro7dqw++eQT7dy5UzabTZL09ttv69lnn9WZM2fk5eWlhIQEtWjRQg0aNNDYsWO1dOlSderUqchxzs3NVW5uruN9VlaWYmJidOrUKYWGhha53vX6+vYWBaYZm5cu120gn/17ZDMFU/Ed36SUeR2wxm6368SJEwoPD+dD2EPQM89E3zwPPfM89Kx4WVlZqly5ss6cOXPNc8NSX7Fo3ry50/uoqCjHrUM7d+5UTEyMI1RIUpMmTVSpUiXt3LlTbdq0Ke3uSrzvojRt2tQRKvLX2bZtm+P95s2bNWHCBG3dulWnTp1yXO45dOiQmjRpUui+o6KiJEmZmZmqWbOmdu7cqSFDhjjtt3379lq7dq3TtLi4OEeoyF8mJydH6enpOnPmjC5duqSOHTs65vv6+qpt27bauXOnpCvj2759e0eokKSOHTsqJydHhw8fVs2aNSVJH3/8sTIzM7V+/fprjrm/v7/8/f0LTPfy8rohP1wBkVHKzThSYLrNGNmMvUCw8I+uwQ+5m7LZbDfs7wluDHrmmeib56FnnoeeFa00Y1Lq0fP19XV6b7PZSnXvlRXXs+/i1jl79qy6deum0NBQffjhh9q0aZOWLFkiSU7PPly9nfwT+/I67tJq2bKlwsPDNWfOHJXiglS5qJU4uJTLP36DKgEAAEBZKtNYFhsbq/T0dKWn/+/rRHfs2KHTp087/vXfz89PeXl5Zbnb67Zr1y6dPHlSU6dO1e23367GjRtf8wpIYWJjY5WcnOw0bcOGDQWW27p1q86fP++0THBwsGJiYlSvXj35+fk5ntuQrtwKtWnTJsfYxcbG6rvvvnMKC+vXr1dISIhq1KjhmFavXj2tXbtWS5cu1VNPPVXq47mRqvX8jQKqx1x7QUkB1WNU7d4+N7YgAAAAlIkyDRZdu3ZVs2bNNGDAAH3//ffauHGjHnnkEXXq1Enx8fGSpNq1aystLU0pKSn66aefnO7vL281a9aUn5+fZs6cqf379+uzzz7TSy+9VOrtPP3005ozZ47mzp2r1NRUjR8/Xj/++GOB5S5evKhBgwZpx44dWr58ucaPH68nn3xSXl5eCgoK0hNPPKHRo0drxYoV2rFjhwYPHqxz585p0KBBkqQ//vGPSk9P11NPPaVdu3Zp6dKlGj9+vEaMGFHgMlXDhg21du1affLJJ271C/O8vLzUev4n1wwXAdVj1Hr+J1ySBAAA8BBletZms9m0dOlSVa5cWXfccYe6du2qunXr6qOPPnIs07dvX3Xv3l2dO3dWeHi4Fi5cWJYllEp4eLjmzZunf/3rX2rSpImmTp2qadOmlXo7/fr10/PPP69nnnlGrVu31sGDB/XEE08UWK5Lly5q0KCB7rjjDvXr10/33Xef07c5TZ06VX379tXDDz+sVq1aae/evfryyy9VuXJlSVL16tW1fPlybdy4UXFxcRoyZIgGDRqkcePGFVpXo0aNtGbNGi1cuFAjR44s9XHdKD4BFdTmo2Vq+OcX5R9dw2mef3QNNfzzi2rz0TL5BFQoYgsAAABwN6X6Vij8+mVlZalixYolevK/rNjtdh07dkyRkZFcofAQdrtdmZmZioiIoGcegp55JvrmeeiZ56FnxSvNuSGjB7fADzIAAIBn42wOAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAAAABgGcECAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFjm4+oC4F6MMZKkrKysctun3W5Xdna2AgIC5OVF1vUE9Mzz0DPPRN88Dz3zPPSsePnnhPnniMUhWMBJdna2JCkmJsbFlQAAAMBdZGdnq2LFisUuYzMliR+4adjtdh09elQhISGy2Wzlss+srCzFxMQoPT1doaGh5bJPWEPPPA8980z0zfPQM89Dz4pnjFF2draio6OveUWHKxZw4uXlpRo1arhk36GhofxAexh65nnomWeib56Hnnkeela0a12pyMeNZAAAAAAsI1gAAAAAsIxgAZfz9/fX+PHj5e/v7+pSUEL0zPPQM89E3zwPPfM89Kzs8PA2AAAAAMu4YgEAAADAMoIFAAAAAMsIFgAAAAAsI1gAAAAAsIxgAZd76623VLt2bQUEBOjWW2/Vxo0bXV0SijBlyhS1adNGISEhioiIUJ8+fbR7925Xl4VSmDp1qmw2m4YPH+7qUlCMI0eO6Pe//73CwsIUGBioZs2a6b///a+ry0IR8vLy9Pzzz6tOnToKDAxUvXr19NJLL4nvx3EvX3/9tXr16qXo6GjZbDZ9+umnTvONMXrhhRcUFRWlwMBAde3aVXv27HFNsR6KYAGX+uijjzRixAiNHz9e33//veLi4tStWzdlZma6ujQUYt26dRo6dKg2bNiglStX6tKlS7r77rt19uxZV5eGEti0aZPeffddNW/e3NWloBinTp1Sx44d5evrq3//+9/asWOHXn31VVWuXNnVpaEIf/nLXzRr1iy9+eab2rlzp/7yl7/olVde0cyZM11dGn7h7NmziouL01tvvVXo/FdeeUVvvPGG3nnnHSUnJysoKEjdunXThQsXyrlSz8XXzcKlbr31VrVp00ZvvvmmJMlutysmJkZPPfWUxowZ4+LqcC0nTpxQRESE1q1bpzvuuMPV5aAYOTk5atWqld5++229/PLLatGihaZPn+7qslCIMWPGaP369frmm29cXQpKqGfPnqpWrZref/99x7S+ffsqMDBQ//jHP1xYGYpis9m0ZMkS9enTR9KVqxXR0dEaOXKkRo0aJUk6c+aMqlWrpnnz5umhhx5yYbWegysWcJmLFy9q8+bN6tq1q2Oal5eXunbtqu+++86FlaGkzpw5I0mqUqWKiyvBtQwdOlT33nuv088b3NNnn32m+Ph4PfDAA4qIiFDLli313nvvubosFKNDhw5avXq1UlNTJUlbt27Vf/7zH/Xo0cPFlaGk0tLSdOzYMafPyIoVK+rWW2/lnKQUfFxdAG5eP/30k/Ly8lStWjWn6dWqVdOuXbtcVBVKym63a/jw4erYsaNuueUWV5eDYixatEjff/+9Nm3a5OpSUAL79+/XrFmzNGLECD333HPatGmThg0bJj8/PyUmJrq6PBRizJgxysrKUuPGjeXt7a28vDxNmjRJAwYMcHVpKKFjx45JUqHnJPnzcG0ECwDXZejQodq+fbv+85//uLoUFCM9PV1PP/20Vq5cqYCAAFeXgxKw2+2Kj4/X5MmTJUktW7bU9u3b9c477xAs3NQ///lPffjhh1qwYIGaNm2qlJQUDR8+XNHR0fQMNxVuhYLLVK1aVd7e3jp+/LjT9OPHjysyMtJFVaEknnzySX3xxRdau3atatSo4epyUIzNmzcrMzNTrVq1ko+Pj3x8fLRu3Tq98cYb8vHxUV5enqtLxFWioqLUpEkTp2mxsbE6dOiQiyrCtYwePVpjxozRQw89pGbNmunhhx/Wn/70J02ZMsXVpaGE8s87OCexhmABl/Hz81Pr1q21evVqxzS73a7Vq1erffv2LqwMRTHG6Mknn9SSJUu0Zs0a1alTx9Ul4Rq6dOmibdu2KSUlxfGKj4/XgAEDlJKSIm9vb1eXiKt07NixwNc4p6amqlatWi6qCNdy7tw5eXk5n1J5e3vLbre7qCKUVp06dRQZGel0TpKVlaXk5GTOSUqBW6HgUiNGjFBiYqLi4+PVtm1bTZ8+XWfPntWjjz7q6tJQiKFDh2rBggVaunSpQkJCHPedVqxYUYGBgS6uDoUJCQkp8AxMUFCQwsLCeDbGTf3pT39Shw4dNHnyZD344IPauHGjZs+erdmzZ7u6NBShV69emjRpkmrWrKmmTZtqy5Yteu211/SHP/zB1aXhF3JycrR3717H+7S0NKWkpKhKlSqqWbOmhg8frpdfflkNGjRQnTp19Pzzzys6OtrxzVEoAQO42MyZM03NmjWNn5+fadu2rdmwYYOrS0IRJBX6mjt3rqtLQyl06tTJPP30064uA8X4/PPPzS233GL8/f1N48aNzezZs11dEoqRlZVlnn76aVOzZk0TEBBg6tata8aOHWtyc3NdXRp+Ye3atYX+PywxMdEYY4zdbjfPP/+8qVatmvH39zddunQxu3fvdm3RHobfYwEAAADAMp6xAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYBnBAgAAAIBlBAsAAAAAlhEsAAAAAFhGsAAA/Crt2rVL7dq1U0BAgFq0aFHi9ZKSkmSz2XT69OkyrWfgwIHq06dPmW4TANwJwQIAbrCBAwfKZrM5XmFhYerevbt++OEHp+Xy52/YsMFpem5ursLCwmSz2ZSUlCRJOnDggAYNGqQ6deooMDBQ9erV0/jx43Xx4kXHegcOHHDab1Hb/7UaP368goKCtHv3bq1evbrQZRISEjR8+PDyLewGye93SkqKq0sBcJMiWABAOejevbsyMjKUkZGh1atXy8fHRz179iywXExMjObOnes0bcmSJQoODnaatmvXLtntdr377rv68ccf9frrr+udd97Rc889V2Cbq1atcuw7IyNDrVu3LtuDs+DSpUs3bNv79u3Tbbfdplq1aiksLOyG7QcAcAXBAgDKgb+/vyIjIxUZGakWLVpozJgxSk9P14kTJ5yWS0xM1KJFi3T+/HnHtDlz5igxMdFpue7du2vu3Lm6++67VbduXd13330aNWqUFi9eXGDfYWFhjn1HRkbK19e3yDpPnTqlAQMGKDw8XIGBgWrQoIFT0Dl8+LD69++vKlWqKCgoSPHx8UpOTnbMnzVrlurVqyc/Pz81atRI8+fPd9q+zWbTrFmzdN999ykoKEiTJk2SJC1dulStWrVSQECA6tatq4kTJ+ry5ctF1mm32/Xiiy+qRo0a8vf3V4sWLbRixQqn/WzevFkvvviibDabJkyYUGAbAwcO1Lp16zRjxgzH1ZwDBw445m/evFnx8fGqUKGCOnTooN27dzutX9qa802cOFHh4eEKDQ3VkCFDnK4y2e12TZkyxXElKi4uTh9//LFjfnH9qVOnjiSpZcuWstlsSkhIKHT/+bd6LVu2TM2bN1dAQIDatWun7du3O5Y5efKk+vfvr+rVq6tChQpq1qyZFi5c6LSdjz/+WM2aNVNgYKDCwsLUtWtXnT171rGPtm3bKigoSJUqVVLHjh118OBBSVcCX+/evVWtWjUFBwerTZs2WrVqldO2MzIydO+99yowMFB16tTRggULVLt2bU2fPt2xzOnTp/XYY485xvLOO+/U1q1brzn+AG4gAwC4oRITE03v3r0d77Ozs83/+3//z9SvX9/k5eU5pksyS5YsMc2bNzfz5883xhhz8OBB4+/vb1JTU40ks3bt2iL3M3bsWNO6dWvH+7S0NCPJxMTEmPDwcNOxY0ezdOnSYmsdOnSoadGihdm0aZNJS0szK1euNJ999pmj7rp165rbb7/dfPPNN2bPnj3mo48+Mt9++60xxpjFixcbX19f89Zbb5ndu3ebV1991Xh7e5s1a9Y4HWNERISZM2eO2bdvnzl48KD5+uuvTWhoqJk3b57Zt2+f+eqrr0zt2rXNhAkTiqzztddeM6GhoWbhwoVm165d5plnnjG+vr4mNTXVGGNMRkaGadq0qRk5cqTJyMgw2dnZBbZx+vRp0759ezN48GCTkZFhMjIyzOXLl83atWuNJHPrrbeapKQk8+OPP5rbb7/ddOjQwbHu9dScmJhogoODTb9+/cz27dvNF198YcLDw81zzz3nWObll182jRs3NitWrDD79u0zc+fONf7+/iYpKema/dm4caORZFatWmUyMjLMyZMnC60j//hiY2PNV199ZX744QfTs2dPU7t2bXPx4kVjjDGHDx82f/3rX82WLVvMvn37zBtvvGG8vb1NcnKyMcaYo0ePGh8fH/Paa6+ZtLQ088MPP5i33nrLZGdnm0uXLpmKFSuaUaNGmb1795odO3aYefPmmYMHDxpjjElJSTHvvPOO2bZtm0lNTTXjxo0zAQEBjvnGGNO1a1fTokULs2HDBrN582bTqVMnExgYaF5//XWnZXr16mU2bdpkUlNTzciRI01YWFiRxw3gxiNYAMANlpiYaLy9vU1QUJAJCgoykkxUVJTZvHmz03L5wWL69Ommc+fOxhhjJk6caO6//35z6tSpYoPFnj17TGhoqJk9e7Zj2okTJ8yrr75qNmzYYDZu3GieffZZY7PZig0XvXr1Mo8++mih8959910TEhJS5Ilbhw4dzODBg52mPfDAA+aee+5xOsbhw4c7LdOlSxczefJkp2nz5883UVFRRdYZHR1tJk2a5DStTZs25o9//KPjfVxcnBk/fnyR2zDGmE6dOpmnn37aaVr+ifeqVasc05YtW2YkmfPnz193zYmJiaZKlSrm7NmzjmmzZs0ywcHBJi8vz1y4cMFUqFDBEdTyDRo0yPTv398YU3x/8oPkli1bij3m/ONbtGiRY9rJkydNYGCg+eijj4pc79577zUjR440xhizefNmI8kcOHCgwHInT540khxhqCSaNm1qZs6caYwxZufOnUaS2bRpk2P+nj17jCRHsPjmm29MaGiouXDhgtN26tWrZ959990S7xdA2fIp5wskAHBT6ty5s2bNmiXpyu0sb7/9tnr06KGNGzeqVq1aTsv+/ve/15gxY7R//37NmzdPb7zxRrHbPnLkiLp3764HHnhAgwcPdkyvWrWqRowY4Xjfpk0bHT16VH/961913333FbqtJ554Qn379tX333+vu+++W3369FGHDh0kSSkpKWrZsqWqVKlS6Lo7d+7U448/7jStY8eOmjFjhtO0+Ph4p/dbt27V+vXrHbdFSVJeXp4uXLigc+fOqUKFCk7LZ2Vl6ejRo+rYsWOBfZXlrTDNmzd3/DkqKkqSlJmZqZo1a5a65nxxcXFO89q3b6+cnBylp6crJydH586d01133eW0zsWLF9WyZUtJxfentNq3b+/4c5UqVdSoUSPt3LnTcSyTJ0/WP//5Tx05ckQXL15Ubm6uo/a4uDh16dJFzZo1U7du3XT33Xfrt7/9rSpXrqwqVapo4MCB6tatm+666y517dpVDz74oGMMc3JyNGHCBC1btkwZGRm6fPmyzp8/r0OHDkmSdu/eLR8fH7Vq1cpRX/369VW5cmXH+61btyonJ6fAszPnz5/Xvn37rms8AFhHsACAchAUFKT69es73v/tb39TxYoV9d577+nll192WjYsLEw9e/bUoEGDdOHCBfXo0UPZ2dmFbvfo0aPq3LmzOnTooNmzZ1+zjltvvVUrV64scn6PHj108OBBLV++XCtXrlSXLl00dOhQTZs2TYGBgSU82uIFBQU5vc/JydHEiRP1m9/8psCyAQEBZbLP6/HLZ1FsNpukK89ASDem5pycHEnSsmXLVL16dad5/v7+korvT1n661//qhkzZmj69Olq1qyZgoKCNHz4cMfzIN7e3lq5cqW+/fZbffXVV5o5c6bGjh2r5ORk1alTR3PnztWwYcO0YsUKffTRRxo3bpxWrlypdu3aadSoUVq5cqWmTZum+vXrKzAwUL/97W+dnjW5lpycHEVFRTm+Je2XKlWqVEajAKC0eHgbAFzAZrPJy8vL6SHtX/rDH/6gpKQkPfLII/L29i50mSNHjighIUGtW7fW3Llz5eV17Y/0lJQUx78cFyU8PFyJiYn6xz/+oenTpzsCS/PmzZWSkqKff/650PViY2O1fv16p2nr169XkyZNit1fq1attHv3btWvX7/Aq7BjCg0NVXR09HXt62p+fn7Ky8sr1TrXU3O+rVu3OvV8w4YNCg4OVkxMjJo0aSJ/f38dOnSowDZjYmIc6xTVHz8/P0kq8fH88muHT506pdTUVMXGxkq6Mpa9e/fW73//e8XFxalu3bpKTU11Wt9ms6ljx46aOHGitmzZIj8/Py1ZssQxv2XLlvrzn/+sb7/9VrfccosWLFjg2PbAgQN1//33q1mzZoqMjHR6aL5Ro0a6fPmytmzZ4pi2d+9enTp1yvG+VatWOnbsmHx8fAqMVdWqVUt0/ADKHlcsAKAc5Obm6tixY5KunMS9+eabysnJUa9evQpdvnv37jpx4oRCQ0MLnZ8fKmrVqqVp06Y5fbtUZGSkJOmDDz6Qn5+f4zaaxYsXa86cOfrb3/5WZJ0vvPCCWrduraZNmyo3N1dffPGF42Szf//+mjx5svr06aMpU6YoKipKW7ZsUXR0tNq3b6/Ro0frwQcfVMuWLdW1a1d9/vnnWrx4cYFv/Clsnz179lTNmjX129/+Vl5eXtq6dau2b99e4GpOvtGjR2v8+PGqV6+eWrRooblz5yolJUUffvhhsfu6Wu3atZWcnKwDBw4oODi4yNu8yqJm6cptTYMGDdK4ceN04MABjR8/Xk8++aS8vLwUEhKiUaNG6U9/+pPsdrtuu+02nTlzRuvXr1doaKgSExOL7U9ERIQCAwO1YsUK1ahRQwEBAapYsWKRtbz44osKCwtTtWrVNHbsWFWtWtXxC/waNGigjz/+WN9++60qV66s1157TcePH3cEt+TkZK1evVp33323IiIilJycrBMnTig2NlZpaWmaPXu27rvvPkVHR2v37t3as2ePHnnkEce2Fy9erF69eslms+n55593XAmSpMaNG6tr1656/PHHNWvWLPn6+mrkyJEKDAx0XDnq2rWr2rdvrz59+uiVV15Rw4YNdfToUS1btkz3339/gdvtAJQTVz/kAQC/domJiUaS4xUSEmLatGljPv74Y6fl9H8Pbxfm6oe3586d67TNX77yzZs3z8TGxpoKFSqY0NBQ07ZtW/Ovf/2r2FpfeuklExsbawIDA02VKlVM7969zf79+x3zDxw4YPr27WtCQ0NNhQoVTHx8vOObgowx5u233zZ169Y1vr6+pmHDhubvf/97iY5xxYoVpkOHDiYwMNBR6y8fRL9aXl6emTBhgqlevbrx9fU1cXFx5t///rfTMiV5eHv37t2mXbt2JjAw0EgyaWlpjoebT5065Vhuy5YtjvnXW3P+t4O98MILJiwszAQHB5vBgwc7PYBst9vN9OnTTaNGjYyvr68JDw833bp1M+vWrTPGXLs/7733nomJiTFeXl6mU6dOhdaRf3yff/65adq0qfHz8zNt27Y1W7dudSxz8uRJ07t3bxMcHGwiIiLMuHHjzCOPPOL4drMdO3aYbt26mfDwcOPv728aNmzoePj62LFjpk+fPiYqKsr4+fmZWrVqmRdeeMHxDWhpaWmmc+fOJjAw0MTExJg333yzwEP0R48eNT169DD+/v6mVq1aZsGCBSYiIsK88847jmWysrLMU089ZaKjo42vr6+JiYkxAwYMMIcOHSqyBwBuLJsxxpR/nAEAAK6QlJSkzp0769SpUx7zPMLhw4cVExOjVatWqUuXLq4uB0ARuBUKAAC4lTVr1ignJ0fNmjVTRkaGnnnmGdWuXVt33HGHq0sDUAyCBQAAcCuXLl3Sc889p/379yskJEQdOnTQhx9+WOxvjQfgetwKBQAAAMAyvm4WAAAAgGUECwAAAACWESwAAAAAWEawAAAAAGAZwQIAAACAZQQLAAAAAJYRLAAAAABYRrAAAAAAYNn/B1kytC8OA/PqAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x280 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "answerable: min 2.36 median 6.37 | unanswerable: max 0.00\n"
     ]
    }
   ],
   "source": [
    "top_scores = {qid: search(bm25, question, k=1)[0][1] for qid, question, gold, _ in QUESTIONS}\n",
    "ans_scores = [top_scores[q[0]] for q in QUESTIONS if q[2]]\n",
    "no_scores = [top_scores[q[0]] for q in QUESTIONS if not q[2]]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 2.8))\n",
    "ax.scatter(ans_scores, np.ones(len(ans_scores)), s=70, label=\"answerable\", color=\"#2a7ab0\")\n",
    "ax.scatter(no_scores, np.zeros(len(no_scores)), s=70, label=\"not in the handbook\", color=\"#c0392b\")\n",
    "ax.set_yticks([0, 1], [\"not in handbook\", \"answerable\"]); ax.set_xlabel(\"BM25 score of the best passage\")\n",
    "ax.set_ylim(-0.6, 1.6); ax.grid(alpha=.3); plt.tight_layout(); plt.show()\n",
    "print(\"answerable: min %.2f median %.2f | unanswerable: max %.2f\" % (min(ans_scores), np.median(ans_scores), max(no_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4f0ca2c8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:52.248622Z",
     "iopub.status.busy": "2026-11-17T21:17:52.248305Z",
     "iopub.status.idle": "2026-11-17T21:17:52.260086Z",
     "shell.execute_reply": "2026-11-17T21:17:52.258725Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q01 What wave height stops the crew boats from sailing?\n",
      "    -> Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. [1]   (source: p02, Safety / Weather limits)\n",
      "\n",
      "q02 How often is the gearbox oil sampled?\n",
      "    -> Gearbox oil is sampled every six months. [1]   (source: p07, Maintenance / Gearbox oil)\n",
      "\n",
      "q03 Who may work alone inside a nacelle?\n",
      "    -> Lone working inside a nacelle is forbidden. [1]   (source: p03, Safety / Lone working)\n",
      "\n",
      "q04 What is the rotor diameter of a turbine?\n",
      "    -> Each of the 40 turbines is rated at 8.0 MW with a rotor diameter of 164 m. [1]   (source: p04, Turbines / Specification)\n",
      "\n",
      "q05 What do I do if the pitch fault comes back after a reset?\n",
      "    -> A pitch system fault (code PF-203) usually means a low accumulator pressure. [1]   (source: p10, Faults / Pitch system)\n",
      "\n",
      "q06 When must a Level 3 alarm be attended on site?\n",
      "    -> Level 3 stops the turbine and needs a site visit within 24 hours when the weather allows. [1]   (source: p09, Faults / Alarm levels)\n",
      "\n",
      "q07 How soon must an injury be reported?\n",
      "    -> Any injury or near miss is reported within one hour and a written report is filed within 24 hours. [1]   (source: p19, Reporting / Incidents)\n",
      "\n",
      "q08 What is the largest amount of power the farm is allowed to send ashore?\n",
      "    -> Used oil, filters and composite offcuts are brought ashore and logged in the waste register. [1]   (source: p15, Environment / Waste)\n",
      "\n",
      "q09 Which turbines are switched off at night in the autumn to protect birds?\n",
      "    -> Between October and March, turbines HV01 to HV08 are curtailed at night during the peak migration periods announced by the ornithology team. [1]   (source: p14, Environment / Bird collision)\n",
      "\n",
      "q10 How many spare wind vanes are kept in stock?\n",
      "    -> The harbour warehouse keeps the critical spares: two pitch accumulators, four wind vanes and a spare converter module. [1]   (source: p16, Logistics / Spare parts)\n",
      "\n",
      "q11 What qualification do I need before getting on the boat?\n",
      "    -> New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer. [1]   (source: p22, Training / Induction)\n",
      "\n",
      "q12 When can the turbines restart after a blackout?\n",
      "    -> On loss of grid connection the turbines stop and the converters isolate. [1]   (source: p12, Faults / Grid loss)\n",
      "\n",
      "q13 When can a helicopter be used?\n",
      "    -> A helicopter is used only for medical evacuation and for urgent Level 3 visits when vessels cannot sail. [1]   (source: p17, Logistics / Helicopter)\n",
      "\n",
      "q14 What happens when a dolphin is seen near the piling work?\n",
      "    -> Piling and heavy cable work pause when a marine mammal is sighted within 500 m. [1]   (source: p13, Environment / Marine mammals)\n",
      "\n",
      "q15 What is the price of electricity per MWh?\n",
      "    -> I cannot find that in the handbook.\n",
      "\n",
      "q16 Who is the chief executive of the operating company?\n",
      "    -> I cannot find that in the handbook.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "THRESHOLD = 1.0     # del gráfico de arriba: el hueco entre las puntuaciones sin respuesta (0) y la respuesta más débil\n",
    "\n",
    "def sentences(text):\n",
    "    return re.split(r\"(?<=[.!?])\\s+\", text)\n",
    "\n",
    "def answer(index, question, k=2, threshold=THRESHOLD):\n",
    "    hits = search(index, question, k)\n",
    "    if hits[0][1] < threshold:\n",
    "        return \"I cannot find that in the handbook.\", []\n",
    "    pid = hits[0][0]\n",
    "    section, text = chunks[ids.index(pid)].split(\". \", 1)\n",
    "    q_terms = set(tokens(question))\n",
    "    best = max(sentences(text), key=lambda s: len(q_terms & set(tokens(s))))\n",
    "    return f\"{best} [1]\", [(1, pid, section)]\n",
    "\n",
    "for qid, question, gold, fact in QUESTIONS:\n",
    "    text, cites = answer(bm25, question)\n",
    "    cite = f\"   (source: {cites[0][1]}, {cites[0][2]})\" if cites else \"\"\n",
    "    print(f\"{qid} {question}\\n    -> {text}{cite}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aab30975",
   "metadata": {},
   "source": [
    "## 7. Calificar las respuestas y qué dicen los números\n",
    "\n",
    "Que la recuperación acierte no significa todavía que la respuesta sea correcta, así que califica también el resultado\n",
    "final. Una respuesta cuenta como **correcta** si contiene el dato requerido, como **rechazada correctamente** cuando la\n",
    "pregunta no tiene respuesta y el sistema lo dice, y como **incorrecta** en cualquier otro caso (responder una pregunta\n",
    "sin respuesta es el peor caso: una invención con aplomo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6eab37ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:52.264665Z",
     "iopub.status.busy": "2026-11-17T21:17:52.264166Z",
     "iopub.status.idle": "2026-11-17T21:17:52.276259Z",
     "shell.execute_reply": "2026-11-17T21:17:52.274176Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "with the threshold at 1.0\n",
      "  wrong: q05 -> A pitch system fault (code PF-203) usually means a low accumulator pressure. [1]\n",
      "  wrong: q08 -> Used oil, filters and composite offcuts are brought ashore and logged in the waste register. [1]\n",
      "  wrong: q11 -> New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer. [1]\n",
      "  wrong: q12 -> On loss of grid connection the turbines stop and the converters isolate. [1]\n",
      "correct                                    10\n",
      "wrong: retrieval missed the passage         2\n",
      "wrong: right passage, wrong sentence        2\n",
      "missed (refused but answerable)             0\n",
      "correctly refused                           2\n",
      "invented (answered, unanswerable)           0\n"
     ]
    }
   ],
   "source": [
    "def grade(index, threshold=THRESHOLD):\n",
    "    out = {\"correct\": 0, \"wrong: retrieval missed the passage\": 0, \"wrong: right passage, wrong sentence\": 0,\n",
    "           \"missed (refused but answerable)\": 0, \"correctly refused\": 0, \"invented (answered, unanswerable)\": 0}\n",
    "    for qid, question, gold, fact in QUESTIONS:\n",
    "        text, cites = answer(index, question, threshold=threshold)\n",
    "        refused = not cites\n",
    "        if gold:\n",
    "            if refused:\n",
    "                out[\"missed (refused but answerable)\"] += 1\n",
    "            elif fact in text:\n",
    "                out[\"correct\"] += 1\n",
    "            else:\n",
    "                out[\"wrong: retrieval missed the passage\" if cites[0][1] != gold else \"wrong: right passage, wrong sentence\"] += 1\n",
    "                print(f\"  wrong: {qid} -> {text}\")\n",
    "        else:\n",
    "            out[\"correctly refused\" if refused else \"invented (answered, unanswerable)\"] += 1\n",
    "    return out\n",
    "\n",
    "print(\"with the threshold at\", THRESHOLD)\n",
    "for k, v in grade(bm25).items():\n",
    "    print(f\"{k:42s}{v:3d}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af4e81c3",
   "metadata": {},
   "source": [
    "Los fallos se dividen en dos tipos, y distinguirlos es el primer hábito de la depuración de RAG. En *\"retrieval missed\n",
    "the passage\"* el generador nunca tuvo opción. En *\"right passage, wrong sentence\"* la recuperación hizo su trabajo y\n",
    "nuestro generador extractivo, muy tosco, eligió la frase equivocada del pasaje; un modelo de lenguaje lee el pasaje\n",
    "entero y no tiene este problema (tiene otros, y por eso lo calificamos).\n",
    "\n",
    "Tres cosas que llevarte, y cada una es un módulo de este curso:\n",
    "\n",
    "1. **La recuperación es el techo.** El generador no puede hacerlo mejor que los pasajes que recibe, y un modelo de\n",
    "   lenguaje no puede inventar un pasaje que falta sin alucinar. La mayor parte del esfuerzo de un sistema RAG real va a\n",
    "   la recuperación. *El módulo 02 sustituye la coincidencia de palabras por significado (embeddings) y muestra por qué\n",
    "   combinar ambos supera a cualquiera de los dos.*\n",
    "2. **Las respuestas correctas por la razón equivocada son invisibles sin una batería.** Solo encontramos estos fallos\n",
    "   porque teníamos preguntas con respuesta conocida. *El módulo 07 construye bien esa batería y califica con\n",
    "   expresiones regulares y con un segundo modelo.*\n",
    "3. **Alguien tiene que mirar los fallos.** Aquí los lees a mano. En producción el mismo bucle lo ejecuta un segundo\n",
    "   modelo, un *supervisor*, que juzga cada respuesta y convierte los fallos en arreglos. *El módulo 08 lo construye.*\n",
    "\n",
    "## Ejercicios\n",
    "\n",
    "1. Cambia `THRESHOLD` a 0 y a 4 y vuelve a ejecutar la calificación. ¿Qué errores aparecen a cada lado y cuál es peor\n",
    "   para un manual de seguridad?\n",
    "2. Añade un stemmer diminuto (quita una `s` o un `ing` finales) a `tokens` y repite la sección 5. ¿Qué fallos arregla y\n",
    "   cuáles no (pista: *qualification* frente a *certificate*)?\n",
    "3. Reescribe tres preguntas como las teclearía un compañero con prisa (`wave limit boats?`). ¿Cómo cambia hit@1? Esa es\n",
    "   la diferencia entre un conjunto de demostración y el tráfico real.\n",
    "4. Prueba `fixed_chunks(handbook, 200)` como troceado para la recuperación en lugar de las secciones. ¿Cuánto baja el\n",
    "   MRR y por qué la respuesta es peor para un lector incluso cuando el trozo correcto queda primero?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
