{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ddc9e9f5",
   "metadata": {},
   "source": [
    "# 07 · Evaluar un sistema RAG\n",
    "\n",
    "Hasta ahora juzgábamos los resultados leyéndolos. Eso no escala y oculta la propiedad más importante de un\n",
    "sistema RAG: **si está mejorando o empeorando**. Cada cambio que haces (un nuevo tamaño de fragmento, un modelo\n",
    "de embeddings, un prompt, un umbral) arregla unas preguntas y rompe otras sin avisar. Sin una batería fija de\n",
    "preguntas con respuesta conocida navegas a ojo.\n",
    "\n",
    "Este módulo construye esa batería como es debido, la califica automáticamente, muestra dónde la calificación\n",
    "automática es **errónea** (lo será), mide cuánta evidencia puede dar de verdad una batería de este tamaño y\n",
    "termina con una comparación antes y después sobre preguntas escritas *después* del arreglo.\n",
    "\n",
    "**Aprenderás**\n",
    "1. Qué contiene una buena batería de preguntas: cobertura, paráfrasis, identificadores exactos y **trampas**\n",
    "2. Referencias tomadas de la fuente, nunca del sistema que se evalúa\n",
    "3. Un calificador automático (datos requeridos, afirmaciones prohibidas, fuente correcta) y dos formas en que falla\n",
    "4. Comprobaciones de fidelidad y de citas basadas en reglas, y lo que no pueden ver\n",
    "5. Métricas de recuperación (hit@k, MRR) junto a métricas de respuesta (corrección, precisión y exhaustividad del rechazo)\n",
    "6. Intervalos de confianza por bootstrap, y cuántas preguntas necesitas para distinguir dos sistemas\n",
    "7. Una comparación antes y después en la que el segundo conjunto **no** se usó para construir el arreglo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0904a3c9",
   "metadata": {},
   "source": [
    "### Imagen: un cambio se juzga con una batería fija\n",
    "\n",
    "Léela de izquierda a derecha. Un cambio en el sistema pasa por la **misma** batería de preguntas con respuesta\n",
    "conocida. Cada pregunta cae en una de tres cajas (ahora correcta, sin cambios, ahora incorrecta) y el resultado\n",
    "es una puntuación **con un intervalo**, no un único número. Los números son los que produce este notebook más abajo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "86fe447e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:05.310267Z",
     "iopub.status.busy": "2026-11-17T21:17:05.309770Z",
     "iopub.status.idle": "2026-11-17T21:17:05.346946Z",
     "shell.execute_reply": "2026-11-17T21:17:05.345367Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 840 330\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"840\" height=\"330\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"420.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Un cambio, una batería fija, una puntuación con su incertidumbre</text><rect x=\"20\" y=\"105\" width=\"170\" height=\"90\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"105.0\" y=\"137.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Un cambio</text><text x=\"105.0\" y=\"154.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">(tamaño de fragmento,</text><text x=\"105.0\" y=\"171.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">modelo, prompt, umbral)</text><line x1=\"190\" y1=\"150\" x2=\"230\" y2=\"150\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"230\" y=\"95\" width=\"170\" height=\"110\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"315.0\" y=\"129.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Batería fija</text><text x=\"315.0\" y=\"146.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">40 preguntas</text><text x=\"315.0\" y=\"163.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">respuestas conocidas</text><text x=\"315.0\" y=\"180.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">tomadas de la fuente</text><line x1=\"400\" y1=\"130\" x2=\"440\" y2=\"80\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"400\" y1=\"150\" x2=\"440\" y2=\"150\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"400\" y1=\"170\" x2=\"440\" y2=\"225\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"440\" y=\"55\" width=\"160\" height=\"50\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"520.0\" y=\"84.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Ahora correctas</text><rect x=\"440\" y=\"125\" width=\"160\" height=\"50\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"520.0\" y=\"154.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Sin cambios</text><rect x=\"440\" y=\"200\" width=\"160\" height=\"55\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"520.0\" y=\"223.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Ahora incorrectas</text><text x=\"520.0\" y=\"240.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">(regresiones)</text><line x1=\"600\" y1=\"80\" x2=\"645\" y2=\"130\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"600\" y1=\"150\" x2=\"645\" y2=\"150\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"600\" y1=\"227\" x2=\"645\" y2=\"170\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"645\" y=\"95\" width=\"180\" height=\"110\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"735.0\" y=\"137.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Puntuación e intervalo</text><text x=\"735.0\" y=\"154.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">A 0.62 [0.47, 0.78]</text><text x=\"735.0\" y=\"171.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">B 0.70 [0.55, 0.85]</text><rect x=\"20\" y=\"275\" width=\"805\" height=\"45\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/><text x=\"30\" y=\"295\" font-size=\"12\" fill=\"#444\">B menos A es +0.07 con intervalo [-0.03, +0.20]: con 40 preguntas no se distingue un sistema del otro.</text><text x=\"30\" y=\"311\" font-size=\"12\" fill=\"#444\">Lección: el intervalo, no solo la puntuación, dice si el cambio es real.</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import SVG, display\n",
    "import html as _h\n",
    "\n",
    "_COL = {  # fill, stroke  (light cards with dark text: readable in light and dark themes)\n",
    "    \"gold\": (\"#fff4c2\", \"#b89d28\"), \"blue\": (\"#dbeafe\", \"#3b82f6\"), \"green\": (\"#dcfce7\", \"#16a34a\"),\n",
    "    \"red\": (\"#fee2e2\", \"#dc2626\"), \"grey\": (\"#f1f1f4\", \"#8a8aa0\"), \"violet\": (\"#ede9fe\", \"#7c3aed\"),\n",
    "}\n",
    "\n",
    "def diagram(width, height, items, title=None):\n",
    "    # items: (\"box\", x, y, w, h, \"text\\nmore text\", colour) | (\"arrow\", x1, y1, x2, y2, \"label\") |\n",
    "    #        (\"text\", x, y, \"text\", size, \"start|middle|end\") | (\"note\", x, y, w, h, \"text\")\n",
    "    out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 {width} {height}\" width=\"100%\" '\n",
    "           f'font-family=\"Helvetica, Arial, sans-serif\">',\n",
    "           '<defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" '\n",
    "           'orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs>',\n",
    "           f'<rect width=\"{width}\" height=\"{height}\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/>']\n",
    "    if title:\n",
    "        out.append(f'<text x=\"{width / 2}\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">{_h.escape(title)}</text>')\n",
    "    for it in items:\n",
    "        kind = it[0]\n",
    "        if kind == \"box\":\n",
    "            _, x, y, w, h, text, col = it\n",
    "            fill, stroke = _COL[col]\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"9\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"1.6\"/>')\n",
    "            lines = text.split(\"\\n\")\n",
    "            y0 = y + h / 2 - (len(lines) - 1) * 8.5 + 4.5\n",
    "            for k, ln in enumerate(lines):\n",
    "                bold = ' font-weight=\"700\"' if k == 0 and len(lines) > 1 else \"\"\n",
    "                out.append(f'<text x=\"{x + w / 2}\" y=\"{y0 + k * 17}\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\"{bold}>{_h.escape(ln)}</text>')\n",
    "        elif kind == \"arrow\":\n",
    "            _, x1, y1, x2, y2, label = it\n",
    "            out.append(f'<line x1=\"{x1}\" y1=\"{y1}\" x2=\"{x2}\" y2=\"{y2}\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/>')\n",
    "            if label:\n",
    "                out.append(f'<text x=\"{(x1 + x2) / 2}\" y=\"{(y1 + y2) / 2 - 6}\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">{_h.escape(label)}</text>')\n",
    "        elif kind == \"text\":\n",
    "            _, x, y, text, size, anchor = it\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x}\" y=\"{y + k * (size + 4)}\" text-anchor=\"{anchor}\" font-size=\"{size}\" fill=\"#333\">{_h.escape(ln)}</text>')\n",
    "        elif kind == \"note\":\n",
    "            _, x, y, w, h, text = it\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/>')\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x + 10}\" y=\"{y + 20 + k * 16}\" font-size=\"12\" fill=\"#444\">{_h.escape(ln)}</text>')\n",
    "    out.append(\"</svg>\")\n",
    "    display(SVG(\"\".join(out)))\n",
    "\n",
    "\n",
    "diagram(840, 330, [\n",
    "    (\"box\", 20, 105, 170, 90, \"Un cambio\\n(tamaño de fragmento,\\nmodelo, prompt, umbral)\", \"gold\"),\n",
    "    (\"arrow\", 190, 150, 230, 150, \"\"),\n",
    "    (\"box\", 230, 95, 170, 110, \"Batería fija\\n40 preguntas\\nrespuestas conocidas\\ntomadas de la fuente\", \"violet\"),\n",
    "    (\"arrow\", 400, 130, 440, 80, \"\"),\n",
    "    (\"arrow\", 400, 150, 440, 150, \"\"),\n",
    "    (\"arrow\", 400, 170, 440, 225, \"\"),\n",
    "    (\"box\", 440, 55, 160, 50, \"Ahora correctas\", \"green\"),\n",
    "    (\"box\", 440, 125, 160, 50, \"Sin cambios\", \"grey\"),\n",
    "    (\"box\", 440, 200, 160, 55, \"Ahora incorrectas\\n(regresiones)\", \"red\"),\n",
    "    (\"arrow\", 600, 80, 645, 130, \"\"),\n",
    "    (\"arrow\", 600, 150, 645, 150, \"\"),\n",
    "    (\"arrow\", 600, 227, 645, 170, \"\"),\n",
    "    (\"box\", 645, 95, 180, 110, \"Puntuación e intervalo\\nA 0.62 [0.47, 0.78]\\nB 0.70 [0.55, 0.85]\", \"violet\"),\n",
    "    (\"note\", 20, 275, 805, 45, \"B menos A es +0.07 con intervalo [-0.03, +0.20]: con 40 preguntas no se distingue un sistema del otro.\\nLección: el intervalo, no solo la puntuación, dice si el cambio es real.\"),\n",
    "], \"Un cambio, una batería fija, una puntuación con su incertidumbre\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d7d0d17",
   "metadata": {},
   "source": [
    "## 1. La batería\n",
    "\n",
    "Cuarenta preguntas en dos conjuntos sobre el manual de Harbourview. Cada una lleva:\n",
    "\n",
    "* el **pasaje correcto** que contiene la respuesta (o `None` cuando el manual no puede responder);\n",
    "* **datos requeridos** como expresiones regulares que la respuesta debe contener, y **afirmaciones prohibidas**\n",
    "  que no debe contener;\n",
    "* un **tipo**: `direct`, `paraphrase`, `identifier` o `trap`.\n",
    "\n",
    "El **conjunto 1** (20 preguntas) es el que miramos mientras construimos. El **conjunto 2** (20 preguntas) se\n",
    "escribe *después* y solo se usa para comprobar que un arreglo generaliza. Las **trampas** son preguntas que un\n",
    "sistema descuidado responde con seguridad y mal: una turbina que no existe (`HV99`) y un resultado que el manual\n",
    "nunca da (la disponibilidad del último mes, cuando solo indica un *objetivo*). Una batería sin trampas premia a\n",
    "los sistemas que nunca dicen \"no lo sé\".\n",
    "\n",
    "Las referencias salen de leer el manual (los datos que hay detrás del sistema), **nunca de ejecutar el\n",
    "sistema**. Una referencia copiada de la salida del sistema solo mide que el sistema coincide consigo mismo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ede9a04b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:05.351650Z",
     "iopub.status.busy": "2026-11-17T21:17:05.351245Z",
     "iopub.status.idle": "2026-11-17T21:17:05.362869Z",
     "shell.execute_reply": "2026-11-17T21:17:05.361145Z"
    }
   },
   "outputs": [],
   "source": [
    "# El manual de Harbourview: 22 pasajes, ficticio, escrito para este curso\n",
    "PASSAGES = [\n",
    "    [\"p01\", \"Safety / Access\", \"Technicians must hold a valid GWO Basic Safety Training certificate before boarding a crew transfer vessel. Certificates are checked at the harbour gate and expire after two years.\"],\n",
    "    [\"p02\", \"Safety / Weather limits\", \"Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later.\"],\n",
    "    [\"p03\", \"Safety / Lone working\", \"Lone working inside a nacelle is forbidden. Every task inside a turbine needs at least two people, and both must carry a personal locator beacon.\"],\n",
    "    [\"p04\", \"Turbines / Specification\", \"Each of the 40 turbines is rated at 8.0 MW with a rotor diameter of 164 m. Cut-in wind speed is 3 m/s, rated speed is 12.5 m/s and cut-out speed is 25 m/s.\"],\n",
    "    [\"p05\", \"Turbines / Naming\", \"Turbines are named HV01 to HV40, numbered from the western edge of the array. The offshore substation is OSS1, in the centre.\"],\n",
    "    [\"p06\", \"Maintenance / Schedule\", \"Every turbine gets a planned service once a year between April and September. The service takes two technicians about two days, plus weather delays.\"],\n",
    "    [\"p07\", \"Maintenance / Gearbox oil\", \"Gearbox oil is sampled every six months. If the particle count exceeds ISO 4406 code 18/16/13, a filter change is scheduled within 14 days.\"],\n",
    "    [\"p08\", \"Maintenance / Blade inspection\", \"Blades are inspected by drone every 12 months and after any lightning alarm. Leading-edge erosion above grade 3 is repaired in the next planned service.\"],\n",
    "    [\"p09\", \"Faults / Alarm levels\", \"Alarms have three levels. Level 1 is informational. Level 2 needs a remote check within 4 hours. Level 3 stops the turbine and needs a site visit within 24 hours when the weather allows.\"],\n",
    "    [\"p10\", \"Faults / Pitch system\", \"A pitch system fault (code PF-203) usually means a low accumulator pressure. Reset it remotely once. If it returns within 48 hours, send a crew to replace the hydraulic accumulator.\"],\n",
    "    [\"p11\", \"Faults / Yaw misalignment\", \"A yaw error above 8 degrees for more than 10 minutes raises alarm YW-110. Persistent errors point to a faulty wind vane: the vane is recalibrated, then replaced if the error remains.\"],\n",
    "    [\"p12\", \"Faults / Grid loss\", \"On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid operator confirms voltage within 5% of nominal for 10 minutes.\"],\n",
    "    [\"p13\", \"Environment / Marine mammals\", \"Piling and heavy cable work pause when a marine mammal is sighted within 500 m. Work resumes after 30 minutes without a sighting.\"],\n",
    "    [\"p14\", \"Environment / Bird collision\", \"Between October and March, turbines HV01 to HV08 are curtailed at night during the peak migration periods announced by the ornithology team.\"],\n",
    "    [\"p15\", \"Environment / Waste\", \"Used oil, filters and composite offcuts are brought ashore and logged in the waste register. Nothing is discharged at sea.\"],\n",
    "    [\"p16\", \"Logistics / Spare parts\", \"The harbour warehouse keeps the critical spares: two pitch accumulators, four wind vanes and a spare converter module. Reorder when stock falls below one unit.\"],\n",
    "    [\"p17\", \"Logistics / Helicopter\", \"A helicopter is used only for medical evacuation and for urgent Level 3 visits when vessels cannot sail. Booking goes through the duty coordinator.\"],\n",
    "    [\"p18\", \"Contacts / Duty coordinator\", \"The duty coordinator is reachable 24 hours a day on the control room line. Out-of-hours safety incidents are reported there first.\"],\n",
    "    [\"p19\", \"Reporting / Incidents\", \"Any injury or near miss is reported within one hour and a written report is filed within 24 hours. Environmental incidents follow the same deadlines.\"],\n",
    "    [\"p20\", \"Reporting / Performance\", \"Availability is reported monthly. The target is 97% time-based availability, counting weather downtime as available.\"],\n",
    "    [\"p21\", \"Grid / Export\", \"Power is exported at 66 kV through two cables to the onshore substation. The export limit agreed with the grid operator is 300 MW.\"],\n",
    "    [\"p22\", \"Training / Induction\", \"New staff complete a two-day induction covering the handbook, emergency drills and the sea survival course before their first transfer.\"],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d049a0d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:05.368023Z",
     "iopub.status.busy": "2026-11-17T21:17:05.367658Z",
     "iopub.status.idle": "2026-11-17T21:17:06.342084Z",
     "shell.execute_reply": "2026-11-17T21:17:06.340916Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40 questions; Counter({'paraphrase': 16, 'direct': 11, 'trap': 8, 'identifier': 5}) ; unanswerable: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patito/.local/lib/python3.12/site-packages/matplotlib/projections/__init__.py:63: UserWarning: Unable to import Axes3D. This may be due to multiple versions of Matplotlib being installed (e.g. as a system package and as a pip package). As a result, the 3D projection is not available.\n",
      "  warnings.warn(\"Unable to import Axes3D. This may be due to multiple versions of \"\n"
     ]
    }
   ],
   "source": [
    "import re, math, time, textwrap\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# (id, pregunta, pasaje correcto o None, patrones de datos requeridos, patrones prohibidos, tipo)\n",
    "SET1 = [\n",
    "    (\"s01\", \"What wave height stops the crew boats from sailing?\", \"p02\", [r\"1\\.5 ?m\"], [], \"direct\"),\n",
    "    (\"s02\", \"How often is the gearbox oil sampled?\", \"p07\", [r\"six months\"], [], \"direct\"),\n",
    "    (\"s03\", \"Who may work alone inside a nacelle?\", \"p03\", [r\"forbidden\"], [], \"paraphrase\"),\n",
    "    (\"s04\", \"What is the rotor diameter of a turbine?\", \"p04\", [r\"164 ?m\"], [], \"direct\"),\n",
    "    (\"s05\", \"What do I do if the pitch fault comes back after a reset?\", \"p10\", [r\"replace the hydraulic accumulator\"], [], \"direct\"),\n",
    "    (\"s06\", \"When must a Level 3 alarm be attended on site?\", \"p09\", [r\"24 hours\"], [], \"direct\"),\n",
    "    (\"s07\", \"How soon must an injury be reported?\", \"p19\", [r\"one hour\"], [], \"direct\"),\n",
    "    (\"s08\", \"What is the largest amount of power the farm is allowed to send ashore?\", \"p21\", [r\"300 ?MW\"], [r\"8\\.0 ?MW\"], \"paraphrase\"),\n",
    "    (\"s09\", \"Which turbines are switched off at night in the autumn to protect birds?\", \"p14\", [r\"HV01 to HV08\"], [], \"paraphrase\"),\n",
    "    (\"s10\", \"How many spare wind vanes are kept in stock?\", \"p16\", [r\"four wind vanes\"], [], \"direct\"),\n",
    "    (\"s11\", \"What qualification do I need before getting on the boat?\", \"p01\", [r\"GWO Basic Safety Training\"], [], \"paraphrase\"),\n",
    "    (\"s12\", \"When can the turbines restart after a blackout?\", \"p12\", [r\"10 minutes\"], [], \"paraphrase\"),\n",
    "    (\"s13\", \"When can a helicopter be used?\", \"p17\", [r\"medical evacuation\"], [], \"direct\"),\n",
    "    (\"s14\", \"What happens when a dolphin is seen near the piling work?\", \"p13\", [r\"pause\"], [], \"paraphrase\"),\n",
    "    (\"s15\", \"What is the price of electricity per MWh?\", None, [], [], \"trap\"),\n",
    "    (\"s16\", \"Who is the chief executive of the operating company?\", None, [], [], \"trap\"),\n",
    "    (\"s17\", \"What is the rated power of turbine HV99?\", None, [], [], \"trap\"),\n",
    "    (\"s18\", \"What availability did the farm achieve last month?\", None, [], [], \"trap\"),\n",
    "    (\"s19\", \"How many turbines does the farm have?\", \"p04\", [r\"40 turbines\"], [], \"direct\"),\n",
    "    (\"s20\", \"What is the cut-in wind speed?\", \"p04\", [r\"3 ?m/s\"], [], \"direct\"),\n",
    "]\n",
    "SET2 = [\n",
    "    (\"t01\", \"Are boats cancelled when the sea is too rough?\", \"p02\", [r\"1\\.5 ?m\"], [], \"paraphrase\"),\n",
    "    (\"t02\", \"Can I go up the tower alone to fix something?\", \"p03\", [r\"forbidden\"], [], \"paraphrase\"),\n",
    "    (\"t03\", \"How big are the blades?\", \"p04\", [r\"164 ?m\"], [], \"paraphrase\"),\n",
    "    (\"t04\", \"Who do I call in the middle of the night about an accident?\", \"p18\", [r\"control room line\"], [], \"paraphrase\"),\n",
    "    (\"t05\", \"What if the oil in the gearbox gets dirty?\", \"p07\", [r\"filter change\"], [], \"paraphrase\"),\n",
    "    (\"t06\", \"Do you stop work when whales or seals swim close?\", \"p13\", [r\"pause\"], [], \"paraphrase\"),\n",
    "    (\"t07\", \"How much spare equipment is kept on shore?\", \"p16\", [r\"two pitch accumulators\"], [], \"paraphrase\"),\n",
    "    (\"t08\", \"Is the farm allowed to produce at full output?\", \"p21\", [r\"300 ?MW\"], [], \"paraphrase\"),\n",
    "    (\"t09\", \"What should new colleagues do before their first trip?\", \"p22\", [r\"induction\"], [], \"paraphrase\"),\n",
    "    (\"t10\", \"How should rubbish and old oil be handled?\", \"p15\", [r\"brought ashore\"], [], \"paraphrase\"),\n",
    "    (\"t11\", \"What does code PF-203 mean?\", \"p10\", [r\"low accumulator pressure\"], [], \"identifier\"),\n",
    "    (\"t12\", \"alarm YW-110\", \"p11\", [r\"8 degrees\"], [], \"identifier\"),\n",
    "    (\"t13\", \"ISO 4406 18/16/13\", \"p07\", [r\"filter change\"], [], \"identifier\"),\n",
    "    (\"t14\", \"Where is OSS1?\", \"p05\", [r\"centre\"], [], \"identifier\"),\n",
    "    (\"t15\", \"66 kV cables\", \"p21\", [r\"two cables\"], [], \"identifier\"),\n",
    "    (\"t16\", \"What is the stock price of the operating company?\", None, [], [], \"trap\"),\n",
    "    (\"t17\", \"What is the rotor diameter of turbine HV77?\", None, [], [], \"trap\"),\n",
    "    (\"t18\", \"What was the monthly availability in July?\", None, [], [], \"trap\"),\n",
    "    (\"t19\", \"How many people work on the farm?\", None, [], [], \"trap\"),\n",
    "    (\"t20\", \"What is the cut-out wind speed?\", None if False else \"p04\", [r\"25 ?m/s\"], [], \"direct\"),\n",
    "]\n",
    "BATTERY = SET1 + SET2\n",
    "ids = [p[0] for p in PASSAGES]\n",
    "chunks = [f\"{section}. {text}\" for _, section, text in PASSAGES]\n",
    "from collections import Counter\n",
    "print(len(BATTERY), \"questions;\", Counter(q[5] for q in BATTERY), \"; unanswerable:\", sum(q[2] is None for q in BATTERY))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dfe737d",
   "metadata": {},
   "source": [
    "## 2. Dos sistemas para comparar\n",
    "\n",
    "Los sistemas evaluados son los de los módulos 01 y 02, pequeños para que la evaluación sea la protagonista:\n",
    "\n",
    "* **Sistema A**: recuperación con BM25.\n",
    "* **Sistema B**: recuperación híbrida, BM25 y embeddings MiniLM fusionados con Reciprocal Rank Fusion\n",
    "  ponderada (peso de BM25 0,25, el mejor ajuste que encontramos en el módulo 02).\n",
    "\n",
    "Ambos usan el mismo **generador extractivo** determinista (cita el pasaje principal y lo referencia como `[1]`)\n",
    "y se niegan a responder cuando la recuperación parece demasiado débil. Un sistema devuelve la respuesta, el\n",
    "pasaje que citó y si se negó."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "901eaaf5",
   "metadata": {},
   "source": [
    "### Imagen: cómo fluyen los datos en este módulo\n",
    "\n",
    "El manual es la fuente de las referencias. La batería alimenta a los dos sistemas, cada sistema devuelve una\n",
    "respuesta, el pasaje que citó y si rechazó, y el calificador v2 lo convierte en las métricas y en el intervalo\n",
    "que calcularás más abajo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b857ec14",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:06.348198Z",
     "iopub.status.busy": "2026-11-17T21:17:06.347444Z",
     "iopub.status.idle": "2026-11-17T21:17:06.362624Z",
     "shell.execute_reply": "2026-11-17T21:17:06.361036Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 840 430\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"840\" height=\"430\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"420.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Flujo de datos: de la batería al intervalo</text><rect x=\"20\" y=\"50\" width=\"170\" height=\"60\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"105.0\" y=\"76.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Manual</text><text x=\"105.0\" y=\"93.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">22 pasajes</text><line x1=\"105\" y1=\"110\" x2=\"105\" y2=\"165\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"105.0\" y=\"131.5\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">referencias</text><rect x=\"20\" y=\"165\" width=\"170\" height=\"100\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"105.0\" y=\"194.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Batería</text><text x=\"105.0\" y=\"211.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">conjunto 1: 20 preguntas</text><text x=\"105.0\" y=\"228.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">conjunto 2: 20 preguntas</text><text x=\"105.0\" y=\"245.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">referencia + datos</text><line x1=\"190\" y1=\"195\" x2=\"250\" y2=\"100\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"190\" y1=\"230\" x2=\"250\" y2=\"200\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"250\" y=\"60\" width=\"190\" height=\"70\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"345.0\" y=\"91.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Sistema A</text><text x=\"345.0\" y=\"108.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">BM25</text><rect x=\"250\" y=\"160\" width=\"190\" height=\"80\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"345.0\" y=\"187.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Sistema B</text><text x=\"345.0\" y=\"204.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">BM25 + MiniLM</text><text x=\"345.0\" y=\"221.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">RRF, peso de BM25 0.25</text><line x1=\"440\" y1=\"95\" x2=\"500\" y2=\"130\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"440\" y1=\"200\" x2=\"500\" y2=\"170\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"500\" y=\"105\" width=\"210\" height=\"80\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"605.0\" y=\"132.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Respuestas</text><text x=\"605.0\" y=\"149.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">texto, pasaje citado,</text><text x=\"605.0\" y=\"166.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">¿rechazó?</text><line x1=\"605\" y1=\"185\" x2=\"605\" y2=\"225\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"500\" y=\"225\" width=\"210\" height=\"85\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"605.0\" y=\"246.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Calificador v2</text><text x=\"605.0\" y=\"263.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">datos requeridos,</text><text x=\"605.0\" y=\"280.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">afirmaciones prohibidas,</text><text x=\"605.0\" y=\"297.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">fuente correcta</text><line x1=\"605\" y1=\"310\" x2=\"605\" y2=\"345\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"410\" y=\"345\" width=\"400\" height=\"80\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"610.0\" y=\"364.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Métricas en 40 preguntas</text><text x=\"610.0\" y=\"381.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">A: hit@1 0.72, correctas 0.62</text><text x=\"610.0\" y=\"398.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">B: hit@1 0.84, correctas 0.70</text><text x=\"610.0\" y=\"415.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">B menos A +0.07, intervalo 95% [-0.03, +0.20]</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diagram(840, 430, [\n",
    "    (\"box\", 20, 50, 170, 60, \"Manual\\n22 pasajes\", \"grey\"),\n",
    "    (\"arrow\", 105, 110, 105, 165, \"referencias\"),\n",
    "    (\"box\", 20, 165, 170, 100, \"Batería\\nconjunto 1: 20 preguntas\\nconjunto 2: 20 preguntas\\nreferencia + datos\", \"violet\"),\n",
    "    (\"arrow\", 190, 195, 250, 100, \"\"),\n",
    "    (\"arrow\", 190, 230, 250, 200, \"\"),\n",
    "    (\"box\", 250, 60, 190, 70, \"Sistema A\\nBM25\", \"blue\"),\n",
    "    (\"box\", 250, 160, 190, 80, \"Sistema B\\nBM25 + MiniLM\\nRRF, peso de BM25 0.25\", \"blue\"),\n",
    "    (\"arrow\", 440, 95, 500, 130, \"\"),\n",
    "    (\"arrow\", 440, 200, 500, 170, \"\"),\n",
    "    (\"box\", 500, 105, 210, 80, \"Respuestas\\ntexto, pasaje citado,\\n¿rechazó?\", \"gold\"),\n",
    "    (\"arrow\", 605, 185, 605, 225, \"\"),\n",
    "    (\"box\", 500, 225, 210, 85, \"Calificador v2\\ndatos requeridos,\\nafirmaciones prohibidas,\\nfuente correcta\", \"violet\"),\n",
    "    (\"arrow\", 605, 310, 605, 345, \"\"),\n",
    "    (\"box\", 410, 345, 400, 80, \"Métricas en 40 preguntas\\nA: hit@1 0.72, correctas 0.62\\nB: hit@1 0.84, correctas 0.70\\nB menos A +0.07, intervalo 95% [-0.03, +0.20]\", \"green\"),\n",
    "], \"Flujo de datos: de la batería al intervalo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b2539bbc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:06.367782Z",
     "iopub.status.busy": "2026-11-17T21:17:06.367337Z",
     "iopub.status.idle": "2026-11-17T21:17:24.383692Z",
     "shell.execute_reply": "2026-11-17T21:17:24.381618Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patito/.local/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights:   0%|          | 0/103 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights: 100%|██████████| 103/103 [00:00<00:00, 3065.25it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A p02 | Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later. [1]\n",
      "B p02 | Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m or the wind speed at the harbour exceeds 20 m/s. The duty coordinator decides at 06:00 and may cancel later. [1]\n"
     ]
    }
   ],
   "source": [
    "STOP = set(\"a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you\".split())\n",
    "def tokens(text):\n",
    "    return [w for w in re.findall(r\"[a-z0-9]+\", text.lower()) if w not in STOP]\n",
    "\n",
    "class BM25:\n",
    "    def __init__(self, docs, k1=1.5, b=0.75):\n",
    "        self.k1, self.b = k1, b\n",
    "        self.tf = [{} for _ in docs]\n",
    "        for i, d in enumerate(docs):\n",
    "            for t in tokens(d):\n",
    "                self.tf[i][t] = self.tf[i].get(t, 0) + 1\n",
    "        self.len = np.array([sum(tf.values()) for tf in self.tf], dtype=float); self.avg = self.len.mean()\n",
    "        df = {}\n",
    "        for tf in self.tf:\n",
    "            for t in tf:\n",
    "                df[t] = df.get(t, 0) + 1\n",
    "        self.idf = {t: math.log(1 + (len(docs) - c + 0.5) / (c + 0.5)) for t, c in df.items()}\n",
    "    def scores(self, query):\n",
    "        s = np.zeros(len(self.tf))\n",
    "        for t in set(tokens(query)):\n",
    "            if t in self.idf:\n",
    "                for i, tf in enumerate(self.tf):\n",
    "                    f = tf.get(t, 0)\n",
    "                    if f:\n",
    "                        s[i] += self.idf[t] * f * (self.k1 + 1) / (f + self.k1 * (1 - self.b + self.b * self.len[i] / self.avg))\n",
    "        return s\n",
    "\n",
    "from sentence_transformers import SentenceTransformer\n",
    "model = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\", device=\"cpu\")\n",
    "E = model.encode(chunks, normalize_embeddings=True)\n",
    "bm25 = BM25(chunks)\n",
    "sem = lambda q: E @ model.encode(q, normalize_embeddings=True)\n",
    "\n",
    "def rrf(scorers, weights, k=60):\n",
    "    def fused(q):\n",
    "        total = np.zeros(len(ids))\n",
    "        for sc, w in zip(scorers, weights):\n",
    "            for rank, i in enumerate(np.argsort(-sc(q)), start=1):\n",
    "                total[i] += w / (k + rank)\n",
    "        return total\n",
    "    return fused\n",
    "\n",
    "def quote(idx):\n",
    "    return chunks[idx].split(\". \", 1)[1]\n",
    "\n",
    "def build_system(scorer, refuse, expand=None, entity_guard=False):\n",
    "    def system(question):\n",
    "        query = expand(question) if expand else question\n",
    "        if entity_guard:\n",
    "            for hv in re.findall(r\"\\bHV(\\d+)\\b\", question):\n",
    "                if not 1 <= int(hv) <= 40:\n",
    "                    return dict(refused=True, cited=None, answer=\"That turbine does not exist in this farm.\", ranking=[])\n",
    "        s = scorer(query)\n",
    "        order = np.argsort(-s)\n",
    "        if refuse(question, query):\n",
    "            return dict(refused=True, cited=None, answer=\"I cannot find that in the handbook.\", ranking=[ids[i] for i in order])\n",
    "        top = int(order[0])\n",
    "        return dict(refused=False, cited=ids[top], answer=f\"{quote(top)} [1]\", ranking=[ids[i] for i in order])\n",
    "    return system\n",
    "\n",
    "A = build_system(bm25.scores, lambda q, e: bm25.scores(e).max() < 1.0)\n",
    "hybrid = rrf([bm25.scores, sem], [0.25, 1.0])\n",
    "B = build_system(hybrid, lambda q, e: bm25.scores(e).max() < 1.0 and sem(e).max() < 0.30)\n",
    "for name, sys_ in [(\"A\", A), (\"B\", B)]:\n",
    "    r = sys_(\"What wave height stops the crew boats from sailing?\")\n",
    "    print(name, r[\"cited\"], \"|\", r[\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356f56b9",
   "metadata": {},
   "source": [
    "## 3. El calificador automático, versión 1: expresiones regulares\n",
    "\n",
    "El calificador más barato no necesita ningún modelo: una respuesta a una pregunta respondible es **correcta**\n",
    "cuando contiene **todos los datos requeridos** y **ninguna afirmación prohibida**; una respuesta a una trampa es\n",
    "correcta cuando el sistema **se niega**. Es rápido, determinista y gratuito, y puedes ejecutarlo en cada commit.\n",
    "Sus límites son el resto de esta sección."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "17930c96",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:24.387752Z",
     "iopub.status.busy": "2026-11-17T21:17:24.386976Z",
     "iopub.status.idle": "2026-11-17T21:17:25.290486Z",
     "shell.execute_reply": "2026-11-17T21:17:25.288879Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grader v1, correct out of 40 : {'A': 24, 'B': 27}\n"
     ]
    }
   ],
   "source": [
    "# Patrones de la versión 1: escritos deprisa, como se suele hacer. Dos tienen fallos a propósito (ver sección 4).\n",
    "V1_OVERRIDES = {\n",
    "    \"s08\": ([r\"\\d+(\\.\\d+)? ?MW\"], []),        # \"algún número de MW\": débil, y sin afirmación prohibida\n",
    "    \"s12\": ([r\"10 min\\b\"], []),               # \\b tras \"min\": demasiado estricto, el texto dice \"minutes\"\n",
    "}\n",
    "\n",
    "def grade_v1(q, out):\n",
    "    qid, question, gold, req, forbid, kind = q\n",
    "    if gold is None:\n",
    "        return \"correct\" if out[\"refused\"] else \"wrong\"\n",
    "    if out[\"refused\"]:\n",
    "        return \"missed\"\n",
    "    req, forbid = V1_OVERRIDES.get(qid, (req, forbid))\n",
    "    ok = all(re.search(p, out[\"answer\"], re.I) for p in req) and not any(re.search(p, out[\"answer\"], re.I) for p in forbid)\n",
    "    return \"correct\" if ok else \"wrong\"\n",
    "\n",
    "def run_all(system, battery=BATTERY):\n",
    "    return {q[0]: system(q[1]) for q in battery}\n",
    "\n",
    "runs = {\"A\": run_all(A), \"B\": run_all(B)}\n",
    "def summary(grader, runs_, battery=BATTERY):\n",
    "    return {name: sum(grader(q, out[q[0]]) == \"correct\" for q in battery) for name, out in runs_.items()}\n",
    "\n",
    "print(\"grader v1, correct out of\", len(BATTERY), \":\", summary(grade_v1, runs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62512259",
   "metadata": {},
   "source": [
    "## 4. Cuando la calificación automática se equivoca\n",
    "\n",
    "Nunca te fíes de una calificación automática que no hayas auditado. Añadimos las dos comprobaciones que\n",
    "exigiría un revisor cuidadoso, una **comprobación de la fuente correcta** (el pasaje citado debe ser el\n",
    "correcto) y **patrones reparados**, llamamos a eso calificador v2, y después miramos cada respuesta en la que\n",
    "v1 y v2 **discrepan**. Esos son exactamente los lugares donde uno de los dos se equivoca, y leerlos nos dice\n",
    "cuál."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d824c05a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.293858Z",
     "iopub.status.busy": "2026-11-17T21:17:25.293529Z",
     "iopub.status.idle": "2026-11-17T21:17:25.305085Z",
     "shell.execute_reply": "2026-11-17T21:17:25.303560Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[A] s12 v1=wrong    v2=correct  cited=p12 gold=p12\n",
      "     Q: When can the turbines restart after a blackout?\n",
      "     A: On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid\n",
      "[B] s12 v1=wrong    v2=correct  cited=p12 gold=p12\n",
      "     Q: When can the turbines restart after a blackout?\n",
      "     A: On loss of grid connection the turbines stop and the converters isolate. Restart is allowed only when the grid\n",
      "2 disagreements\n"
     ]
    }
   ],
   "source": [
    "def grade_v2(q, out):\n",
    "    qid, question, gold, req, forbid, kind = q\n",
    "    if gold is None:\n",
    "        return \"correct\" if out[\"refused\"] else \"wrong\"\n",
    "    if out[\"refused\"]:\n",
    "        return \"missed\"\n",
    "    ok = (out[\"cited\"] == gold\n",
    "          and all(re.search(p, out[\"answer\"], re.I) for p in req)\n",
    "          and not any(re.search(p, out[\"answer\"], re.I) for p in forbid))\n",
    "    return \"correct\" if ok else \"wrong\"\n",
    "\n",
    "disputes = []\n",
    "for name, out in runs.items():\n",
    "    for q in BATTERY:\n",
    "        g1, g2 = grade_v1(q, out[q[0]]), grade_v2(q, out[q[0]])\n",
    "        if g1 != g2:\n",
    "            disputes.append((name, q[0], g1, g2))\n",
    "            print(f\"[{name}] {q[0]} v1={g1:8s} v2={g2:8s} cited={out[q[0]]['cited']} gold={q[2]}\\n     Q: {q[1]}\\n     A: {out[q[0]]['answer'][:110]}\")\n",
    "print(len(disputes), \"disagreements\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4214aea6",
   "metadata": {},
   "source": [
    "Leer las discrepancias es la **revisión**: una persona (o, en el módulo 08, un segundo modelo) lee cada\n",
    "respuesta en disputa frente a la referencia y escribe un veredicto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2a9f1eda",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.308285Z",
     "iopub.status.busy": "2026-11-17T21:17:25.307980Z",
     "iopub.status.idle": "2026-11-17T21:17:25.322691Z",
     "shell.execute_reply": "2026-11-17T21:17:25.320712Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1 agrees with the review on 0/2 disputed answers\n",
      "v2 agrees with the review on 2/2 disputed answers\n",
      "known case                                          expected        v1        v2\n",
      "right answer, right passage (s12)                    correct     wrong   correct   <- v1 wrong\n",
      "turbine rating given as the export limit (s08)         wrong   correct     wrong   <- v1 wrong\n",
      "right answer, right passage (s08)                    correct   correct   correct\n",
      "a refusal to an answerable question (s08)             missed    missed    missed\n"
     ]
    }
   ],
   "source": [
    "# La revisión escrita: veredictos decididos leyendo cada respuesta en disputa frente al manual, no ejecutando un calificador.\n",
    "REVIEW = {\n",
    "    (\"A\", \"s12\"): \"correct\",    # el pasaje citado dice que el reinicio exige 10 minutos de tensión estable: respuesta correcta, patrón v1 demasiado estricto\n",
    "    (\"B\", \"s12\"): \"correct\",\n",
    "}\n",
    "def agreement(grader):\n",
    "    agree = 0\n",
    "    for (name, qid), verdict in REVIEW.items():\n",
    "        q = next(x for x in BATTERY if x[0] == qid)\n",
    "        agree += grader(q, runs[name][qid]) == verdict\n",
    "    return agree, len(REVIEW)\n",
    "print(\"v1 agrees with the review on %d/%d disputed answers\" % agreement(grade_v1))\n",
    "print(\"v2 agrees with the review on %d/%d disputed answers\" % agreement(grade_v2))\n",
    "\n",
    "# Las ejecuciones reales solo mostraron un fallo. Prueba el propio calificador con salidas cuyo veredicto conocemos de antemano,\n",
    "# incluido el caso del patrón débil: la potencia nominal de la turbina citada como si fuera el límite de exportación.\n",
    "q08 = next(x for x in BATTERY if x[0] == \"s08\")\n",
    "q12 = next(x for x in BATTERY if x[0] == \"s12\")\n",
    "KNOWN = [\n",
    "    (\"right answer, right passage (s12)\", q12, dict(refused=False, cited=\"p12\", answer=chunks[ids.index(\"p12\")].split(\". \", 1)[1]), \"correct\"),\n",
    "    (\"turbine rating given as the export limit (s08)\", q08, dict(refused=False, cited=\"p04\", answer=chunks[ids.index(\"p04\")].split(\". \", 1)[1]), \"wrong\"),\n",
    "    (\"right answer, right passage (s08)\", q08, dict(refused=False, cited=\"p21\", answer=chunks[ids.index(\"p21\")].split(\". \", 1)[1]), \"correct\"),\n",
    "    (\"a refusal to an answerable question (s08)\", q08, dict(refused=True, cited=None, answer=\"I cannot find that.\"), \"missed\"),\n",
    "]\n",
    "print(f\"{'known case':50s}{'expected':>10s}{'v1':>10s}{'v2':>10s}\")\n",
    "for label, q, out, expected in KNOWN:\n",
    "    g1, g2 = grade_v1(q, out), grade_v2(q, out)\n",
    "    print(f\"{label:50s}{expected:>10s}{g1:>10s}{g2:>10s}\" + (\"   <- v1 wrong\" if g1 != expected else \"\") + (\"   <- v2 wrong\" if g2 != expected else \"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7d79fd",
   "metadata": {},
   "source": [
    "Dos lecciones, una de cada mitad de esa salida.\n",
    "\n",
    "* **Un patrón demasiado estricto que suspende una respuesta correcta.** `10 min\\b` no coincide con\n",
    "  `10 minutes`, así que v1 marcó como errónea la respuesta correcta sobre el reinicio en **ambos** sistemas (la\n",
    "  única discrepancia real, que una revisión resuelve).\n",
    "* **Un patrón débil que aprueba una respuesta errónea.** v1 solo pide \"algún número de MW\" para la pregunta del\n",
    "  límite de exportación, así que una respuesta que cita la potencia nominal de **8.0 MW** de la turbina,\n",
    "  tomada del pasaje equivocado, se califica como correcta. Ninguno de nuestros dos sistemas cometió ese error,\n",
    "  por lo que las ejecuciones reales nunca lo mostraron: por eso hay que **probar el calificador con casos\n",
    "  conocidos**, incluidos los erróneos. Es el mismo fallo que aceptar \"la eficiencia no supera el límite físico\"\n",
    "  porque contenía las palabras correctas: **los datos requeridos por sí solos no pueden detectar una\n",
    "  afirmación imposible o fuera de tema**. La lista de afirmaciones prohibidas y la comprobación de la fuente\n",
    "  correcta existen para esto."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45ff0231",
   "metadata": {},
   "source": [
    "En una auditoría real la revisión es **independiente** (alguien lee las respuestas sin mirar ninguno de los\n",
    "calificadores) y son los veredictos del *revisor* los que sirven de referencia. En la evaluación real del\n",
    "parque eólico que hizo este portafolio, la calificación por expresiones regulares se equivocó **dos veces**, y\n",
    "una vez la *propia referencia* estaba mal (la respuesta escrita nombraba la turbina equivocada; los datos decían\n",
    "otra cosa), así que se corrigió la referencia, no la respuesta. Una evaluación tiene errores como cualquier\n",
    "otro código: audita el calificador, los patrones **y** las referencias.\n",
    "\n",
    "A partir de aquí usamos el **calificador v2**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "921f9ff7",
   "metadata": {},
   "source": [
    "## 5. Comprobaciones de fidelidad y de citas, con reglas\n",
    "\n",
    "La corrección pregunta *¿es correcta la respuesta?*. La **fidelidad** pregunta *¿está respaldada cada\n",
    "afirmación por los pasajes que cita la respuesta?*, lo que importa incluso cuando la respuesta resulta ser\n",
    "correcta (una respuesta correcta de memoria es una alucinación a la espera). Dos comprobaciones basadas en\n",
    "reglas:\n",
    "\n",
    "* **Números**: todo número de la respuesta debe aparecer en el pasaje citado.\n",
    "* **Citas**: la respuesta debe llevar una cita, la cita debe apuntar a un pasaje recuperado y las palabras de\n",
    "  contenido de la respuesta deben aparecer en su mayoría en ese pasaje.\n",
    "\n",
    "Nuestro generador extractivo copia texto, así que es fiel por construcción. Para ver las comprobaciones en\n",
    "acción las ejecutamos sobre seis respuestas escritas a mano que imitan lo que hace un modelo de escritura libre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e17476e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.327646Z",
     "iopub.status.busy": "2026-11-17T21:17:25.327330Z",
     "iopub.status.idle": "2026-11-17T21:17:25.341759Z",
     "shell.execute_reply": "2026-11-17T21:17:25.339865Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "faithful, cited              {'numbers_ok': True, 'citation': 'ok'}\n",
      "invented number              {'numbers_ok': False, 'citation': 'unsupported (25% of words)'}\n",
      "wrong passage cited          {'numbers_ok': True, 'citation': 'unsupported (0% of words)'}\n",
      "no citation at all           {'numbers_ok': None, 'citation': 'missing'}\n",
      "faithful, two facts          {'numbers_ok': True, 'citation': 'ok'}\n",
      "right numbers, wrong claim   {'numbers_ok': True, 'citation': 'ok'}\n"
     ]
    }
   ],
   "source": [
    "NUM = re.compile(r\"\\d+(?:\\.\\d+)?\")\n",
    "P = {p[0]: p[2] for p in PASSAGES}\n",
    "\n",
    "def faithfulness(answer, retrieved):\n",
    "    cites = [int(c) for c in re.findall(r\"\\[(\\d+)\\]\", answer)]\n",
    "    body = re.sub(r\"\\s*\\[\\d+\\]\", \"\", answer)\n",
    "    if not cites:\n",
    "        return dict(numbers_ok=None, citation=\"missing\")\n",
    "    if any(c < 1 or c > len(retrieved) for c in cites):\n",
    "        return dict(numbers_ok=None, citation=\"points nowhere\")\n",
    "    cited_text = \" \".join(P[retrieved[c - 1]] for c in cites)\n",
    "    numbers_ok = all(n in NUM.findall(cited_text) for n in NUM.findall(body))\n",
    "    words = set(tokens(body))\n",
    "    support = len(words & set(tokens(cited_text))) / max(len(words), 1)\n",
    "    return dict(numbers_ok=numbers_ok, citation=\"ok\" if support >= 0.6 else f\"unsupported ({support:.0%} of words)\")\n",
    "\n",
    "SAMPLES = [\n",
    "    (\"faithful, cited\", \"Crew transfer vessels do not sail when the significant wave height exceeds 1.5 m [1].\", [\"p02\", \"p09\"]),\n",
    "    (\"invented number\", \"Vessels stop sailing when waves exceed 2.0 m [1].\", [\"p02\", \"p09\"]),\n",
    "    (\"wrong passage cited\", \"Gearbox oil is sampled every six months [2].\", [\"p07\", \"p19\"]),\n",
    "    (\"no citation at all\", \"Gearbox oil is sampled every six months.\", [\"p07\", \"p19\"]),\n",
    "    (\"faithful, two facts\", \"The export limit is 300 MW through two cables at 66 kV [1].\", [\"p21\", \"p04\"]),\n",
    "    (\"right numbers, wrong claim\", \"The rotor diameter is 8.0 m [1].\", [\"p04\", \"p05\"]),\n",
    "]\n",
    "for label, ans, retrieved in SAMPLES:\n",
    "    print(f\"{label:28s} {faithfulness(ans, retrieved)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b1dcf1",
   "metadata": {},
   "source": [
    "Las cinco primeras se comportan como se esperaba: un número inventado, una cita que no respalda la frase y una\n",
    "cita ausente se detectan sin llamar a ningún modelo. La última es el límite de las reglas: todos los números de\n",
    "*\"The rotor diameter is 8.0 m\"* existen en el pasaje citado (la potencia nominal de **8.0 MW** de la turbina),\n",
    "así que la comprobación de números deja pasar una afirmación errónea. Detectarlo requiere algo que\n",
    "**entienda** la frase, que es el trabajo del modelo juez del módulo 08.\n",
    "\n",
    "## 6. Métricas de recuperación y métricas de respuesta, lado a lado\n",
    "\n",
    "Informa siempre de ambas. Si baja la calidad de las respuestas, las métricas de recuperación te dicen si el\n",
    "pasaje correcto llegó siquiera a encontrarse.\n",
    "\n",
    "* **hit@1, hit@3, MRR** sobre las preguntas respondibles (como en el módulo 01).\n",
    "* **Corrección** según el calificador v2 sobre toda la batería.\n",
    "* **Precisión del rechazo**: cuando el sistema se niega, ¿con qué frecuencia debía hacerlo? **Exhaustividad del\n",
    "  rechazo**: de las preguntas que debía rechazar, ¿cuántas rechaza? Un sistema que lo rechaza todo tiene\n",
    "  precisión perfecta en las trampas y exhaustividad inútil en las preguntas reales; informa de ambas para que\n",
    "  ninguna pueda esconderse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b55f1407",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.345873Z",
     "iopub.status.busy": "2026-11-17T21:17:25.345520Z",
     "iopub.status.idle": "2026-11-17T21:17:25.362119Z",
     "shell.execute_reply": "2026-11-17T21:17:25.360161Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  system   hit@1   hit@3    MRR  correct  refusal prec  refusal rec\n",
      "       A    0.72    0.88   0.80     0.62          0.67         0.25\n",
      "       B    0.84    1.00   0.92     0.70          1.00         0.12\n",
      "\n",
      "[A] s08 wrong   (paraphrase) What is the largest amount of power the farm is allowed to sen\n",
      "[A] s11 wrong   (paraphrase) What qualification do I need before getting on the boat?\n",
      "[A] s17 wrong   (trap      ) What is the rated power of turbine HV99?\n",
      "[A] s18 wrong   (trap      ) What availability did the farm achieve last month?\n",
      "[A] s19 wrong   (direct    ) How many turbines does the farm have?\n",
      "[A] t01 wrong   (paraphrase) Are boats cancelled when the sea is too rough?\n",
      "[A] t02 missed  (paraphrase) Can I go up the tower alone to fix something?\n",
      "[A] t03 wrong   (paraphrase) How big are the blades?\n",
      "[A] t04 wrong   (paraphrase) Who do I call in the middle of the night about an accident?\n",
      "[A] t08 wrong   (paraphrase) Is the farm allowed to produce at full output?\n",
      "[A] t10 wrong   (paraphrase) How should rubbish and old oil be handled?\n",
      "[A] t16 wrong   (trap      ) What is the stock price of the operating company?\n",
      "[A] t17 wrong   (trap      ) What is the rotor diameter of turbine HV77?\n",
      "[A] t18 wrong   (trap      ) What was the monthly availability in July?\n",
      "[A] t19 wrong   (trap      ) How many people work on the farm?\n",
      "[B] s11 wrong   (paraphrase) What qualification do I need before getting on the boat?\n",
      "[B] s15 wrong   (trap      ) What is the price of electricity per MWh?\n",
      "[B] s17 wrong   (trap      ) What is the rated power of turbine HV99?\n",
      "[B] s18 wrong   (trap      ) What availability did the farm achieve last month?\n",
      "[B] t01 wrong   (paraphrase) Are boats cancelled when the sea is too rough?\n",
      "[B] t04 wrong   (paraphrase) Who do I call in the middle of the night about an accident?\n",
      "[B] t08 wrong   (paraphrase) Is the farm allowed to produce at full output?\n",
      "[B] t10 wrong   (paraphrase) How should rubbish and old oil be handled?\n",
      "[B] t16 wrong   (trap      ) What is the stock price of the operating company?\n",
      "[B] t17 wrong   (trap      ) What is the rotor diameter of turbine HV77?\n",
      "[B] t18 wrong   (trap      ) What was the monthly availability in July?\n",
      "[B] t19 wrong   (trap      ) How many people work on the farm?\n"
     ]
    }
   ],
   "source": [
    "def metrics(runs_, battery=BATTERY):\n",
    "    out = {}\n",
    "    for name, res in runs_.items():\n",
    "        ans = [q for q in battery if q[2]]\n",
    "        ranks = np.array([res[q[0]][\"ranking\"].index(q[2]) + 1 if res[q[0]][\"ranking\"] else 99 for q in ans])\n",
    "        correct = np.array([grade_v2(q, res[q[0]]) == \"correct\" for q in battery])\n",
    "        refused = np.array([res[q[0]][\"refused\"] for q in battery]); should = np.array([q[2] is None for q in battery])\n",
    "        out[name] = dict(hit1=(ranks == 1).mean(), hit3=(ranks <= 3).mean(), mrr=(1 / ranks).mean(), correct=correct.mean(),\n",
    "                         ref_prec=(refused & should).sum() / max(refused.sum(), 1), ref_rec=(refused & should).sum() / should.sum())\n",
    "    return out\n",
    "\n",
    "def show(m):\n",
    "    print(f\"{'system':>8s}{'hit@1':>8s}{'hit@3':>8s}{'MRR':>7s}{'correct':>9s}{'refusal prec':>14s}{'refusal rec':>13s}\")\n",
    "    for name, v in m.items():\n",
    "        print(f\"{name:>8s}{v['hit1']:8.2f}{v['hit3']:8.2f}{v['mrr']:7.2f}{v['correct']:9.2f}{v['ref_prec']:14.2f}{v['ref_rec']:13.2f}\")\n",
    "show(metrics(runs))\n",
    "print()\n",
    "for name, res in runs.items():\n",
    "    for q in BATTERY:\n",
    "        g = grade_v2(q, res[q[0]])\n",
    "        if g != \"correct\":\n",
    "            print(f\"[{name}] {q[0]} {g:7s} ({q[5]:10s}) {q[1][:62]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323af9d4",
   "metadata": {},
   "source": [
    "Mira la lista de fallos por **tipo** antes de mirar los promedios. Las trampas son donde ambos sistemas fallan\n",
    "igual: la recuperación siempre encuentra *algo* relacionado con \"turbine\" o \"availability\", y un sistema cuya\n",
    "única prueba de rechazo es \"¿se recuperó algo?\" no distingue un dato ausente de uno cercano. Es un defecto de\n",
    "diseño que ninguna mejora del ranking arregla (la sección 8 repara un tipo de trampa, la turbina inexistente, y\n",
    "muestra que las demás siguen fallando).\n",
    "\n",
    "## 7. ¿Cuántas preguntas hacen falta?\n",
    "\n",
    "Con 40 preguntas, una diferencia de dos o tres respuestas correctas parece un resultado. Normalmente es ruido.\n",
    "Haz un **bootstrap** de la batería: remuestrea las 40 preguntas con reemplazo muchas veces y recalcula la\n",
    "métrica para ver cuánto se mueve. Un bootstrap *pareado* hace lo mismo con la **diferencia** entre dos sistemas\n",
    "sobre las mismas preguntas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "833ad82c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.366695Z",
     "iopub.status.busy": "2026-11-17T21:17:25.366369Z",
     "iopub.status.idle": "2026-11-17T21:17:25.388821Z",
     "shell.execute_reply": "2026-11-17T21:17:25.387107Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "system A: correct 0.62, 95% interval [0.47, 0.78]\n",
      "system B: correct 0.70, 95% interval [0.55, 0.85]\n",
      "B minus A: +0.07, 95% interval [-0.03, +0.20]  -> NOT separable at this size\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng(0)\n",
    "cA = np.array([grade_v2(q, runs[\"A\"][q[0]]) == \"correct\" for q in BATTERY], dtype=float)\n",
    "cB = np.array([grade_v2(q, runs[\"B\"][q[0]]) == \"correct\" for q in BATTERY], dtype=float)\n",
    "\n",
    "def boot(x, n=5000):\n",
    "    idx = rng.integers(0, len(x), (n, len(x)))\n",
    "    return x[idx].mean(1)\n",
    "\n",
    "for name, c in [(\"A\", cA), (\"B\", cB)]:\n",
    "    lo, hi = np.percentile(boot(c), [2.5, 97.5])\n",
    "    print(f\"system {name}: correct {c.mean():.2f}, 95% interval [{lo:.2f}, {hi:.2f}]\")\n",
    "d = boot(cB - cA)\n",
    "lo, hi = np.percentile(d, [2.5, 97.5])\n",
    "print(f\"B minus A: {np.mean(cB - cA):+.2f}, 95% interval [{lo:+.2f}, {hi:+.2f}]  ->\", \"separable\" if lo > 0 or hi < 0 else \"NOT separable at this size\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbc69c6",
   "metadata": {},
   "source": [
    "Ahora la pregunta de planificación: *¿qué tamaño debe tener la batería para que una diferencia real sea\n",
    "detectable?* Lo simulamos. Supón que el sistema B es realmente mejor que A: en una pregunta al azar, A acierta y\n",
    "B falla el 5 % de las veces, A falla y B acierta el 12 % de las veces (así que B es unos 7 puntos mejor), y en\n",
    "el resto coinciden. Para cada tamaño de batería generamos muchas baterías y contamos con qué frecuencia una\n",
    "prueba exacta del signo sobre las preguntas discordantes detecta la diferencia al nivel del 5 %. Esa fracción\n",
    "es la **potencia**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "57b65046",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:25.393782Z",
     "iopub.status.busy": "2026-11-17T21:17:25.393462Z",
     "iopub.status.idle": "2026-11-17T21:17:32.011978Z",
     "shell.execute_reply": "2026-11-17T21:17:32.009901Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  20 questions: a true 7-point improvement is detected    1% of the time\n",
      "  40 questions: a true 7-point improvement is detected    9% of the time\n",
      " 100 questions: a true 7-point improvement is detected   34% of the time\n",
      " 200 questions: a true 7-point improvement is detected   63% of the time\n",
      " 400 questions: a true 7-point improvement is detected   90% of the time\n",
      " 800 questions: a true 7-point improvement is detected  100% of the time\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAEiCAYAAAAF9zFeAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAASFlJREFUeJzt3XlcVPX+P/DXzAAz7AgMqyO4C7igIIhZLmFort9b32tWbllmpZm0qP3Ksvu9l8osTC2v3crqZtmqpuaGomWW+4IKCIGgbAMIA8M+5/z+QEaJYRU4DLyej4cPmc98zpn3DHDmNR8+53NkoiiKICIiIiIyM3KpCyAiIiIiagkGWSIiIiIySwyyRERERGSWGGSJiIiIyCwxyBIRERGRWWKQJSIiIiKzxCBLRERERGaJQZaIiIiIzJKF1AW0N0EQkJGRAXt7e8hkMqnLISIiIqLbiKKIoqIieHl5QS5veMy1ywXZjIwMaDQaqcsgIiIiogakp6eje/fuDfbpckHW3t4eQPWL4+DgIHE1RGSOBEGAVquFWq1udLSAiIiaR6fTQaPRGDNbQ7pckK2ZTuDg4MAgS0QtIggCysrK4ODgwCBLRNRGmjIFlEdgIiIiIjJLDLJEREREZJa63NQCIiIiImoagyDiVGoetEVlUNurEOTrAoW846z6JOmI7JEjRzBlyhR4eXlBJpNh27ZtjW4TGxuLYcOGQalUok+fPti8eXOb10lERETU1ey/mIHxq/dh3sdH8dI3pzDv46MYv3of9l/MkLo0I0mDrF6vx5AhQ7Bhw4Ym9U9JScGkSZMwduxYnD17Fs899xwef/xx7N27t40rJSIiIuo69l/MwNItJ5CtK6vVnqMrw9ItJzpMmJV0asHEiRMxceLEJvffuHEjevbsiTVr1gAA/Pz88Ouvv+K9995DREREW5VJRERE1GUYBBFROy9ANHGfCEAG4M1dcRjn5yn5NAOzmiN77NgxhIeH12qLiIjAc889V+825eXlKC8vN97W6XQAqpfPEQShTeokos5NEASIoshjCBF1SjGXMuuMxN5OBJBVWIqTKVoM7+na6o/fnGOrWQXZrKwsuLu712pzd3eHTqdDaWkprK2t62wTFRWFVatW1WnXarUoK6v/m0REVB9BEFBYWAhRFLmOLBGZtYLSSiTm6Gv90xZXNGnb5Os58LFt/Q/0RUVFTe5rVkG2JVasWIHIyEjj7ZqrRajVal4QgYhaRBAEyGQyXtmLiMzKDX05LmUU4lJGIS5eL8CljEJkFpa2eH+9vd3g5tb6I7IqlarJfc0qyHp4eCA7O7tWW3Z2NhwcHEyOxgKAUqmEUqms0y6Xy/kGREQtJpPJeBwhog6roKQClzIKcPF6gTG0Xr9RUqefTAb0dLWDv5cTAryr//V1t8f09w8hR1dmcp6sDIC7ozWCe6ohb4M5ss05rppVkA0LC8Pu3btrte3fvx9hYWESVUREREQkrcLSClzOKETc9QJcuhlcr5kIrQDg62pbK7T6eTrCTmVZp9+KyYOwdMsJyIBaYbYmti6fNFDyE70AiYNscXExkpKSjLdTUlJw9uxZODs7o0ePHlixYgWuX7+Ozz//HACwcOFCrF+/Hi+99BIee+wxHDx4EN988w127dol1VMgIiIiaje60kpcyii4bbS1EOn5epN9e7jYVgfWm8HVz8sR9iZCqynjA7zw3sPDEbXzQq0Tv9wdrbF80kCMD/BqledzpyQNsidPnsTYsWONt2vmss6ZMwebN29GZmYm0tLSjPf37NkTu3btwtKlS7F27Vp0794d//nPf7j0FhEREXU6xWWVN+e0FhhHW6/mmQ6tGmcbBHg7GUdb/b2c4GDdtNBan/EBXhjn59mhr+wlE0XR1PSHTkun08HR0RGFhYU82YuIWkQQBOTk5MDNzY1zZImoVejLK29ND8goxMXrN5Caazq0du9mA/+bUwMCvBzh5+UEJxurdq647TQnq5nVHFkiIiIic6cvr0J8ZuFtJ2IVICW3GKaGFr2crOHv7YSBt422dqbQeqcYZImIiIjaSElFdWitOQnrYkYh/tQWmQytHo7WNwOrIwK6d0OAlyO62dZdeYluYZAlIiIiagWlFVVIyNIZR1ovXi/An9oiCCZCq7uDyjiXdaC3E/y9neBix9DaXAyyRERERM1UVmlAQmYhLt5cPeDS9UIka4tgMJFa3exVtea0+ns7QW3f9EX/qX4MskREREQNKK80ICFLZ1zyKu56AZJzTIdWFzslBnrfWqc1wMsJageG1rbCIEtERER0U0WVAYk10wNuXso1KVuHKlOh1VZZPT3A29EYWt0cVJDJOs7yVJ0dgywRERF1SRVVAq5k62qtHpCYrUOVoW5o7WZjdWuU9eY/d4ZWyTHIEhERUadXUSUgOUd388IC1eu0JmYXodIg1OnrZGMFfy9H40lYAd5O8HS0ZmjtgBhkiYiIqFOpNAhIzimqtXpAYrYOFVV1Q6uDtWWtNVoDvJ3g5cTQai4YZImIiKjdGQSxVS59WmUQkKwtwqWbJ2FdyihEfGah6dCqsoS/t2OtJa+6d7NhaDVjDLJERETUrvZfzEDUzgvI1pUZ29wdVFgxeRDGB3jVu12VQUBKbvHN6QHVI60JWTqUVRrq9LVTWty25FX1/xpnhtbOhkGWiIiI2s3+ixlYuuUE/no6VY6uDEu3nMB7Dw/H+AAvGAQRKblFxjVa464XICGzEKUmQqut0qL6ali3XWBA42wLeQtGeMm8MMgSERFRuzAIIqJ2XqgTYgEY217+7jQ++zUJ8Vk6lFbUDa02Vgr4eznVusCAj4sdQ2sXxSBLRERE7eJUal6t6QSmlFQYcCbtBgDA2lIBv5tXwqq5yICPi12L5tJS58QgS0RERO1CW9RwiK0xI8QHD4f1Qk9Xe4ZWahCDLBEREbW5SoOAi9cLmtR3wqDu6OPm0LYFUafAIEtERERtpsogYOe5a/jwYAKu3ShpsK8MgLujNYJ8XdqnODJ7DLJERETU6gRBxM8XruODg/FIzdUDAFxslbi7vxu2nU6HDKh10lfNBILlkwZyOgE1GYMsERERtRpRFHHgUibWH4hHUk4RAMDR2hLz7+mLmSN6wsbKAmMGeNRdR9bRGssnDWxwHVmiv2KQJSIiojsmiiIOJ2Rj/YF4XM4sBADYqyww564+mDWyF+xUlsa+4wO8MM7Ps1Wu7EVdG4MsERERtZgoijiWrMW6A/E4n169bJaNlQKzRvbGnFG94WhtZXI7hVyGkF6u7VkqdUIMskRERNQiJ1Ny8f6BeJxKzQMAqCwVmDmiJ+bf3QfdbJUSV0ddAYMsERERNcu5tHysOxCPY8laAIClQo4ZIb54fHRfqO1VEldHXQmDLBERETXJpesFWB8Tj8MJ2QAAC7kMDwT7YMGYfvBwtJa4OuqKGGSJiIioQYlZOmyIiceBS5kAque3Tg3UYOHYfujubCtxddSVMcgSERGRSSnaImw4mIA9F65DFAGZDLh/cHc8Pa4/fF3tpC6PiEGWiIiIakvP1+PDgwn46Ww6hJtXLbhvoBeeGdcffdx56VjqOBhkiYiICACQUVCCfx9KxLbTaai6mWDHDPDAonsHwM/LUeLqiOpikCUiIuritLoybDqciG9PXEWlQQAAjOyjxuJwPwzWdJO4OqL6McgSERF1Ufn6cnx85Aq++j0F5VXVAXZ4TxcsDvdDkK+LxNURNU4udQEbNmyAr68vVCoVQkNDcfz48Qb7R0dHo3///rC2toZGo8HSpUtRVlbW4DZERER0S0FJBaL3XcJ97+zH5l+TUV4lILBHN3z82Eh8Ov8uhlgyG5KOyG7duhWRkZHYuHEjQkNDER0djYiICCQkJMDNza1O/y1btmD58uX45JNPMHLkSCQmJmLu3LmQyWR49913JXgGRERE5qOorBJfHE3GZ0eTUVxeBQAI8HLE4nA/jOrnBplMJnGFRM0jE0VRlOrBQ0NDMXz4cKxfvx4AIAgCNBoNFi9ejOXLl9fpv2jRIly+fBkxMTHGtueffx5//PEHfv311yY9pk6ng6OjI7RaLRwc6p55KZfLYWFxK99XVFTUuy+ZTAZLS8sW9a2srER9L31b9QUAKyurFvWtqqqCIAit0tfS0tJ4sGyrvgaDAQaDoVX6WlhYQC6Xd5i+giCgqqqq3r4KhQIKhaLD9BVFEZWVla3S9/bfz7bqCzT8uyyXyyGXy5GTkwM3N7cGXwceI1rWl8eIO+tr6vdTX16Fb06k4oujySgoM0CEDP3cHfDMuH64u69rvQGWx4i6fYHGjxHMEXX7NucYkZ+fDxcXFxQWFprMareTbES2oqICp06dwooVK4xtcrkc4eHhOHbsmMltRo4cif/+9784fvw4QkJC8Oeff2L37t2YNWtWvY9TXl6O8vJy422dTgcAWLNmDVSqupfR69OnD2bOnGm8/c4779T7w+3j44PZs2cbb69duxYlJSUm+3p6euLxxx833t6wYQMKCwtN9nV1dcVTTz1lvL1p0ybk5uaa7Ovo6Ihnn33WePvTTz9FZmamyb42NjZ4/vnnjbe//PJLXL161WRfS0vLWh8mtm7diqSkJJN9AeDVV181fv3DDz/g8uXL9fZdtmyZ8Qf2p59+wvnz5+vtGxkZCVvb6sW29+7di5MnT9bbd/HixXBycgIAxMTE1PtzBABPPvmkcdT/yJEjOHLkSL1958+fDy8vLwDAsWPHan2Q+qtZs2bB19cXAHDy5Ens2bOn3r4PPfQQ+vbtCwA4f/48duzYUW/fBx54AP7+/gCAS5cu4fvvv6+379SpUzFkyBAAwJUrV/D111/X23fChAkYPnw4ACA1NRVffPFFvX3vvfdejBw5EgCQkZGBjz/+uN6+99xzD0aPHg0AyMnJwb///e96+4aFhSE8PBwAUFBQgHXr1tXbNzg4GBMnTgQA6PX6Bv8SM3jwYEybNg1A9fHmrbfeqrevn58fHnzwQePtqKioevv26dMHM2bMgCiKEASBx4ibeIwwj2PEeAB/OvTFjAl3474ALyQnJ+HNN/9T7355jKjW3GMEc8SdHSMa+ln/K8mCbG5uLgwGA9zd3Wu1u7u7Iz4+3uQ2Dz/8MHJzczFq1CiIooiqqiosXLgQL7/8cr2PExUVhVWrVjW5roqKCuTk5BhvN/RJ4699G/qkUVVVVatvQ5+sDQZDi/s29AlYEIRafRv65CeKYpP7AqjV9/YPDqZotVrjp7/G5jfn5uZCr9cDQL2/3DXy8vKMddZsU5/8/Hzj103pW/Ppuri4uMG+BQUFxteiqKioyX1rPmDVp7Cw0Ni3vgNXDZ1OZ+xbUFDQYN+ioqIm9y0uLjb2vf31M0Wv17eob2OvWUlJibFvaWlpg33LysqMfRsaaQGqf2Zv/xluSM3vfWFhIURR5DGiCX0BHiNqdIRjxNxQL/i5WyI3V8tjRBseI2rwGNF4XwDN6ns7yaYWZGRkwNvbG7/99hvCwsKM7S+99BIOHz6MP/74o842sbGxeOihh/B///d/CA0NRVJSEpYsWYInnniiVpK/nakRWY1Gg+zsbE4t4J8Nm9TXHP5seDtOLbizvkDTphZotVqo1WpOLbiJx4iOcYyoNAjYduoqPj6ciCxddZDzcLDG/Lt74/4h3WGpkPMYcYd9AU4taEnf5k4tUKvVTZpaIFmQraiogI2NDb777jtMnz7d2D5nzhwUFBRg+/btdba5++67MWLECKxevdrY9t///hcLFixAcXGx8Re+ITVzZJvy4hARmVIzKuHm5tak4w5RWzMIInaeS8eHBxOQnl89Mq22V2LBmH54MNgHVhYKiSskarrmZDXJphZYWVkhKCgIMTExxiArCAJiYmKwaNEik9uUlJTUedO4/VMaERFRVyIIIvbEXccHMQlIya2e1uBsa4XHR/fDjBBfqCwZYKlzk3T5rcjISMyZMwfBwcEICQlBdHQ09Ho95s2bBwCYPXs2vL29jZOqp0yZgnfffRdDhw41Ti149dVXMWXKFGOgJSIi6uxEUUTMpUxsiElAYnb1/FlHa0vMu7sPHh7RC7ZKXu+IugZJf9JnzJgBrVaLlStXIisrC4GBgdizZ4/xBLC0tLRaI7CvvPIKZDIZXnnlFVy/fh1qtRpTpkzBP//5T6meAhERUbsRRRG/JOZg3YHLuJRRfVKXndICc0b1weyRvWCnsmxkD0Sdi6TryEqBc2SJ6E5xjiy1N1EU8XuyFusOxONc+g0AgLWVArPCemHOqD5wsrFqZA9E5sMs5sgSERFR406l5mHdgcs4kZIHAFBayPHwiF547J4+cLZVSlwdkbQYZImIiDqg8+n5WHcgHr8laQEAlgo5/h7igyfu6Qe1Q90L+hB1RQyyREREHcjljAKsOxCPwwnZAAALuQz/E9QDT47pD08na4mrI+pYGGSJiIg6gKRsHdbHxGP/xerLg8plwNShGiwc2x8aZ1uJqyPqmBhkiYiIJJSaW4wNMfH4+cJ1iCIgkwETB3nj6XH90VNtL3V5RB0agywREZEEruXr8eGhBOw4kw7h5vpB4QGeWHTvAPR156o6RE3BIEtERNSOMgtKsSk2ET+cuoqqmwl2dH93LA4fAD8vJ2mLIzIzDLJERETtQKsrw0dHEvHN8auoNAgAgJF91Fh07wAM6eEscXVE5olBloiIqA3l68vxyZEkfPVHCsoqDQCAYF8XLA4fgOCerhJXR2TeGGSJiIjaQGFpBTb/moz//paMkorqADtE0w2LwwdgRG81ZDKZxBUSmT8GWSIiolZUXFaJL377E58dTUJRWRUAwM/TEYvDB+Ce/u4MsEStiEGWiIioFZRUVGHL7yn45MgVFJZWAgD6uttj0b0DcK+/JwMsURtgkCUiIroD5ZUGbD2eiv8cvoI8fTkAoKerHZ6+tz8mDPSGXM4AS9RWGGSJiIhaoKLKgO9PpmFTbCJyisoAAN272eCpcf0xeUh3WCjkEldI1PkxyBIRETVDpUHAjjPp+PBQAjILSgEAHo7WWDi2H6YP6wFLBliidsMgS0RE1AQGQcSuc9fwwcEEpOfrAQCudkosGNMP/zvcB1YWCokrJOp6GGSJiIgaIAgi9l3MwIaYePypLQYAdLOxwuOj+2JGiC+srfhWSiQV/vYRERGZIIoiDl3OwrqYeCRm6QAADtaWeOzuPnh4RC/YKvkWSiQ1/hYSERHdRhRF/JqYg3Ux8bh4vQAAYKe0wOy7emP2Xb1hr7KUtkAiMmKQJSIiuun3ZC3WHYjH2bR8AIC1pQKPjuyFuaP6wMnGSuLqiOivGGSJiKjLO5Wah3UH4nEiJRcAoLSQ46HQnph/T1+42Cklro6I6sMgS0REXdaFazew7kA8jl7JAQBYKGT432BfLBjTF24O1hJXR0SNYZAlIqIuJz6zEOsPxONQfBYAQCGX4X+G9cCTY/vBy8lG4uqIqKkYZImIqFMxCCJOpeZBW1QGtb0KQb4uUNy8TGxSjg4bYhKwLy4DACCXAZMDNXhqbH/0cLGVsmwiagEGWSIi6jT2X8xA1M4LyNaVGdvcHVR4/J6+OJd+A7vOX4MoVrdPHOSNp+/tj15qe4mqJaI7xSBLRESdwv6LGVi65QTEv7Rn68rwz50XjLfD/T3xzL0D0M/DoX0LJKJWxyBLRERmzyCIiNp5oU6IvZ3SQo7PHh+FQZpu7VYXEbUtudQFEBER3alTqXm1phOYUl4loLTS0E4VEVF7YJAlIiKzp9WVNq1fUcNhl4jMi+RBdsOGDfD19YVKpUJoaCiOHz/eYP+CggI888wz8PT0hFKpRL9+/bB79+52qpaIiDqac2n5+HdsYpP6qu1VbVwNEbUnSefIbt26FZGRkdi4cSNCQ0MRHR2NiIgIJCQkwM3NrU7/iooKjB8/Hm5ubvjuu+/g7e2Nq1evwsnJqf2LJyIiSWUUlCB632XsOnet0b4yAO6O1gjydWn7woio3UgaZN9991088cQTmDdvHgBg48aN2LVrFz755BMsX768Tv9PPvkE+fn5+O2332BpaQkA8PX1bc+SiYhIYvryKnx85Ao2/5qE8ioBMhkwbagGQ3s44/Vt5wCg1klfspv/L5800LieLBF1DpIF2YqKCpw6dQorVqwwtsnlcoSHh+PYsWMmt9mxYwfCwsLwzDPPYPv27VCr1Xj44YexbNkyKBSK9iqdiIgkIAgitp9Jw9r9l6EtKgcABPu6YNn9A+Hv7QQAcLSxqruOrKM1lk8aiPEBXlKUTURtSLIgm5ubC4PBAHd391rt7u7uiI+PN7nNn3/+iYMHD+KRRx7B7t27kZSUhKeffhqVlZV47bXXTG5TXl6O8vJy422dTgcAEAQBgiC00rMhoq5EEASIoshjSDs6mZqHt3++iMsZhQAA7242eD7CD+H+npDJZMbvxb1+HhjT3x2nr+ZBW1QOtb0Sw3yqr+zF7xeReWjO76pZrSMrCALc3NywadMmKBQKBAUF4fr161i9enW9QTYqKgqrVq2q067ValFWxrNXiaj5BEFAYWEhRFGEXC75ObOdWkZhGTYdTcMvyTcAALZWCjwy3Av/M8QDVgo5tFqtye18bAEfW0sAAvJyTfchoo6pqKioyX2bHWQrKythbW2Ns2fPYuDAgc3d3MjV1RUKhQLZ2dm12rOzs+Hh4WFyG09PT1haWtaaRuDn54esrCxUVFTAysqqzjYrVqxAZGSk8bZOp4NGo4FarYaDA6/qQkTNJwgCZDIZ1Go1g2wbKSqrxKbYK/jy9xRUGgTIZcCDwT54elx/uNgppS6PiNqQStX01UWaHWQtLS3Ro0cPGAx3tqi0lZUVgoKCEBMTg+nTpwOofnOIiYnBokWLTG5z1113YcuWLRAEwfjmkZiYCE9PT5MhFgCUSiWUyroHPblczjcgImoxmUzG40gbqDII+P7kVayPiUe+vgIAMLKPGi9OHMhLyhJ1Ec05rrboCPz//t//w8svv4z8/PyWbG4UGRmJjz76CJ999hkuX76Mp556Cnq93riKwezZs2udDPbUU08hPz8fS5YsQWJiInbt2oV//etfeOaZZ+6oDiIikt7RKzl4YH0s3thxHvn6CvR0tcMHs0KxaW4YQywRmdSiObLr169HUlISvLy84OPjA1tb21r3nz59ukn7mTFjBrRaLVauXImsrCwEBgZiz549xhPA0tLSaqVyjUaDvXv3YunSpRg8eDC8vb2xZMkSLFu2rCVPg4iIOoA/tUVYvfsijiRWTzVzsLbEonsH4O8hvrBUcMSbiOonE0VRbLxbbaZOnrpdfSdedQQ6nQ6Ojo4oLCzkHFkiahFBEJCTkwM3NzdOLbgDBSUV+OBgPL7+IxUGQYSFXIaHRvTEU2P7w8nG9HQxIur8mpPVWjQi25GDKhERdWwVVQK+/iMFHx5KgK60EgAwZoAHXpjgj55qe4mrIyJz0uLltwoKCvDdd98hOTkZL774IpydnXH69Gm4u7vD29u7NWskIqJOQBRFxMZn4509cUjN1QMA+rk74KX7AxDWp+5lyYmIGtOiIHv+/HmEh4fD0dERqampeOKJJ+Ds7IwffvgBaWlp+Pzzz1u7TiIiMmMJWYV4e3ccfk/OBQC42CqxKHwAHgj24WVjiajFWhRkIyMjMXfuXLz99tuwt7/1Z6D7778fDz/8cKsVR0RE5i23uAzr9sfjh1NXIYiApUKO2Xf1woLR/WCnspS6PCIycy0KsidOnMC///3vOu3e3t7Iysq646KIiMi8lVca8MWxP7EpNhH68ioAQMRAL0RG+KO7s20jWxMRNU2LgqxSqYROp6vTnpiYCLVafcdFERGReRJFEfviMrBm7yVcv1ECAAjwcsSySYMQ5OsicXVE1Nm0KMhOnToVb7zxBr755hsA1Ve4SUtLw7Jly/DAAw+0aoFERGQe4q7dwFu743D6avXFctzsVXguwg9Thmgg5zxYImoDLQqya9aswYMPPgg3NzeUlpZi9OjRyMrKQlhYGP75z3+2do1ERNSBZReWInr/Zew4kw4AUFkqMG9UHzx2Tx/YWLV4cRwioka16Ajj6OiI/fv349dff8X58+dRXFyMYcOGITw8vLXrIyKiDqq0ogqf/pKET35JQmmlAQAwJbA7nrvPHx6O1hJXR0RdQYuCbFlZGVQqFUaNGoVRo0a1dk1ERNSBCYKIneeuIXrfJWTrygAAgT2csez+gRis6SZxdUTUlbQoyDo5OSEkJASjR4/G2LFjERYWBmtrfvomIursTl/Nw1u74hB3vQAA4OVkjciIAEwY5AWZjPNgiah9tSjIHjhwAEeOHEFsbCzee+89VFVVITg4GKNHj8aYMWMwfvz41q6TiIgkdP1GCd7dexF7LmQAAGysFFgwph9mj+wNpaVC4uqIqKuSiaIo3skOqqqqjOvKfvnllxAEAQaDobXqa3U6nQ6Ojo4oLCyEg4OD1OUQkRkSBAE5OTlwc3ODXC6Xupw2VVxWiY8OX8HnvyWjokqATAb8LcgHi8MHQG2vkro8IuqEmpPVWnw6aWJiImJjY43/ysvLMXnyZIwZM6aluyQiog7CIIj48VQa3j9wGXnF5QCAkF6ueGniQPh5OUpcHRFRtRYFWW9vb5SWlmLMmDEYM2YMli1bhsGDB3N+FBFRJ/B7shZv745DQlb1hW96uNjixQkBGOvnweM8EXUoLQqyarUa8fHxyMrKQlZWFrKzs1FaWgobG5vWro+IiNrJ1dxirP75Ig7FV19q3EFliYXj+mNmaE9YWXTuKRREZJ5aFGTPnj2LgoICHDlyBIcPH8bLL7+MS5cuITAwEGPHjuVFEYiIzEhhaQU2HkrElt//RJVBhEIuw4wQXzw9rj+62SqlLo+IqF53fLJXXl4eYmNjsX37dnz11Vc82YuIOr3OcrJXpUHAN8dT8cHBBBSUVAAA7u7nhhcmBqCPG4+PRCSNNj/Z64cffjCe5HXp0iU4Oztj1KhRWLNmDUaPHt2ioomIqP38kpCNt3+Ow5/aYgBAbzd7vHT/QIzq6yZxZURETdeiILtw4ULcc889WLBgAUaPHo1Bgwa1dl1ERNQGkrJ1ePvnizh6JQcA4GRjhcXhA/BgsA8sFOY7ukxEXVOLgmxOTk5r10FERG0oX1+O9Qfi8d3JqzAIIiwUMjwa1gtPjukPB2tLqcsjImqRFq8jazAYsG3bNly+fBkA4O/vj2nTpkGh4BVeiIg6iooqA7b8noKNhxJQVFYFAAj390TkBH/4uNhJXB0R0Z1pUZBNSkrC/fffj+vXr6N///4AgKioKGg0GuzatQu9e/du1SKJiKh5RFFEzKVMvLPnEtLz9QCAAZ6OWHb/QIT0cpW4OiKi1tGiIPvss8+id+/e+P333+Hs7AygevWCRx99FM8++yx27drVqkUSEVHTXc4owFu7L+JESi4AwNVOiSXj/TBtWA8o5LygARF1Hi0KsocPH64VYgHAxcUFb775Ju66665WK46IiJpOqyvD+wcu48fTaRBFwMpCjrl39cbjo/vCVsl5sETU+bQoyCqVShQVFdVpLy4uhpWV1R0XRURETVdWacBnvybhoyNXUFpRvY73xMHeiLzPH17deMVFIuq8WhRkJ0+ejAULFuDjjz9GSEgIAOCPP/7AwoULMXXq1FYtkIiITBNFET9fuI53915CZkEpAGCwphuW3T8QgT2cG9maiMj8tSjIvv/++5gzZw7CwsJgaVn956rKykpMmzYNa9eubdUCiYiorvPp+XhrdxzOpt0AAHg4WmNphD/uH+QNOefBElEX0aIg6+TkhO3btyMpKQmXLl0CUL38Vp8+fVq1OCIiqi2zoBTv7buEXeeuAQCsLRWYf09fzB3VG9ZWLV5RkYjILLX4qPfxxx/jvffew5UrVwAAffv2xXPPPYfHH3+81YojIqJq+vIqfPLLFXz6SxLKqwTIZMC0oRosGe8HNwdrqcsjIpJEi65HuHLlSixZsgRTpkzBt99+i2+//RZTpkzB0qVLsXLlymbvb8OGDfD19YVKpUJoaCiOHz/epO2+/vpryGQyTJ8+vdmPSURkDgRBxI+n0zDpvQPYeCgR5VUCgn1d8M1To/HPB4YxxBJRlyYTRVFs7kZqtRrvv/8+Zs6cWav9q6++wuLFi5Gbm9vkfW3duhWzZ8/Gxo0bERoaiujoaHz77bdISEiAm5tbvdulpqZi1KhR6NWrF5ydnbFt27YmPZ5Op4OjoyMKCwvh4ODQ5DqJiGoIgoCcnBy4ublBLm/ReECTnEzJxVu743ApoxAA0L2bDZ6fEIDxAZ6QyTgPlog6p+ZktRYdgSsrKxEcHFynPSgoCFVVVc3a17vvvosnnngC8+bNg7+/PzZu3AgbGxt88skn9W5jMBjwyCOPYNWqVejVq1ez6yci6sjS8vRYsuU45vznKC5lFMJOaYHnJ/jjp+fG4b6BXgyxREQ3tSjIzpo1Cx9++GGd9k2bNuGRRx5p8n4qKipw6tQphIeH3ypILkd4eDiOHTtW73ZvvPEG3NzcMH/+/OYVTkTUgRWVVeKdPRcxde1BHLiYCbkMmBHii92R4Xjs7r6wslBIXSIRUYdyRyd77du3DyNGjABQvY5sWloaZs+ejcjISGO/d999t9595ObmwmAwwN3dvVa7u7s74uPjTW7z66+/4uOPP8bZs2ebVGd5eTnKy8uNt3U6HYDqPw0KgtCkfRAR3U4QBIii2GrHkCqDgB9OpWHDwQTk6ysAAGG9XfHChAD083AwPiYRUVfQnONdi4JsXFwchg0bBgBITk4GALi6usLV1RVxcXHGfq3956+ioiLMmjULH330EVxdXZu0TVRUFFatWlWnXavVoqysrFXrI6KuQRAEFBYWQhTFO54jezKtABt/TUNKXvUFDTROKiwc1QOhvk6QycqQk8PjFBF1LaauHlufFgXZQ4cOtWSzOlxdXaFQKJCdnV2rPTs7Gx4eHnX6JycnIzU1FVOmTDG21aR2CwsLJCQkoHfv3rW2WbFiRa0RYp1OB41GA7VazZO9iKhFBEGATCaDWq1ucZD9U1uENXsu4UhiDgDAwdoST4/rj78P94Glou1OICMi6uhUKlWT+0q6eraVlRWCgoIQExNjXEJLEATExMRg0aJFdfoPGDAAFy5cqNX2yiuvoKioCGvXroVGo6mzjVKphFKprNMul8vb9GxjIurcZDJZi44jBSUV+OBgPLb+kYoqQYSFXIaHRvTEU2P7w8nGqo2qJSIyH805rkp+GZjIyEjMmTMHwcHBCAkJQXR0NPR6PebNmwcAmD17Nry9vREVFQWVSoWBAwfW2t7JyQkA6rQTEXUklQYBX/+Rgg8OJkBXWgkAGDPAAy9M8EdPtb3E1RERmSfJg+yMGTOg1WqxcuVKZGVlITAwEHv27DGeAJaWlsaRUyIyW6IoIjY+G+/siUNqrh4A0M/dAS/dH4CwPvWvlU1ERI1r0QURzBkviEBEd6qpF0RIyCrE27vj8Hty9UViXGyVWBQ+AA8E+0Ah51qwRESmNCerST4iS0TU2eQWl2H9gXh8f/IqBBGwVMgx+65eWDC6H+xUllKXR0TUaTDIEhG1kvJKA7449ic2xSZCX159lcP7BnohMsIfGmdbiasjIup8GGSJiJrBIIg4mZKL5Ou56K2XI7inGnIZsO9iJt7dcxHXbpQAAAK8HLFs0iAE+bpIXDERUefFIEtE1ET7L2YgaucFZOtqLlKQDGdbKzjZWOFPbTEAwM1eheci/DBliAZyzoMlImpTDLJERE2w/2IGlm45gb+eHZuvr0C+vgKWCjkev6cvHrunD2yseGglImoPPNoSETXCIIiI2nmhToi9nZONJZ4a15+rERARtSMu0EpE1IijV3Jum05gmraoHKdS89qpIiIiAjgiS0RkkkEQcSIlF9vPpOPn89eatI22qOGwS0RErYtBlojoNsk5RfjpbDp+OpuOrMLmBVO1vaqNqiIiIlMYZImoy7uhL8fu89ex40w64q4XGNsdVJaYONgbk4d0xwtbTyJHV2ZynqwMgLujNZfaIiJqZwyyRNQlVVQZcDghGzvOpONIQjaqhOqIaiGXYVQ/d0wbqsHo/u5QWioAACsmD8LSLScgA2qF2ZpTu5ZPGsgTvYiI2hmDLBF1GaIo4nz6jep5rxeuQ1daabwvwMsRU4dqMHFwd7jYKetsOz7AC+89PPwv68hWj8QunzQQ4wO82uU5EBHRLQyyRNTpZdwowU9n07HjbDpSc/XGdncHFSYHdsfUQA36uDs0up/xAV4Y5+eJkylaJF/PQW9vNwT3VHMklohIIgyyRNQpFZdVYv/FDGw/cw0nUnKN7daWCowP8MTUoRqE9Gp+CFXIZRje0xU+tgLc3Fx59S4iIgkxyBJRp2EQRBxL1mLHmXTEXMpEWaUBACCTASE9XTFtmAbh/l6wVfLQR0TUGfBoTkRmLzFLhx1n0rHzXDq0ReXG9l5qO0wdqsGkId3h5WQjYYVERNQWGGSJyCzlFpdh97nr2H4mHfGZhcZ2Jxsr3D/YG1OHajDQ2wkyGf/0T0TUWTHIEpHZKK804FB8FnacScevV3JgqFkySyHDmAEemBaowah+7rCy4NW3iYi6AgZZIurQRFHE6av52HEmHXvjrqOorMp432BNN0wN1GDiYG842VhJWCUREUmBQZaIOqS0PL3xUrHp+SXGdk8na0wJ1GBqYHf0VNtLWCEREUmNQZaIOgxdaSX2xlXPez1zNd/YbmOlQMTA6nmvwb4uXPKKiIgAMMgSkcQqDQJ+u5KDHWfTcfByFiqqBACAXAaE9XHD1KEa3OvnAWsrHq6IiKg2vjMQUbsTRRHxmYXYfiYdu89dR57+1pJZfdzsMW2YBpOHdIebg7WEVRIRUUfHIEtE7SZHV4qd565hx5l0XMkuMra72Cpx/xBvTBuqwQBPRy6ZRURETcIgS0RtqrSiCjGXq5fMOpaUg5srZsHKQo5xfh6YGqjByL5usFRwySwiImoeBlkianWCIOJkap5xyaySCoPxvmE+zpg6VIOIgd5wsLaUsEoiIjJ3DLJE1GpStEXYcfYafjqbjsyCUmN79242mDpUgymBGvRwsZWwQiIi6kwYZInojhSUVODn89ex42w6zqffMLbbqyyMS2YN83HmvFciImp1DLJE1GwVVQJ+TczG9rPpiI3PQpWheuKrQi7DqL7VS2aNGeABlaVC4kqJiKgzY5AloiYRRRFx1wuw40w6dp+/joKSCuN9fp6OmDpUg/uHeMPVTiVhlURE1JV0iCC7YcMGrF69GllZWRgyZAjWrVuHkJAQk30/+ugjfP7554iLiwMABAUF4V//+le9/YnozmQWlGLnuXTsOJOOP7XFxnZXO2X1pWKHatDPw0HCComIqKuSPMhu3boVkZGR2LhxI0JDQxEdHY2IiAgkJCTAzc2tTv/Y2FjMnDkTI0eOhEqlwltvvYX77rsPFy9ehLe3twTPgKjz0ZdXYf/FDOw4k47jKbkQby6ZpbJU4F5/T0wdqsGIXq6w4JJZREQkIZko1rxFSSM0NBTDhw/H+vXrAQCCIECj0WDx4sVYvnx5o9sbDAZ069YN69evx+zZsxvtr9Pp4OjoiMLCQjg4cBSJqIZBEHH8Ty12nEnH/ouZKK28tWTW8J6umDa0O8YHeMFOxSWzBEFATk4O3NzcIJczzBMRtabmZDVJR2QrKipw6tQprFixwtgml8sRHh6OY8eONWkfJSUlqKyshLOzc1uVSdSpJeXosONMOnaevYZsXZmx3cfFFtOGajA5UAPvbjYSVkhERGSapEE2NzcXBoMB7u7utdrd3d0RHx/fpH0sW7YMXl5eCA8PN3l/eXk5ystvXcddp9MBqB5REQShhZUTmbd8ffnNJbOu4VJGobHdwdoSEwd5YWqgBoO6OxmXzOLvSm2CIEAURb4uRERtoDnHVsnnyN6JN998E19//TViY2OhUpk+UzoqKgqrVq2q067ValFWVmZiC6LOqcIg4FjKDeyPz8Xxq4UwCLeWzAr1ccJ9A1wR2tMJVgo5gEpotVppC+7ABEFAYWEhRFHk1AIiolZWVFTU5L6SBllXV1coFApkZ2fXas/OzoaHh0eD277zzjt48803ceDAAQwePLjefitWrEBkZKTxtk6ng0ajgVqt5hxZ6vREUcS59Bv46ew1/HwhA0Vllcb7Bno7YUpgd0wc5IVutkoJqzQ/giBAJpNBrVYzyBIRtbL6BidNkTTIWllZISgoCDExMZg+fTqA6jeImJgYLFq0qN7t3n77bfzzn//E3r17ERwc3OBjKJVKKJV136TlcjnfgKjTupavx09nr2HH2XSk5emN7R6OKkwJrL5UbG83ewkrNH8ymYzHESKiNtCc46rkUwsiIyMxZ84cBAcHIyQkBNHR0dDr9Zg3bx4AYPbs2fD29kZUVBQA4K233sLKlSuxZcsW+Pr6IisrCwBgZ2cHOzs7yZ4HkdSKyiqxL656yayTqXnGdmsrBcYHeGHaUA2G93SFQs5LxRIRUecgeZCdMWMGtFotVq5ciaysLAQGBmLPnj3GE8DS0tJqJfMPP/wQFRUVePDBB2vt57XXXsPrr7/enqUTSa7KIOBYkhbbz6bj4KVMlFdVT5CXyYARvdSYOlSDe/09YauU/FediIio1Um+jmx74zqy1BkkZBUal8zKLb61KkcvtR2mDeuByUO6w8PRWsIKOzeuI0tE1HbMZh1ZImo6bVEZdp27hh1n0pGQpTO2d7OxwqQh3TF1qAb+Xo7GJbOIiIg6OwZZog6srNKAg5czseNMOo5eycHNFbNgqZBjrJ8HpgZqMKqfGyx5qVgiIuqCGGSJOhhBEHEmLR/bT6dhb1wGisurjPcF9uiGqYEaRAzyhpONlYRVEhERSY9BlqiDuJpXjJ/OXMNPZ9Nx7UaJsd27mw2mBHbH1EANfFy5MgcREVENBlkiCRWWVmDvhQxsP5OOs2n5xnZbpQUmDPTC1KEaDPNxgZxLZhEREdXBIEvUzioNAn5NzMGOs+k4dDkLlYbqJbPkMuCuvm6YOlSDsQM8YG3FX08iIqKG8J2SqB2IoojLGYXYfjYdu89dQ76+wnhfP3cHTBumwaTB3aF2aPpl+YiIiLo6BlmiNpRdWIqdN5fMSsopMra72Ckx+eaSWQM8HSWskIiIyHwxyBK1spKKKsRcql4y61iyFjWXHFFayDHO3xPTAjUI66OGBZfMIiIiuiMMskStQBBEHE/JxY4z6dh3MQOlFQbjfcG+Lpg6VIP7BnrBXmUpYZVERESdC4Ms0R34U1uEn86kY8fZa8gqLDW2a5xtMXWoBlMCu0PjbCthhURERJ0XgyxRMxWUVGD3+ep5rxeuFRjbHVSWmDDIG1OHahDYoxsvFUtERNTGGGSpyzMIIk6l5kFbVAa1vQpBvi5Q/GXd1ooqA44kZGPHmXQcTsxGlaF64qtCLsPd/dwxdagGY/q7Q2mpkOIpEBERdUkMstSl7b+YgaidF5CtKzO2uTuosGLyIIT7e+LCtRvYfiYdP5+/jsLSSmOfAC9HTB2qwcTB3eFip5SidCIioi6PQZa6rP0XM7B0ywmIf2nP1pXhuS0n4GavQk7RrYDrZq/C5MDqJbP6uju0b7FERERUB4MsdUkGQUTUzgt1QuztcorKoLKQY/zNS8WG9lLXmXJARERE0mGQpS5BFEXcKKnA1Vw90vKLceyKttZ0gvq89/Bw3NPfox0qJCIiouZikKVOQxRFFJRU4GqeHlfzipGWp8fVPD3ScouRlq9HUVlVs/fZkm2IiIiofTDIklmpCas1IbVWYM0rbjR4ejhaw8fFFipLBQ4nZDf6eGp7VWuVTkRERK2MQZY6pIKSCly9OZJ6NfdWYE3L00NXVtngth6O1ujhYgsfF9ub/9vBx8UW3Z2rAyxQPUd2/Op9yNGVmZwnKwPg7miNIF+X1n9yRERE1CoYZEky1SOrxTdHU2+OsN4Mr7rSxsKqChrn6oDq41odVnu42EJzW1htiEIuw4rJg7B0ywnIgFphtuZ0ruWTBvLkLiIiog6MQZbaVM00gNqBtfrrxsKqu4MKPW6OptaMsPq42qF7NxtYW935j+74AC+89/DwuuvIOlpj+aSBGB/gdcePQURERG2HQZbuWGFpzWoA1SdWXc3T35wSUFzrIgKmVIdV278EVjtonFsnrDZmfIAXxvl5NnplLyIiIup4GGSpSXSllcaR1L+OsBaUVDS4rZu96tacVddbgVXjbAubdgirjVHIZQjp5Sp1GURERNRM0qcI6jB0pZXGgHo1T4/0275uLKyq7ZXGk6o0LrfmrPZw6RhhlYiIiDofJowupqisstZqAGl5t76+0YSwevsUgJqvNc62sFXyR4mIiIjaF9NHJ1RcVmlyjdW0PD3y9Q2HVVc75a0lq1xt0cO5OrD2cGFYJSIioo6FyaQNGQSxzU4iKi6rrPeiAI2FVRc7ZfUKADcD6q0RVlvYKi1bpT4iIiKitsYg20b2X8you6yTgworJg9q8rJO+vJK42oAV29bDSAtV488fXmD27oYR1brBlaGVSIiIuoMGGTbwP6LGVi65USdK0bl6MqwdMsJvPfwcGOY1ZdX3jaaqq+1MkBecSNh1VZ588SqmpB6a/6qnYphlYiIiDq3DhFkN2zYgNWrVyMrKwtDhgzBunXrEBISUm//b7/9Fq+++ipSU1PRt29fvPXWW7j//vvbseL6GQQRUTsvmLzsaU3bim9P47Nfk5CWX9JoWHW2taqzxmrNNAB7hlUiIiLqwiQPslu3bkVkZCQ2btyI0NBQREdHIyIiAgkJCXBzc6vT/7fffsPMmTMRFRWFyZMnY8uWLZg+fTpOnz6NgQMHSvAMajuVmldrOoEppZUGnEm7YbxdHVbrXhSAYZWIiIiofjJRFE0NHrab0NBQDB8+HOvXrwcACIIAjUaDxYsXY/ny5XX6z5gxA3q9Hjt37jS2jRgxAoGBgdi4cWOjj6fT6eDo6IjCwkI4ODi03hO5ade5a3jpm1ON9ps5oiemD9Wgh4sdHKwZVonMiSAIyMnJgZubG+RyudTlEBF1Ks3JapIegSsqKnDq1CmEh4cb2+RyOcLDw3Hs2DGT2xw7dqxWfwCIiIiot397U9urmtTvvgAvDOzejSGWiIiIqIUknVqQm5sLg8EAd3f3Wu3u7u6Ij483uU1WVpbJ/llZWSb7l5eXo7z81jxUnU4HoHpERRCEOynfpKE9usHdQYUcXZnJebIyAO6OKgzt0a1NHp+I2p4gCBBFkb/DRERtoDnHVsnnyLa1qKgorFq1qk67VqtFWVnDc1lb6qm7NHj95ysm7xMBLBypQV6utk0em4janiAIKCwshCiKnFpARNTKioqKmtxX0iDr6uoKhUKB7OzsWu3Z2dnw8PAwuY2Hh0ez+q9YsQKRkZHG2zqdDhqNBmq1uk3myALAA25ucHR0xJu742qd+OXhqMKyiQMRHuDZJo9LRO1DEATIZDKo1WoGWSKiVqZSNW2aJiBxkLWyskJQUBBiYmIwffp0ANVvEDExMVi0aJHJbcLCwhATE4PnnnvO2LZ//36EhYWZ7K9UKqFUKuu0y+XyNn0Dum+QN+4N8GqzK3sRkbRkMlmbH0eIiLqi5hxXJZ9aEBkZiTlz5iA4OBghISGIjo6GXq/HvHnzAACzZ8+Gt7c3oqKiAABLlizB6NGjsWbNGkyaNAlff/01Tp48iU2bNkn5NExSyGUI6eUqdRlEREREnZLkQXbGjBnQarVYuXIlsrKyEBgYiD179hhP6EpLS6uVzEeOHIktW7bglVdewcsvv4y+ffti27ZtHWINWSIiIiJqP5KvI9ve2nodWSLq/LiOLBFR22lOVpN8RLa91eT2mmW4iIiaSxAEFBUVQaVSMcgSEbWymozWlLHWLhdka5Z00Gg0EldCRERERPUpKiqCo6Njg3263NQCQRCQkZEBe3t7yGTts4LA8OHDceLEiXZ5rI7E3J53R6m3vetoy8dri3231j7vZD81y/ilp6dzilIH1lF+p9ubuT3vjlJvZzr2tsX+2/PYK4oiioqK4OXl1ehfvbrciKxcLkf37t3b9TEVCkWXfLMzt+fdUept7zra8vHaYt+ttc/W2I+Dg0OH+Jkh0zrK73R7M7fn3VHq7UzH3rbYf3sfexsbia3ByV3t4JlnnpG6BEmY2/PuKPW2dx1t+Xhtse/W2mdH+X5T2+mq32Nze94dpd7OdOxti/131GNvl5taQER0p7j6CRFRx8ARWSKiZlIqlXjttddMXjWQiIjaD0dkiYiIiMgscUSWiIiIiMwSgywRERERmSUGWSIiIiIySwyyRERERGSWGGSJiFpJeno6xowZA39/fwwePBjffvut1CUREXVqXLWAiKiVZGZmIjs7G4GBgcjKykJQUBASExNha2srdWlERJ1Sl7tELRFRW/H09ISnpycAwMPDA66ursjPz2eQJSJqI5xaQER005EjRzBlyhR4eXlBJpNh27Ztdfps2LABvr6+UKlUCA0NxfHjx03u69SpUzAYDNBoNG1cNRFR18UgS0R0k16vx5AhQ7BhwwaT92/duhWRkZF47bXXcPr0aQwZMgQRERHIycmp1S8/Px+zZ8/Gpk2b2qNsIqIui3NkiYhMkMlk+PHHHzF9+nRjW2hoKIYPH47169cDAARBgEajweLFi7F8+XIAQHl5OcaPH48nnngCs2bNkqJ0IqIugyOyRERNUFFRgVOnTiE8PNzYJpfLER4ejmPHjgEARFHE3LlzMW7cOIZYIqJ2wCBLRNQEubm5MBgMcHd3r9Xu7u6OrKwsAMDRo0exdetWbNu2DYGBgQgMDMSFCxekKJeIqEvgqgVERK1k1KhREARB6jKIiLoMjsgSETWBq6srFAoFsrOza7VnZ2fDw8NDoqqIiLo2BlkioiawsrJCUFAQYmJijG2CICAmJgZhYWESVkZE1HVxagER0U3FxcVISkoy3k5JScHZs2fh7OyMHj16IDIyEnPmzEFwcDBCQkIQHR0NvV6PefPmSVg1EVHXxeW3iIhuio2NxdixY+u0z5kzB5s3bwYArF+/HqtXr0ZWVhYCAwPx/vvvIzQ0tJ0rJSIigEGWiIiIiMwU58gSERERkVlikCUiIiIis8QgS0RERERmiUGWiIiIiMwSgywRERERmSUGWSIiIiIySwyyRERERGSWGGSJiIiIyCwxyBIRERGRWWKQJSIyQSaTYdu2bZI9/ubNm+Hk5NQm+/b19UV0dHSr7jM2NhYymQwFBQWtul8iooYwyBJRl/b6668jMDCwTntmZiYmTpzY/gXdNGPGDCQmJt7RPtoyDLeVMWPG4LnnnqvVxpBMRPWxkLoAIqKOyMPDQ9LHt7a2hrW1taQ1dDaiKMJgMMDCgm99RJ0FR2SJSDJ6vR6zZ8+GnZ0dPD09sWbNmjojcqb+xO/k5ITNmzcbb6enp+Pvf/87nJyc4OzsjGnTpiE1NdV4f2xsLEJCQmBrawsnJyfcdddduHr1KjZv3oxVq1bh3LlzkMlkkMlkxv3+9XEvXLiAcePGwdraGi4uLliwYAGKi4uN98+dOxfTp0/HO++8A09PT7i4uOCZZ55BZWWlsc8HH3yAvn37QqVSwd3dHQ8++GC9r81fR1NrRo6/+OIL+Pr6wtHREQ899BCKiopMbh8bG4t58+ahsLDQ+Nxef/114/0lJSV47LHHYG9vjx49emDTpk21tm/sNa3P0aNHMXjwYKhUKowYMQJxcXHG+/Ly8jBz5kx4e3vDxsYGgwYNwldffVXrNTx8+DDWrl1rrDk1NRVjx44FAHTr1g0ymQxz584FAAiCgKioKPTs2RPW1tYYMmQIvvvuu1qvgUwmw88//4ygoCAolUr897//hVwux8mTJ2vVHR0dDR8fHwiC0OhzJKKOg0GWiCTz4osv4vDhw9i+fTv27duH2NhYnD59uln7qKysREREBOzt7fHLL7/g6NGjsLOzw4QJE1BRUYGqqipMnz4do0ePxvnz53Hs2DEsWLAAMpkMM2bMwPPPP4+AgABkZmYiMzMTM2bMqPMYer0eERER6NatG06cOIFvv/0WBw4cwKJFi2r1O3ToEJKTk3Ho0CF89tln2Lx5szEYnzx5Es8++yzeeOMNJCQkYM+ePbjnnnua9VyTk5Oxbds27Ny5Ezt37sThw4fx5ptvmuw7cuRIREdHw8HBwfjcXnjhBeP9a9asQXBwMM6cOYOnn34aTz31FBISEpr0mjbkxRdfxJo1a3DixAmo1WpMmTLFGObLysoQFBSEXbt2IS4uDgsWLMCsWbNw/PhxAMDatWsRFhaGJ554wlizRqPB999/DwBISEhAZmYm1q5dCwCIiorC559/jo0bN+LixYtYunQpHn30URw+fLhWTcuXL8ebb76Jy5cvY+rUqQgPD8enn35aq8+nn36KuXPnQi7n2yKRWRGJiCRQVFQkWllZid98842xLS8vT7S2thaXLFlibAMg/vjjj7W2dXR0FD/99FNRFEXxiy++EPv37y8KgmC8v7y8XLS2thb37t0r5uXliQDE2NhYk3W89tpr4pAhQ+q03/64mzZtErt16yYWFxcb79+1a5col8vFrKwsURRFcc6cOaKPj49YVVVl7PO///u/4owZM0RRFMXvv/9edHBwEHU6XaOvjSiK4qeffio6OjrWqtPGxqbW9i+++KIYGhra5H3U8PHxER999FHjbUEQRDc3N/HDDz8URbHx19SUQ4cOiQDEr7/+2thW8/3cunVrvTVOmjRJfP755423R48eXev7f/u+b9y4YWwrKysTbWxsxN9++61W3/nz54szZ86std22bdtq9dm6davYrVs3saysTBRFUTx16pQok8nElJSUeuskoo6JHz2JSBLJycmoqKhAaGiosc3Z2Rn9+/dv1n7OnTuHpKQk2Nvbw87ODnZ2dnB2dkZZWRmSk5Ph7OyMuXPnIiIiAlOmTMHatWuRmZnZrMe4fPkyhgwZAltbW2PbXXfdBUEQjKOYABAQEACFQmG87enpiZycHADA+PHj4ePjg169emHWrFn48ssvUVJS0qw6fH19YW9vb3L/zTV48GDj1zKZDB4eHsZ9NfaaNiQsLMz4dc338/LlywAAg8GAf/zjHxg0aBCcnZ1hZ2eHvXv3Ii0trdn1JyUloaSkBOPHjzfWaGdnh88//7xOjcHBwbVuT58+HQqFAj/++COA6mkcY8eOha+vb7PrICJpccY7EXVoMpkMoijWart93mlxcTGCgoLw5Zdf1tlWrVYDqP6z8bPPPos9e/Zg69ateOWVV7B//36MGDGiVWu1tLSsU3vNnEt7e3ucPn0asbGx2LdvH1auXInXX38dJ06caPLKAg3tvzVrbcpr2hKrV6/G2rVrER0djUGDBsHW1hbPPfdco9MVTKmZn7xr1y54e3vXuk+pVNa6ffsHEACwsrLC7Nmz8emnn+Jvf/sbtmzZYpyuQETmhUGWiCTRu3dvWFpa4o8//kCPHj0AADdu3EBiYiJGjx5t7KdWq2uNoF65cqXWSOawYcOwdetWuLm5wcHBod7HGzp0KIYOHYoVK1YgLCwMW7ZswYgRI2BlZQWDwdBgrX5+fti8eTP0er0xFB09ehRyubxZI8gWFhYIDw9HeHg4XnvtNTg5OeHgwYP429/+1uR9NEdTnpspTX1NTfn999/rfD/9/PwAVL9m06ZNw6OPPgqg+mStxMRE+Pv7N1izlZUVANRq9/f3h1KpRFpaWq2fl6Z6/PHHMXDgQHzwwQeoqqpqs+8BEbUtTi0gIknY2dlh/vz5ePHFF3Hw4EHExcWZPNlm3LhxWL9+Pc6cOYOTJ09i4cKFtUYTH3nkEbi6umLatGn45ZdfkJKSgtjYWDz77LO4du0aUlJSsGLFChw7dgxXr17Fvn37cOXKFWO48vX1RUpKCs6ePYvc3FyUl5fXqfWRRx6BSqXCnDlzEBcXh0OHDmHx4sWYNWsW3N3dm/R8d+7ciffffx9nz57F1atX8fnnn0MQhGZPpWgOX19fFBcXIyYmBrm5uU2eytDYa9qQN954AzExMcbvp6urK6ZPnw4A6Nu3L/bv34/ffvsNly9fxpNPPons7Ow6Nf/xxx9ITU1Fbm4uBEGAj48PZDIZdu7cCa1Wi+LiYtjb2+OFF17A0qVL8dlnnyE5ORmnT5/GunXr8NlnnzX6HP38/DBixAgsW7YMM2fO5FJnRGaKQZaIJLN69WrcfffdmDJlCsLDwzFq1CgEBQXV6rNmzRpoNBrcfffdePjhh/HCCy/AxsbGeL+NjQ2OHDmCHj164G9/+xv8/Pwwf/58lJWVwcHBATY2NoiPj8cDDzyAfv36YcGCBXjmmWfw5JNPAgAeeOABTJgwAWPHjoVara61HNTtj7F3717k5+dj+PDhePDBB3Hvvfdi/fr1TX6uTk5O+OGHHzBu3Dj4+flh48aN+OqrrxAQENDCV69xI0eOxMKFCzFjxgyo1Wq8/fbbTdqusde0IW+++SaWLFmCoKAgZGVl4aeffjKOqL7yyisYNmwYIiIiMGbMGHh4eBhDbo0XXngBCoUC/v7+UKvVSEtLg7e3N1atWoXly5fD3d3duFrEP/7xD7z66quIioqCn58fJkyYgF27dqFnz55Nep7z589HRUUFHnvssSb1J6KORyb+dfIZEZGExowZg8DAwFa/hCrRX/3jH//At99+i/Pnz0tdChG1EEdkiYioSykuLkZcXBzWr1+PxYsXS10OEd0BBlkiIupSFi1ahKCgIIwZM4bTCojMHKcWEBEREZFZ4ogsEREREZklBlkiIiIiMksMskRERERklhhkiYiIiMgsMcgSERERkVlikCUiIiIis8QgS0RERERmiUGWiIiIiMwSgywRERERmaX/D2803/Qo0UTcAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 700x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from scipy.stats import binomtest\n",
    "rng = np.random.default_rng(1)\n",
    "\n",
    "def power(n_q, p_ab=0.05, p_ba=0.12, sims=1000):\n",
    "    hits = 0\n",
    "    for _ in range(sims):\n",
    "        ab, ba, _ = rng.multinomial(n_q, [p_ab, p_ba, 1 - p_ab - p_ba])\n",
    "        if ab + ba and binomtest(int(ba), int(ab + ba), 0.5).pvalue < 0.05:\n",
    "            hits += 1\n",
    "    return hits / sims\n",
    "\n",
    "sizes = [20, 40, 100, 200, 400, 800]\n",
    "pw = [power(n) for n in sizes]\n",
    "for n_q, p in zip(sizes, pw):\n",
    "    print(f\"{n_q:4d} questions: a true 7-point improvement is detected {p:5.0%} of the time\")\n",
    "fig, ax = plt.subplots(figsize=(7, 3)); ax.plot(sizes, pw, \"o-\", color=\"#2a7ab0\"); ax.axhline(0.8, ls=\"--\", color=\"grey\")\n",
    "ax.set_xscale(\"log\"); ax.set_xlabel(\"questions in the battery\"); ax.set_ylabel(\"power\"); ax.grid(alpha=.3); plt.tight_layout(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ddefb2",
   "metadata": {},
   "source": [
    "El objetivo estándar es una potencia del 80 %. Una batería de cuarenta preguntas detecta una mejora de siete\n",
    "puntos con mucha menos frecuencia, así que las diferencias de unas pocas preguntas en una batería de este\n",
    "tamaño deben presentarse como *\"sin diferencia detectable\"*, no como una victoria. De ahí salen dos reglas\n",
    "prácticas: **haz crecer la batería** siempre que una decisión dependa de una diferencia pequeña (el tráfico de\n",
    "producción es la mejor fuente de preguntas nuevas, que es justo lo que automatiza el módulo 08), y **mira los\n",
    "fallos uno por uno**, porque un puñado de fallos entendidos enseña más que una décima de precisión.\n",
    "\n",
    "## 8. Antes y después, con preguntas que el arreglo no ha visto\n",
    "\n",
    "Ahora reparamos los fallos que muestra el conjunto 1 y medimos en ambos conjuntos. Dos arreglos, ambos\n",
    "encontrados leyendo el conjunto 1:\n",
    "\n",
    "1. **Guarda de entidades**: se rechaza una pregunta que nombra una turbina que no existe (`HV99`). Esto\n",
    "   resuelve una trampa.\n",
    "2. **Expansión de vocabulario**: añadir las palabras del propio manual junto a las del usuario (*boats* con\n",
    "   *vessels*, *qualification* con *certificate*, *ashore* con *onshore export*...), tomadas de los fallos del\n",
    "   conjunto 1.\n",
    "\n",
    "La prueba honesta es el **conjunto 2**, escrito después de diseñar los arreglos: no se usó para elegirlos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a8f73729",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:17:32.016350Z",
     "iopub.status.busy": "2026-11-17T21:17:32.016020Z",
     "iopub.status.idle": "2026-11-17T21:17:33.808606Z",
     "shell.execute_reply": "2026-11-17T21:17:33.806870Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "set 1 (used to design the fixes)\n",
      "  system  correct  of   hit@1    MRR  refusal rec\n",
      "       A       15  20    0.81   0.90         0.50\n",
      "   A+fix       18  20    0.94   0.96         0.75\n",
      "       B       16  20    0.94   0.97         0.25\n",
      "   B+fix       18  20    1.00   1.00         0.50\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "set 2 (written after, held out)\n",
      "  system  correct  of   hit@1    MRR  refusal rec\n",
      "       A       10  20    0.62   0.70         0.00\n",
      "   A+fix       12  20    0.69   0.75         0.25\n",
      "       B       12  20    0.75   0.86         0.00\n",
      "   B+fix       14  20    0.81   0.90         0.25\n",
      "\n"
     ]
    }
   ],
   "source": [
    "SYN = {\"boats\": \"vessels\", \"boat\": \"vessel\", \"qualification\": \"certificate\", \"ashore\": \"onshore export\", \"alone\": \"lone\",\n",
    "       \"blackout\": \"grid loss\", \"dolphin\": \"marine mammal\", \"autumn\": \"october march migration\", \"injury\": \"injury near miss\"}\n",
    "def expand(question):\n",
    "    extra = [v for k, v in SYN.items() if re.search(rf\"\\b{k}\\b\", question, re.I)]\n",
    "    return question + \" \" + \" \".join(extra)\n",
    "\n",
    "B_fix = build_system(hybrid, lambda q, e: bm25.scores(e).max() < 1.0 and sem(e).max() < 0.30, expand=expand, entity_guard=True)\n",
    "A_fix = build_system(bm25.scores, lambda q, e: bm25.scores(e).max() < 1.0, expand=expand, entity_guard=True)\n",
    "\n",
    "systems = {\"A\": A, \"A+fix\": A_fix, \"B\": B, \"B+fix\": B_fix}\n",
    "for label, battery in [(\"set 1 (used to design the fixes)\", SET1), (\"set 2 (written after, held out)\", SET2)]:\n",
    "    res = {n: run_all(s, battery) for n, s in systems.items()}\n",
    "    m = metrics(res, battery)\n",
    "    print(label)\n",
    "    print(f\"{'system':>8s}{'correct':>9s}{'of':>4s}{'hit@1':>8s}{'MRR':>7s}{'refusal rec':>13s}\")\n",
    "    for n, v in m.items():\n",
    "        print(f\"{n:>8s}{int(round(v['correct'] * len(battery))):9d}{len(battery):4d}{v['hit1']:8.2f}{v['mrr']:7.2f}{v['ref_rec']:13.2f}\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6814ff60",
   "metadata": {},
   "source": [
    "Compara los dos bloques como lo hizo la evaluación real (el conjunto 1 antes y después de sus arreglos, y luego\n",
    "el conjunto 2 escrito después). La mejora en el **conjunto 1** es en parte un artefacto: diseñamos los arreglos\n",
    "mirándolo (A pasa de 15 a 18 correctas, B de 16 a 18). Los números del **conjunto 2** son los honestos: ambos\n",
    "sistemas ganan allí también dos preguntas (A de 10 a 12, B de 12 a 14), así que la reparación generalizó al\n",
    "menos un poco. Pero dos preguntas de veinte están dentro del ruido de la sección 7, y la mayoría de los fallos\n",
    "restantes siguen sin tocarse: las trampas sin un identificador que comprobar (*stock price*, *how many people\n",
    "work on the farm*) y las paráfrasis cuyas palabras no están en nuestra pequeña lista de sinónimos. Digan lo que\n",
    "digan las cifras, preséntalas tal como son. Un arreglo que ayuda en el conjunto 1 y no hace nada en el\n",
    "conjunto 2 solo ha memorizado el conjunto 1.\n",
    "\n",
    "Como referencia, el agente real del parque eólico evaluado así pasó de **5 de 15** a **13 de 15** correctas en\n",
    "su primer conjunto tras los arreglos, y en un segundo conjunto de 20 preguntas escritas después (con trampas\n",
    "como una turbina que no existe y un resultado nunca calculado) de **12 de 20** a **20 de 20**. El segundo\n",
    "conjunto importó más que el primero: es lo que mostró que los arreglos eran reales.\n",
    "\n",
    "## Ejercicios\n",
    "\n",
    "1. Añade cinco preguntas tuyas al conjunto 2 (una trampa, dos paráfrasis, un identificador, una directa).\n",
    "   Vuelve a ejecutar la sección 8. ¿Cambia la conclusión?\n",
    "2. Añade tolerancia del estilo `10 minutes?` a un patrón y rompe otro: escribe un patrón demasiado débil y uno\n",
    "   demasiado estricto, y encuéntralos con la lista de discrepancias de la sección 4.\n",
    "3. En `faithfulness`, cambia el umbral de apoyo de 0.6 a 0.3 y a 0.9. ¿Qué muestras de las seis cambian de\n",
    "   veredicto y qué ajuste usarías para un manual de seguridad?\n",
    "4. Cambia `p_ab` y `p_ba` en `power` a una mejora de 3 puntos. ¿Qué tamaño debe tener ahora la batería?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
