{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03398bdf",
   "metadata": {},
   "source": [
    "# 12 · Un corpus real: nueve discursos del Estado de la Unión\n",
    "\n",
    "El proyecto final usó un corpus que **escribimos nosotros**, así que conocíamos cada respuesta y cada defecto. Los\n",
    "documentos reales no son así: son largos, retóricos, llenos de temas repetidos y escritos por personas distintas en\n",
    "años distintos. Este módulo repite todo el método con **nueve discursos del Estado de la Unión** (de 1950 a 2006),\n",
    "discursos de dominio público del gobierno de Estados Unidos, incluidos en el curso para que no haya que descargar nada.\n",
    "\n",
    "Misma tubería, misma disciplina (conjunto de desarrollo, batería y conjunto retenido), material distinto. La idea es ver\n",
    "qué conclusiones del proyecto final **sobreviven** y cuáles no.\n",
    "\n",
    "**Aprenderás**\n",
    "1. Qué cambia cuando el corpus es real: el tamaño del fragmento, los metadatos, el orador y el año como parte de la pregunta\n",
    "2. A elegir el tamaño del fragmento y el peso de BM25 en un conjunto de desarrollo, y por qué una batería pequeña puede contradecirlo\n",
    "3. Por qué una pregunta sobre algo que **nunca se dijo** es el caso más difícil, y una regla de rechazo que lo resuelve\n",
    "4. Arreglos que viven del **lado de la consulta**, porque no se puede editar un discurso histórico (el bucle del supervisor del módulo 08 cambia de forma)\n",
    "5. A informar de un resultado modesto con honestidad, con intervalos de confianza y un conjunto retenido"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e00aba9",
   "metadata": {},
   "source": [
    "### Imagen: en qué se diferencia la tubería con un corpus real\n",
    "\n",
    "El proyecto final arregló sus fallos **editando la documentación**. Un discurso no se puede editar, así que cada arreglo\n",
    "pasa al lado que sí controlamos: la pregunta, el filtro de recuperación y la regla de rechazo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "74b71905",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:31:05.943504Z",
     "iopub.status.busy": "2026-11-17T21:31:05.942912Z",
     "iopub.status.idle": "2026-11-17T21:31:06.011955Z",
     "shell.execute_reply": "2026-11-17T21:31:06.009726Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 820 270\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"820\" height=\"270\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"410.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Documentos reales: la misma tubería, arreglos del lado de la consulta</text><rect x=\"20\" y=\"60\" width=\"150\" height=\"60\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"95.0\" y=\"86.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Discursos</text><text x=\"95.0\" y=\"103.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">9 textos, 1950-2006</text><line x1=\"170\" y1=\"90\" x2=\"220\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"195.0\" y=\"84.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">dividir</text><rect x=\"220\" y=\"60\" width=\"150\" height=\"60\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"295.0\" y=\"86.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Fragmentos</text><text x=\"295.0\" y=\"103.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">~50 palabras + etiqueta</text><line x1=\"370\" y1=\"90\" x2=\"420\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"395.0\" y=\"84.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">indexar</text><rect x=\"420\" y=\"60\" width=\"150\" height=\"60\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"495.0\" y=\"86.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Recuperador</text><text x=\"495.0\" y=\"103.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">BM25 + embeddings</text><line x1=\"570\" y1=\"90\" x2=\"620\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"595.0\" y=\"84.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">top 3</text><rect x=\"620\" y=\"60\" width=\"180\" height=\"60\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"710.0\" y=\"86.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Respuesta o rechazo</text><text x=\"710.0\" y=\"103.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">(frase citada)</text><rect x=\"220\" y=\"160\" width=\"150\" height=\"54\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"295.0\" y=\"191.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Arreglos de la consulta</text><line x1=\"295\" y1=\"160\" x2=\"295\" y2=\"122\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"420\" y=\"160\" width=\"150\" height=\"54\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"495.0\" y=\"191.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Filtro de orador/año</text><line x1=\"495\" y1=\"160\" x2=\"495\" y2=\"122\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"620\" y=\"160\" width=\"180\" height=\"54\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"710.0\" y=\"191.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Rechazo: cobertura</text><line x1=\"710\" y1=\"160\" x2=\"710\" y2=\"122\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"20\" y=\"250\" text-anchor=\"start\" font-size=\"12\" fill=\"#333\">El texto real no se edita: el bucle de mejora actúa sobre la pregunta, el filtro y la regla de rechazo.</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import SVG, display\n",
    "import html as _h\n",
    "\n",
    "_COL = {  # fill, stroke  (light cards with dark text: readable in light and dark themes)\n",
    "    \"gold\": (\"#fff4c2\", \"#b89d28\"), \"blue\": (\"#dbeafe\", \"#3b82f6\"), \"green\": (\"#dcfce7\", \"#16a34a\"),\n",
    "    \"red\": (\"#fee2e2\", \"#dc2626\"), \"grey\": (\"#f1f1f4\", \"#8a8aa0\"), \"violet\": (\"#ede9fe\", \"#7c3aed\"),\n",
    "}\n",
    "\n",
    "def diagram(width, height, items, title=None):\n",
    "    # items: (\"box\", x, y, w, h, \"text\\nmore text\", colour) | (\"arrow\", x1, y1, x2, y2, \"label\") |\n",
    "    #        (\"text\", x, y, \"text\", size, \"start|middle|end\") | (\"note\", x, y, w, h, \"text\")\n",
    "    out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 {width} {height}\" width=\"100%\" '\n",
    "           f'font-family=\"Helvetica, Arial, sans-serif\">',\n",
    "           '<defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" '\n",
    "           'orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs>',\n",
    "           f'<rect width=\"{width}\" height=\"{height}\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/>']\n",
    "    if title:\n",
    "        out.append(f'<text x=\"{width / 2}\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">{_h.escape(title)}</text>')\n",
    "    for it in items:\n",
    "        kind = it[0]\n",
    "        if kind == \"box\":\n",
    "            _, x, y, w, h, text, col = it\n",
    "            fill, stroke = _COL[col]\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"9\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"1.6\"/>')\n",
    "            lines = text.split(\"\\n\")\n",
    "            y0 = y + h / 2 - (len(lines) - 1) * 8.5 + 4.5\n",
    "            for k, ln in enumerate(lines):\n",
    "                bold = ' font-weight=\"700\"' if k == 0 and len(lines) > 1 else \"\"\n",
    "                out.append(f'<text x=\"{x + w / 2}\" y=\"{y0 + k * 17}\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\"{bold}>{_h.escape(ln)}</text>')\n",
    "        elif kind == \"arrow\":\n",
    "            _, x1, y1, x2, y2, label = it\n",
    "            out.append(f'<line x1=\"{x1}\" y1=\"{y1}\" x2=\"{x2}\" y2=\"{y2}\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/>')\n",
    "            if label:\n",
    "                out.append(f'<text x=\"{(x1 + x2) / 2}\" y=\"{(y1 + y2) / 2 - 6}\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">{_h.escape(label)}</text>')\n",
    "        elif kind == \"text\":\n",
    "            _, x, y, text, size, anchor = it\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x}\" y=\"{y + k * (size + 4)}\" text-anchor=\"{anchor}\" font-size=\"{size}\" fill=\"#333\">{_h.escape(ln)}</text>')\n",
    "        elif kind == \"note\":\n",
    "            _, x, y, w, h, text = it\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/>')\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x + 10}\" y=\"{y + 20 + k * 16}\" font-size=\"12\" fill=\"#444\">{_h.escape(ln)}</text>')\n",
    "    out.append(\"</svg>\")\n",
    "    display(SVG(\"\".join(out)))\n",
    "\n",
    "diagram(820, 270, [\n",
    "    (\"box\", 20, 60, 150, 60, \"Discursos\\n9 textos, 1950-2006\", \"grey\"),\n",
    "    (\"arrow\", 170, 90, 220, 90, \"dividir\"),\n",
    "    (\"box\", 220, 60, 150, 60, \"Fragmentos\\n~50 palabras + etiqueta\", \"blue\"),\n",
    "    (\"arrow\", 370, 90, 420, 90, \"indexar\"),\n",
    "    (\"box\", 420, 60, 150, 60, \"Recuperador\\nBM25 + embeddings\", \"blue\"),\n",
    "    (\"arrow\", 570, 90, 620, 90, \"top 3\"),\n",
    "    (\"box\", 620, 60, 180, 60, \"Respuesta o rechazo\\n(frase citada)\", \"gold\"),\n",
    "    (\"box\", 220, 160, 150, 54, \"Arreglos de la consulta\", \"violet\"),\n",
    "    (\"arrow\", 295, 160, 295, 122, \"\"),\n",
    "    (\"box\", 420, 160, 150, 54, \"Filtro de orador/año\", \"violet\"),\n",
    "    (\"arrow\", 495, 160, 495, 122, \"\"),\n",
    "    (\"box\", 620, 160, 180, 54, \"Rechazo: cobertura\", \"violet\"),\n",
    "    (\"arrow\", 710, 160, 710, 122, \"\"),\n",
    "    (\"text\", 20, 250, \"El texto real no se edita: el bucle de mejora actúa sobre la pregunta, el filtro y la regla de rechazo.\", 12, \"start\"),\n",
    "], title=\"Documentos reales: la misma tubería, arreglos del lado de la consulta\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1577a7a",
   "metadata": {},
   "source": [
    "## 1. El corpus\n",
    "\n",
    "Nueve discursos, uno por época: Truman (1950), Eisenhower (1957), Kennedy (1962), Johnson (1966), Nixon (1974),\n",
    "Reagan (1982), Clinton (1994) y G. W. Bush (2002 y 2006). Proceden del corpus `state_union` que distribuye NLTK, que\n",
    "reproduce los textos del American Presidency Project; al ser obras del gobierno federal, son de dominio público.\n",
    "\n",
    "Dividimos cada discurso en **fragmentos** de unas 50 palabras formados por frases completas, y ponemos una etiqueta delante de\n",
    "cada fragmento (`1974 address by President Nixon.`). Si la etiqueta ayuda se mide en la sección 3, y la respuesta es menos\n",
    "clara de lo que nos gustaría."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "597d4c63",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:31:06.017256Z",
     "iopub.status.busy": "2026-11-17T21:31:06.016731Z",
     "iopub.status.idle": "2026-11-17T21:31:17.111679Z",
     "shell.execute_reply": "2026-11-17T21:31:17.109415Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patito/Developments/academy/rag-systems/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 speeches, 2399 sentences, 763 chunks of about 50 words\n",
      "speech               words  chunks\n",
      "1950-Truman           5122      81\n",
      "1957-Eisenhower       4097      65\n",
      "1962-Kennedy          6629     102\n",
      "1966-Johnson          5397      84\n",
      "1974-Nixon            5199      80\n",
      "1982-Reagan           5202      80\n",
      "1994-Clinton          7382     117\n",
      "2002-GWBush           3941      63\n",
      "2006-GWBush           5576      91\n"
     ]
    }
   ],
   "source": [
    "import os, re, math, glob, urllib.request\n",
    "from collections import Counter, defaultdict\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "SPEECHES = [\"1950-Truman\", \"1957-Eisenhower\", \"1962-Kennedy\", \"1966-Johnson\", \"1974-Nixon\",\n",
    "            \"1982-Reagan\", \"1994-Clinton\", \"2002-GWBush\", \"2006-GWBush\"]\n",
    "DATA = next((p for p in (\"data/sotu\", \"../data/sotu\") if os.path.isdir(p)), \"data/sotu\")\n",
    "os.makedirs(DATA, exist_ok=True)\n",
    "for name in SPEECHES:                       # solo hace falta si descargaste este notebook por separado\n",
    "    path = f\"{DATA}/{name}.txt\"\n",
    "    if not os.path.exists(path):\n",
    "        urllib.request.urlretrieve(f\"https://raw.githubusercontent.com/fborbon/academy/main/rag-systems/data/sotu/{name}.txt\", path)\n",
    "\n",
    "def read_sentences(name):\n",
    "    paras = [p.strip() for p in open(f\"{DATA}/{name}.txt\", encoding=\"utf-8\").read().split(\"\\n\")\n",
    "             if len(p.strip()) > 60 and not p.strip().startswith(\"Note:\") and not p.strip().isupper()]\n",
    "    return [s for p in paras for s in re.split(r\"(?<=[.!?])\\s+\", p) if s]\n",
    "\n",
    "def make_chunks(target_words):\n",
    "    # consecutive whole sentences until the chunk reaches target_words; the speech, the year and the speaker travel with it\n",
    "    out = []\n",
    "    for name in SPEECHES:\n",
    "        year, pres = name.split(\"-\")\n",
    "        cur, n, k = [], 0, 1\n",
    "        for s in read_sentences(name):\n",
    "            cur.append(s); n += len(s.split())\n",
    "            if n >= target_words:\n",
    "                out.append(dict(id=f\"{year}-{k:03d}\", year=year, pres=pres, text=\" \".join(cur))); cur, n, k = [], 0, k + 1\n",
    "        if cur:\n",
    "            out.append(dict(id=f\"{year}-{k:03d}\", year=year, pres=pres, text=\" \".join(cur)))\n",
    "    return out\n",
    "\n",
    "chunks = make_chunks(50)\n",
    "print(f\"{len(SPEECHES)} speeches, {sum(len(read_sentences(n)) for n in SPEECHES)} sentences, {len(chunks)} chunks of about 50 words\")\n",
    "print(f\"{'speech':<18}{'words':>8}{'chunks':>8}\")\n",
    "for n in SPEECHES:\n",
    "    print(f\"{n:<18}{sum(len(s.split()) for s in read_sentences(n)):>8}{sum(c['year'] == n[:4] for c in chunks):>8}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49d0d8d4",
   "metadata": {},
   "source": [
    "## 2. Tres conjuntos de preguntas, escritas para corregirse automáticamente\n",
    "\n",
    "No podemos pedir a una persona que califique cada respuesta, así que cada pregunta lleva una **clave**: una frase corta que aparece en\n",
    "la frase que la responde, en un discurso concreto. Una respuesta es **correcta** cuando cita un fragmento de ese discurso\n",
    "**y** su frase contiene la clave. Una pregunta sin respuesta solo se responde bien con un rechazo.\n",
    "\n",
    "* **conjunto de desarrollo (18):** se usa solo para elegir el tamaño del fragmento, el peso de BM25 y la regla de rechazo. Tiene cuatro preguntas sin respuesta.\n",
    "* **batería (28):** las preguntas sobre las que informamos, 22 con respuesta y 6 sin ella, escritas antes de los arreglos.\n",
    "* **conjunto retenido (16):** escrito **después** de diseñar los arreglos, como en el módulo 07, para comprobar si generalizan.\n",
    "\n",
    "Tipos de pregunta: `direct` (palabras cercanas al texto), `paraphrase` (palabras distintas), `who` (pregunta por el orador o el año)\n",
    "y `trap` (otro discurso dice algo parecido). Las preguntas sin respuesta son las peligrosas: el tema suele aparecer en\n",
    "**otro** discurso, o cerca en el mismo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dfe4956b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:31:17.117631Z",
     "iopub.status.busy": "2026-11-17T21:31:17.116917Z",
     "iopub.status.idle": "2026-11-17T21:31:17.167926Z",
     "shell.execute_reply": "2026-11-17T21:31:17.165899Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev       18 questions: {'direct': 6, 'paraphrase': 6, 'who': 2, 'unanswerable': 4}\n",
      "battery   28 questions: {'direct': 7, 'trap': 1, 'paraphrase': 8, 'who': 6, 'unanswerable': 6}\n",
      "held-out  16 questions: {'paraphrase': 6, 'direct': 5, 'who': 1, 'unanswerable': 4}\n"
     ]
    }
   ],
   "source": [
    "# (id, pregunta, año del discurso, clave en la frase que responde, tipo); año None = nada en el corpus la responde\n",
    "DEV = [\n",
    " (\"d01\", \"How many fewer pages did the register of regulations have in 1981 than in 1980?\", \"1982\", \"23,000 fewer pages\", \"direct\"),\n",
    " (\"d02\", \"How many reservists were called up for service?\", \"1962\", \"155,000 members\", \"paraphrase\"),\n",
    " (\"d03\", \"By how much will the federal workforce shrink?\", \"1994\", \"252,000 people\", \"paraphrase\"),\n",
    " (\"d04\", \"What does the military campaign cost each day?\", \"2002\", \"$30 million a day\", \"paraphrase\"),\n",
    " (\"d05\", \"How many people of the post-war generation are about to turn sixty?\", \"2006\", \"78 million baby boomers\", \"paraphrase\"),\n",
    " (\"d06\", \"Has the income of the average family grown over the last half century?\", \"1950\", \"buying power has doubled\", \"paraphrase\"),\n",
    " (\"d07\", \"How many troops are in Vietnam tonight?\", \"1966\", \"190,000 men\", \"direct\"),\n",
    " (\"d08\", \"What share of the budget did defense take in 1969?\", \"1974\", \"44 percent in 1969\", \"direct\"),\n",
    " (\"d09\", \"How much has been committed to rebuild New Orleans?\", \"2006\", \"$85 billion\", \"direct\"),\n",
    " (\"d10\", \"How many people is the administration serving?\", \"1957\", \"170 million people\", \"direct\"),\n",
    " (\"d11\", \"How many Americans lack any health insurance at some point in the year?\", \"1994\", \"58 million Americans\", \"paraphrase\"),\n",
    " (\"d12\", \"How many meals a day does the government subsidize?\", \"1982\", \"95 million meals\", \"direct\"),\n",
    " (\"d13\", \"Who announced the Advanced Energy Initiative?\", \"2006\", \"Advanced Energy Initiative\", \"who\"),\n",
    " (\"d14\", \"Which president asked for volunteers to serve the nation?\", \"2002\", \"USA Freedom Corps\", \"who\"),\n",
    " (\"d15\", \"What did Johnson say about the Internet?\", None, None, \"unanswerable\"),\n",
    " (\"d16\", \"What did Eisenhower say about the space shuttle?\", None, None, \"unanswerable\"),\n",
    " (\"d17\", \"What did Bush say about the stock market crash?\", None, None, \"unanswerable\"),\n",
    " (\"d18\", \"What did Clinton say about September the 11th?\", None, None, \"unanswerable\"),\n",
    "]\n",
    "BATTERY = [\n",
    " (\"b01\", \"How much did Kennedy propose for the Alliance for Progress?\", \"1962\", \"fund of $3 billion\", \"direct\"),\n",
    " (\"b02\", \"How fast would the proposed supersonic airplane fly?\", \"1966\", \"three times the speed of sound\", \"direct\"),\n",
    " (\"b03\", \"How much federal money is planned for energy research over five years?\", \"1974\", \"$10 billion in Federal funds\", \"trap\"),\n",
    " (\"b04\", \"What increase in clean-energy research was announced?\", \"2006\", \"22-percent increase\", \"paraphrase\"),\n",
    " (\"b05\", \"What share of Middle East oil imports should be replaced by 2025?\", \"2006\", \"75 percent of our oil imports\", \"direct\"),\n",
    " (\"b06\", \"How far had interest rates fallen by early 1982?\", \"1982\", \"15\\\\3/4\\\\ percent\", \"paraphrase\"),\n",
    " (\"b07\", \"By when will the states control the federal grant programs?\", \"1982\", \"By 1988 the States\", \"direct\"),\n",
    " (\"b08\", \"What happened to crime in Houston?\", \"1994\", \"crime rate dropped 17 percent\", \"direct\"),\n",
    " (\"b09\", \"How much spending did Congress cut?\", \"1994\", \"$255 billion in spending\", \"direct\"),\n",
    " (\"b10\", \"How much time were Americans asked to give to service?\", \"2002\", \"4,000 hours\", \"paraphrase\"),\n",
    " (\"b11\", \"How many volunteers will the new corps recruit?\", \"2002\", \"200,000 new volunteers\", \"paraphrase\"),\n",
    " (\"b12\", \"What was the nation's yearly output in 1950?\", \"1950\", \"$255 billion a year\", \"paraphrase\"),\n",
    " (\"b13\", \"What did the president propose against road accidents?\", \"1966\", \"Highway Safety Act\", \"paraphrase\"),\n",
    " (\"b14\", \"What tax incentive was proposed to modernize equipment?\", \"1962\", \"8 percent tax credit\", \"paraphrase\"),\n",
    " (\"b15\", \"Which housing law is credited with helping low-income families?\", \"1950\", \"Housing Act of 1949\", \"paraphrase\"),\n",
    " (\"b16\", \"How much child support goes unpaid?\", \"1994\", \"$34 billion\", \"direct\"),\n",
    " (\"b17\", \"Which president said the Union was at peace with every nation?\", \"1974\", \"at peace with every nation\", \"who\"),\n",
    " (\"b18\", \"Who proposed the USA Freedom Corps?\", \"2002\", \"USA Freedom Corps\", \"who\"),\n",
    " (\"b19\", \"Which president announced Project Independence?\", \"1974\", \"Project Independence\", \"who\"),\n",
    " (\"b20\", \"Who joked that the baby boomers include President Clinton?\", \"2006\", \"President Clinton\", \"who\"),\n",
    " (\"b21\", \"In which year were the hijackers said to be trained in Afghanistan's camps?\", \"2002\", \"trained in Afghanistan\", \"who\"),\n",
    " (\"b22\", \"Who said the 17-year rise in crime had been stopped?\", \"1974\", \"17-year rise in crime\", \"who\"),\n",
    " (\"u01\", \"What did the president say about the Internet?\", None, None, \"unanswerable\"),\n",
    " (\"u02\", \"What did Reagan say about the Challenger disaster?\", None, None, \"unanswerable\"),\n",
    " (\"u03\", \"What did Kennedy say about Cuba?\", None, None, \"unanswerable\"),\n",
    " (\"u04\", \"What did Clinton say about the budget surplus?\", None, None, \"unanswerable\"),\n",
    " (\"u05\", \"What did Bush say about Hurricane Katrina?\", None, None, \"unanswerable\"),\n",
    " (\"u06\", \"How did the 2010 address describe health care reform?\", None, None, \"unanswerable\"),\n",
    "]\n",
    "HELDOUT = [\n",
    " (\"e01\", \"How many adults cannot read well enough to function?\", \"1962\", \"Eight million adult Americans\", \"paraphrase\"),\n",
    " (\"e02\", \"What change did Johnson propose to the term of House members?\", \"1966\", \"4-year term for Members of the House\", \"direct\"),\n",
    " (\"e03\", \"How many people live in the country, according to the 1974 address?\", \"1974\", \"211 million people\", \"direct\"),\n",
    " (\"e04\", \"By how many will the federal work force be reduced?\", \"1982\", \"75,000\", \"direct\"),\n",
    " (\"e05\", \"How many more police officers will be funded?\", \"1994\", \"100,000 of them\", \"paraphrase\"),\n",
    " (\"e06\", \"What did the president ask Congress to protect for retirement savings?\", \"2002\", \"401K and pension plans\", \"paraphrase\"),\n",
    " (\"e07\", \"Who was thanked for 24 years of service on the court?\", \"2006\", \"Sandra Day O'Connor\", \"who\"),\n",
    " (\"e08\", \"How does job creation compare with Japan and the European Union?\", \"2006\", \"more than Japan and the European Union combined\", \"paraphrase\"),\n",
    " (\"e09\", \"What share of government spending pays for past wars?\", \"1950\", \"More than 70 percent\", \"paraphrase\"),\n",
    " (\"e10\", \"What happened to farm income in 1961?\", \"1962\", \"billion dollar rise in farm income\", \"direct\"),\n",
    " (\"e11\", \"How many private peace talks took place in 1965?\", \"1966\", \"300 private talks\", \"direct\"),\n",
    " (\"e12\", \"How much did the food stamp program grow in fifteen years?\", \"1982\", \"16,000 percent\", \"paraphrase\"),\n",
    " (\"v01\", \"What did Truman say about television?\", None, None, \"unanswerable\"),\n",
    " (\"v02\", \"What did Johnson say about the Berlin Wall?\", None, None, \"unanswerable\"),\n",
    " (\"v03\", \"What did Bush say about the housing bubble?\", None, None, \"unanswerable\"),\n",
    " (\"v04\", \"What did Reagan say about Gorbachev?\", None, None, \"unanswerable\"),\n",
    "]\n",
    "OFF_TOPIC = [\"What is the capital of France?\", \"How do I bake sourdough bread?\", \"Who won the football match yesterday?\", \"What is the speed of light?\"]\n",
    "\n",
    "# una clave debe existir en su discurso (y en ningún otro), o la pregunta es injusta\n",
    "for q in DEV + BATTERY + HELDOUT:\n",
    "    if q[2]:\n",
    "        in_speech = [c[\"id\"] for c in chunks if c[\"year\"] == q[2] and q[3] in c[\"text\"]]\n",
    "        elsewhere = [c[\"id\"] for c in chunks if c[\"year\"] != q[2] and q[3] in c[\"text\"]]\n",
    "        assert in_speech and not elsewhere, (q[0], in_speech, elsewhere)\n",
    "for name, S in ((\"dev\", DEV), (\"battery\", BATTERY), (\"held-out\", HELDOUT)):\n",
    "    print(f\"{name:<9}{len(S):>3} questions:\", dict(Counter(q[4] for q in S)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4476d41f",
   "metadata": {},
   "source": [
    "## 3. Recuperación: qué medir antes de construir nada\n",
    "\n",
    "Un recuperador es bueno para una pregunta si algún fragmento con la clave llega a lo más alto del ranking. Tres decisiones,\n",
    "cada una **elegida solo con el conjunto de desarrollo**: qué tamaño tiene un fragmento, si el orador y el año se escriben dentro del fragmento, y\n",
    "cuánto BM25 se mezcla con los embeddings (módulo 02)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "924cb5a3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:31:17.172817Z",
     "iopub.status.busy": "2026-11-17T21:31:17.172485Z",
     "iopub.status.idle": "2026-11-17T21:33:18.446786Z",
     "shell.execute_reply": "2026-11-17T21:33:18.445030Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights:   0%|          | 0/103 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights: 100%|██████████| 103/103 [00:00<00:00, 3036.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk size     chunks   dev hit@1  hit@3   MRR      battery hit@1  hit@3   MRR\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 25 words      1280        0.64   0.79   0.72            0.73   0.95   0.84\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 50 words       763        0.64   0.93   0.79            0.64   0.95   0.80\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 words       434        0.43   0.93   0.68            0.95   0.95   0.97\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200 words       232        0.43   0.71   0.60            0.73   0.95   0.85\n",
      "chosen on the dev set (best MRR): 50 words\n"
     ]
    }
   ],
   "source": [
    "embedder = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\", device=\"cpu\")\n",
    "STOP = set(\"a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you my me did was were has have had been will\".split())\n",
    "tokens = lambda text: [w for w in re.findall(r\"[a-z0-9]+\", text.lower()) if w not in STOP]\n",
    "\n",
    "class Index:\n",
    "    # BM25 + embeddings over a list of chunks; label=True writes the year and speaker in front of each chunk\n",
    "    def __init__(self, chunks, label=True):\n",
    "        self.chunks = chunks\n",
    "        self.texts = [(f\"{c['year']} address by President {c['pres']}. \" if label else \"\") + c[\"text\"] for c in chunks]\n",
    "        self.tf = [Counter(tokens(t)) for t in self.texts]\n",
    "        self.len = np.array([sum(d.values()) for d in self.tf], float)\n",
    "        df = Counter(w for d in self.tf for w in d)\n",
    "        n = len(self.texts)\n",
    "        self.idf = {w: math.log(1 + (n - c + 0.5) / (c + 0.5)) for w, c in df.items()}\n",
    "        self.emb = embedder.encode(self.texts, normalize_embeddings=True, batch_size=64)\n",
    "        self.avg = self.len.mean()\n",
    "    def bm25(self, q):\n",
    "        s = np.zeros(len(self.texts))\n",
    "        for w in set(tokens(q)):\n",
    "            if w in self.idf:\n",
    "                for i, d in enumerate(self.tf):\n",
    "                    f = d.get(w, 0)\n",
    "                    if f:\n",
    "                        s[i] += self.idf[w] * f * 2.5 / (f + 1.5 * (0.25 + 0.75 * self.len[i] / self.avg))\n",
    "        return s\n",
    "    def cosine(self, q):\n",
    "        return self.emb @ embedder.encode(q, normalize_embeddings=True)\n",
    "    def fused(self, q, w_bm25):\n",
    "        tot = np.zeros(len(self.texts))\n",
    "        for sc, w in ((self.bm25(q), w_bm25), (self.cosine(q), 1.0)):\n",
    "            for rank, i in enumerate(np.argsort(-sc), 1):\n",
    "                tot[i] += w / (60 + rank)\n",
    "        return tot\n",
    "\n",
    "def retrieval(idx, S, w):\n",
    "    # hit@1, hit@3, MRR for the answerable questions of S\n",
    "    ranks = []\n",
    "    for _, q, year, key, _t in S:\n",
    "        if year is None:\n",
    "            continue\n",
    "        order = list(np.argsort(-idx.fused(q, w)))\n",
    "        gold = [i for i, c in enumerate(idx.chunks) if c[\"year\"] == year and key in c[\"text\"]]\n",
    "        ranks.append(min(order.index(g) for g in gold) + 1)\n",
    "    r = np.array(ranks)\n",
    "    return (r == 1).mean(), (r <= 3).mean(), (1 / r).mean()\n",
    "\n",
    "print(f\"{'chunk size':<14}{'chunks':>7}   dev hit@1  hit@3   MRR      battery hit@1  hit@3   MRR\")\n",
    "SIZES = {}\n",
    "for target in (25, 50, 100, 200):\n",
    "    ch = make_chunks(target); idx_t = Index(ch)\n",
    "    SIZES[target] = (idx_t, retrieval(idx_t, DEV, 0.5), retrieval(idx_t, BATTERY, 0.5))\n",
    "    d, b = SIZES[target][1], SIZES[target][2]\n",
    "    print(f\"{target:>3} words{len(ch):>10}      {d[0]:>6.2f} {d[1]:>6.2f} {d[2]:>6.2f}          {b[0]:>6.2f} {b[1]:>6.2f} {b[2]:>6.2f}\")\n",
    "CHUNK_WORDS = max(SIZES, key=lambda t: (SIZES[t][1][2], -t))\n",
    "print(\"chosen on the dev set (best MRR):\", CHUNK_WORDS, \"words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92eb6e75",
   "metadata": {},
   "source": [
    "El **conjunto de desarrollo elige un tamaño de fragmento y la batería habría elegido otro**. Con 14 y 22 preguntas con respuesta\n",
    "respectivamente, una diferencia de una o dos preguntas cambia el ranking: la misma advertencia del módulo 07, ahora sobre texto real.\n",
    "Mantenemos la elección hecha en desarrollo, porque elegir con la batería sería filtrar información. Ambas medidas coinciden en que\n",
    "los fragmentos muy grandes (200 palabras, varios temas mezclados en un vector) son una mala idea para estos discursos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23aa7c1e",
   "metadata": {},
   "source": [
    "A continuación, dos elecciones más, de nuevo en desarrollo: si la etiqueta (orador y año) va delante de cada fragmento, y el peso de BM25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b676166c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:33:18.450476Z",
     "iopub.status.busy": "2026-11-17T21:33:18.450158Z",
     "iopub.status.idle": "2026-11-17T21:33:49.719179Z",
     "shell.execute_reply": "2026-11-17T21:33:49.717424Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "speaker and year written into the chunk? (BM25 weight 0.5, chunk size 50 )\n",
      "                     who questions hit@1   all dev hit@1   all battery hit@1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no label                            0.88            0.57                0.77\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "with label                          0.88            0.64                0.64\n",
      "\n",
      "BM25 weight (semantic weight = 1) on the DEV set:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  weight 0     hit@1 0.71  hit@3 0.71  MRR 0.76\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  weight 0.25  hit@1 0.79  hit@3 1.00  MRR 0.88\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  weight 0.5   hit@1 0.64  hit@3 0.93  MRR 0.79\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  weight 1     hit@1 0.64  hit@3 0.86  MRR 0.75\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  weight 2     hit@1 0.50  hit@3 0.71  MRR 0.64\n",
      "chosen BM25 weight: 0.25\n"
     ]
    }
   ],
   "source": [
    "P = SIZES[CHUNK_WORDS][0]\n",
    "labelled = P\n",
    "plain = Index(P.chunks, label=False)\n",
    "WHO = [q for q in DEV + BATTERY if q[4] == \"who\"]\n",
    "print(\"speaker and year written into the chunk? (BM25 weight 0.5, chunk size\", CHUNK_WORDS, \")\")\n",
    "print(f\"{'':<18}{'who questions hit@1':>22}{'all dev hit@1':>16}{'all battery hit@1':>20}\")\n",
    "for name, ix in ((\"no label\", plain), (\"with label\", labelled)):\n",
    "    print(f\"{name:<18}{retrieval(ix, WHO, 0.5)[0]:>22.2f}{retrieval(ix, DEV, 0.5)[0]:>16.2f}{retrieval(ix, BATTERY, 0.5)[0]:>20.2f}\")\n",
    "\n",
    "print(\"\\nBM25 weight (semantic weight = 1) on the DEV set:\")\n",
    "dev_w = {}\n",
    "for w in (0, 0.25, 0.5, 1, 2):\n",
    "    dev_w[w] = retrieval(labelled, DEV, w)\n",
    "    print(f\"  weight {w:<5} hit@1 {dev_w[w][0]:.2f}  hit@3 {dev_w[w][1]:.2f}  MRR {dev_w[w][2]:.2f}\")\n",
    "W_BM25 = max(dev_w, key=lambda w: (dev_w[w][2], -w))\n",
    "print(\"chosen BM25 weight:\", W_BM25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc0c624",
   "metadata": {},
   "source": [
    "Tres cosas que leer en esta tabla, ninguna de las que te diría un tutorial:\n",
    "\n",
    "* **La etiqueta ayuda en desarrollo y perjudica en la batería.** En las preguntas \"who\" no cambia nada. Ambos conjuntos son diminutos, así que no podemos decir que ayude; la mantenemos, como se eligió en desarrollo, y **no** afirmamos ningún beneficio.\n",
    "* **La búsqueda híbrida sí ayuda aquí**, con un peso pequeño de BM25 (0,25). En el proyecto final el mejor peso era cero. El peso correcto depende del corpus: estos discursos están llenos de cifras y nombres que BM25 encuentra exactamente. Es la lección del módulo 02 otra vez, con el signo contrario.\n",
    "* **El peso hay que medirlo, no recordarlo.** Una regla \"lo híbrido siempre es mejor\" y una regla \"lo híbrido nunca es mejor\" son ambas falsas.\n",
    "\n",
    "### Imagen: una pregunta real a través del sistema mejorado\n",
    "\n",
    "El flujo de datos de este módulo en un ejemplo. Sigue la pregunta desde la izquierda: el filtro limita la búsqueda\n",
    "al discurso que nombra la pregunta, la recuperación ordena fragmentos dentro de él, la comprobación de cobertura decide si hay algo\n",
    "que responder y, solo entonces, se elige una frase. La pregunta sin respuesta se detiene en el rechazo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8f8b087f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:33:49.723399Z",
     "iopub.status.busy": "2026-11-17T21:33:49.722932Z",
     "iopub.status.idle": "2026-11-17T21:33:49.735219Z",
     "shell.execute_reply": "2026-11-17T21:33:49.733326Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 820 300\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"820\" height=\"300\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"410.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">De una pregunta a una respuesta, o a un rechazo</text><rect x=\"20\" y=\"50\" width=\"150\" height=\"70\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"95.0\" y=\"72.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Pregunta</text><text x=\"95.0\" y=\"89.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">What did Kennedy</text><text x=\"95.0\" y=\"106.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">say about Cuba?</text><line x1=\"170\" y1=\"85\" x2=\"205\" y2=\"85\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"205\" y=\"50\" width=\"170\" height=\"70\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"290.0\" y=\"72.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">1. Filtro</text><text x=\"290.0\" y=\"89.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">orador = Kennedy</text><text x=\"290.0\" y=\"106.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">-&gt; solo el discurso de 1962</text><line x1=\"375\" y1=\"85\" x2=\"410\" y2=\"85\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"410\" y=\"50\" width=\"150\" height=\"70\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"485.0\" y=\"72.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">2. Recuperar</text><text x=\"485.0\" y=\"89.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">BM25 + embeddings</text><text x=\"485.0\" y=\"106.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">dentro de ese discurso</text><line x1=\"560\" y1=\"85\" x2=\"595\" y2=\"85\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"577.5\" y=\"79.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">top 3</text><rect x=\"595\" y=\"50\" width=\"205\" height=\"70\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"697.5\" y=\"72.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">3. Comprobar cobertura</text><text x=\"697.5\" y=\"89.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">'Cuba' no está en ninguno</text><text x=\"697.5\" y=\"106.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">de los mejores -&gt; baja</text><line x1=\"700\" y1=\"120\" x2=\"700\" y2=\"190\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"700.0\" y=\"149.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">baja</text><rect x=\"595\" y=\"190\" width=\"205\" height=\"60\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"697.5\" y=\"216.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Rechazar: nada en</text><text x=\"697.5\" y=\"233.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">el discurso de 1962</text><line x1=\"620\" y1=\"120\" x2=\"470\" y2=\"190\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"545.0\" y=\"149.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">suficiente</text><rect x=\"350\" y=\"190\" width=\"230\" height=\"60\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"465.0\" y=\"216.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">4. Elegir la mejor frase</text><text x=\"465.0\" y=\"233.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">y citar su fragmento</text><rect x=\"20\" y=\"150\" width=\"300\" height=\"100\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/><text x=\"30\" y=\"170\" font-size=\"12\" fill=\"#444\">Sin el filtro, 'Cuba' coincide con el</text><text x=\"30\" y=\"186\" font-size=\"12\" fill=\"#444\">discurso de 1966 y el primer sistema</text><text x=\"30\" y=\"202\" font-size=\"12\" fill=\"#444\">respondía con el presidente equivocado.</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diagram(820, 300, [\n",
    "    (\"box\", 20, 50, 150, 70, \"Pregunta\\nWhat did Kennedy\\nsay about Cuba?\", \"gold\"),\n",
    "    (\"arrow\", 170, 85, 205, 85, \"\"),\n",
    "    (\"box\", 205, 50, 170, 70, \"1. Filtro\\norador = Kennedy\\n-> solo el discurso de 1962\", \"violet\"),\n",
    "    (\"arrow\", 375, 85, 410, 85, \"\"),\n",
    "    (\"box\", 410, 50, 150, 70, \"2. Recuperar\\nBM25 + embeddings\\ndentro de ese discurso\", \"blue\"),\n",
    "    (\"arrow\", 560, 85, 595, 85, \"top 3\"),\n",
    "    (\"box\", 595, 50, 205, 70, \"3. Comprobar cobertura\\n'Cuba' no está en ninguno\\nde los mejores -> baja\", \"red\"),\n",
    "    (\"arrow\", 700, 120, 700, 190, \"baja\"),\n",
    "    (\"box\", 595, 190, 205, 60, \"Rechazar: nada en\\nel discurso de 1962\", \"red\"),\n",
    "    (\"arrow\", 620, 120, 470, 190, \"suficiente\"),\n",
    "    (\"box\", 350, 190, 230, 60, \"4. Elegir la mejor frase\\ny citar su fragmento\", \"blue\"),\n",
    "    (\"note\", 20, 150, 300, 100, \"Sin el filtro, 'Cuba' coincide con el\\ndiscurso de 1966 y el primer sistema\\nrespondía con el presidente equivocado.\"),\n",
    "], title=\"De una pregunta a una respuesta, o a un rechazo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc910fb1",
   "metadata": {},
   "source": [
    "## 4. El primer sistema: el método del proyecto final, sin cambios\n",
    "\n",
    "Mismo diseño que el proyecto final: recuperar, rechazar cuando el mejor coseno está por debajo de un umbral, y responder con la frase del\n",
    "fragmento principal más cercana a la pregunta. El umbral sale de las preguntas con respuesta de desarrollo y de las preguntas fuera de tema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fdcace10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:33:49.739390Z",
     "iopub.status.busy": "2026-11-17T21:33:49.738950Z",
     "iopub.status.idle": "2026-11-17T21:33:55.033640Z",
     "shell.execute_reply": "2026-11-17T21:33:55.031902Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "top cosine: answerable dev 0.42 to 0.73; off-topic 0.16 to 0.22  ->  refuse below 0.32\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev      correct  7 of 18   direct 4/6  paraphrase 1/6  who 2/2  unanswerable 0/4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "battery  correct 14 of 28   direct 5/7  trap 1/1  paraphrase 3/8  who 5/6  unanswerable 0/6\n"
     ]
    }
   ],
   "source": [
    "SENT = {}                                                    # sentence embeddings cached per chunk\n",
    "def sentences_of(i):\n",
    "    if i not in SENT:\n",
    "        s = [x for x in re.split(r\"(?<=[.!?])\\s+\", P.chunks[i][\"text\"]) if x]\n",
    "        SENT[i] = (s, embedder.encode(s, normalize_embeddings=True))\n",
    "    return SENT[i]\n",
    "\n",
    "dev_top = [P.cosine(q[1]).max() for q in DEV if q[2]]\n",
    "off_top = [P.cosine(q).max() for q in OFF_TOPIC]\n",
    "T_COS = (max(off_top) + min(dev_top)) / 2\n",
    "print(f\"top cosine: answerable dev {min(dev_top):.2f} to {max(dev_top):.2f}; off-topic {min(off_top):.2f} to {max(off_top):.2f}  ->  refuse below {T_COS:.2f}\")\n",
    "\n",
    "def answer_v1(q):\n",
    "    cos = P.cosine(q)\n",
    "    if cos.max() < T_COS:\n",
    "        return \"refuse\", None, None\n",
    "    top = int(np.argmax(P.fused(q, W_BM25)))\n",
    "    s, e = sentences_of(top)\n",
    "    return \"answer\", P.chunks[top][\"id\"], s[int(np.argmax(e @ embedder.encode(q, normalize_embeddings=True)))]\n",
    "\n",
    "def grade(q, res):\n",
    "    kind, cid, sent = res\n",
    "    if q[2] is None:\n",
    "        return \"ok\" if kind == \"refuse\" else \"wrong\"\n",
    "    if kind == \"refuse\":\n",
    "        return \"missed\"\n",
    "    return \"ok\" if cid[:4] == q[2] and q[3] in sent else \"wrong\"\n",
    "\n",
    "def report(system, S):\n",
    "    res = [(q, grade(q, system(q[1]))) for q in S]\n",
    "    by = defaultdict(lambda: [0, 0])\n",
    "    for q, g in res:\n",
    "        by[q[4]][1] += 1; by[q[4]][0] += g == \"ok\"\n",
    "    return sum(g == \"ok\" for _, g in res), len(S), dict(by), res\n",
    "\n",
    "for name, S in ((\"dev\", DEV), (\"battery\", BATTERY)):\n",
    "    ok, n, by, res = report(answer_v1, S)\n",
    "    print(f\"{name:<8} correct {ok:>2} of {n}   \" + \"  \".join(f\"{t} {a}/{b}\" for t, (a, b) in by.items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2858e55",
   "metadata": {},
   "source": [
    "El primer sistema acierta más o menos la mitad de la batería, muy por debajo del 75 % del proyecto final. Falla de dos maneras distintas, y\n",
    "necesitan arreglos distintos:\n",
    "\n",
    "* **Las preguntas sin respuesta nunca se rechazan.** El tema de \"what did Reagan say about the Challenger disaster\" está cerca de muchas frases\n",
    "  del corpus, así que el coseno máximo es tan alto como en una pregunta real. Un umbral de similitud no puede separarlas.\n",
    "* **Discurso correcto con frase equivocada, o discurso equivocado.** Varias preguntas devuelven un fragmento de otro año, o el fragmento correcto con una frase que solo\n",
    "  repite las palabras de la pregunta (\"How many men who listen to me tonight...\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "facaa873",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:33:55.037187Z",
     "iopub.status.busy": "2026-11-17T21:33:55.036857Z",
     "iopub.status.idle": "2026-11-17T21:33:57.493312Z",
     "shell.execute_reply": "2026-11-17T21:33:57.491862Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "battery failures of the first system:\n",
      "  b01 [wrong] How much did Kennedy propose for the Alliance for Progress?\n",
      "        -> 1962-065 The Alliance for Progress is being rapidly transformed from proposal to progra\n",
      "  b05 [wrong] What share of Middle East oil imports should be replaced by 20\n",
      "        -> 1957-058 But these people, whose economic strength is largely dependent on free and uni\n",
      "  b06 [wrong] How far had interest rates fallen by early 1982?\n",
      "        -> 1982-008 Interest rates reached a staggering 21\\1/2\\ percent.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  b10 [wrong] How much time were Americans asked to give to service?\n",
      "        -> 1994-070 The American people provide those of us in Government service with terrific he\n",
      "  b12 [wrong] What was the nation's yearly output in 1950?\n",
      "        -> 1950-040 If our productive power continues to increase at the same rate as it has incre\n",
      "  b14 [wrong] What tax incentive was proposed to modernize equipment?\n",
      "        -> 1982-039 While meeting all these needs, we intend to plug unwarranted tax loopholes and\n",
      "  b15 [wrong] Which housing law is credited with helping low-income families\n",
      "        -> 1950-055 To aid the middle-income families, I recommend that the Congress enact new leg\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  b17 [wrong] Which president said the Union was at peace with every nation?\n",
      "        -> 1966-072 We have carried our quest for peace to many nations and peoples because we sha\n",
      "  u01 [wrong] What did the president say about the Internet?\n",
      "        -> 1994-039 And without blinking an eye, she looked at 40 Governors, and she said, \"When m\n",
      "  u02 [wrong] What did Reagan say about the Challenger disaster?\n",
      "        -> 1982-012 There were also pessimistic predictions about the relationship between our adm\n",
      "  u03 [wrong] What did Kennedy say about Cuba?\n",
      "        -> 1966-045 In this pursuit we have defended against Communist aggression--in Korea under \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  u04 [wrong] What did Clinton say about the budget surplus?\n",
      "        -> 1994-016 We froze domestic spending and used honest budget numbers.\n",
      "  u05 [wrong] What did Bush say about Hurricane Katrina?\n",
      "        -> 2002-047 (Applause.) The American people have responded magnificently, with courage and\n",
      "  u06 [wrong] How did the 2010 address describe health care reform?\n",
      "        -> 1994-067 Now, in the coming months I hope very much to work with both Democrats and Rep\n"
     ]
    }
   ],
   "source": [
    "ok, n, by, res = report(answer_v1, BATTERY)\n",
    "print(\"battery failures of the first system:\")\n",
    "for q, g in res:\n",
    "    if g != \"ok\":\n",
    "        kind, cid, sent = answer_v1(q[1])\n",
    "        print(f\"  {q[0]} [{g}] {q[1][:62]}\\n        -> {cid} {(sent or '')[:78]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9cda2ba",
   "metadata": {},
   "source": [
    "## 5. Tres arreglos que viven del lado de la consulta\n",
    "\n",
    "Un supervisor que leyera estos fallos (módulo 08) los marcaría y pediría a una persona que arreglara la documentación. Aquí la\n",
    "documentación es un registro histórico, así que los arreglos van al sistema que la rodea. Cada uno es una regla **general**, no un parche para\n",
    "una pregunta, y cada uno tiene un parámetro que elegimos en el **conjunto de desarrollo**:\n",
    "\n",
    "1. **Filtro de orador y año.** Si la pregunta nombra a un presidente o un año, se busca solo en los discursos correspondientes. Un año sin discurso (2010) significa que no hay nada que encontrar.\n",
    "2. **Comprobación de cobertura para rechazar.** En una pregunta con respuesta, casi todas sus palabras de contenido (con un stemming ligero) aparecen en los mejores fragmentos. Con \"Challenger\", \"Katrina\" o \"Internet\" no.\n",
    "3. **Refuerzo de cantidades.** En preguntas \"how many\", \"how much\" o \"what share\", se prefiere la frase que contiene un número."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "75b95152",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:33:57.496716Z",
     "iopub.status.busy": "2026-11-17T21:33:57.496396Z",
     "iopub.status.idle": "2026-11-17T21:34:38.934969Z",
     "shell.execute_reply": "2026-11-17T21:34:38.933372Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " coverage  boost   dev correct (of 18)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        0      0     7\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        0    0.1     8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        0    0.2     8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.3      0    10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.3    0.1    11\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.3    0.2    11\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.4      0    11\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.4    0.1    12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.4    0.2    12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.5      0    11\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.5    0.1    12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.5    0.2    12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.6      0    10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.6    0.1    10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.6    0.2    10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.7      0     9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.7    0.1     9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0.7    0.2     9\n",
      "chosen on the dev set: {'coverage': 0.5, 'boost': 0.1}\n"
     ]
    }
   ],
   "source": [
    "PRES = {\"truman\": {\"1950\"}, \"eisenhower\": {\"1957\"}, \"kennedy\": {\"1962\"}, \"johnson\": {\"1966\"}, \"nixon\": {\"1974\"},\n",
    "        \"reagan\": {\"1982\"}, \"clinton\": {\"1994\"}, \"bush\": {\"2002\", \"2006\"}}\n",
    "ALL_YEARS = set(c[\"year\"] for c in P.chunks)\n",
    "GENERIC = set(\"president said say address speech tell tonight states united american americans year years much many\".split())\n",
    "QTY = re.compile(r\"\\b(how many|how much|what share|how far|how fast|how long|what increase)\\b\", re.I)\n",
    "\n",
    "def stem(w):\n",
    "    for suf in (\"ing\", \"ed\", \"es\", \"s\"):\n",
    "        if w.endswith(suf) and len(w) - len(suf) >= 4:\n",
    "            return w[:-len(suf)]\n",
    "    return w\n",
    "\n",
    "def allowed_speeches(q):\n",
    "    named = [y for n, ys in PRES.items() if n in q.lower() for y in ys]\n",
    "    years = set(re.findall(r\"\\b(19[4-9]\\d|20[01]\\d)\\b\", q))\n",
    "    allowed = set(named) if named else set(ALL_YEARS)\n",
    "    return allowed & years if years else allowed                # empty set: the question asks about a year we do not have\n",
    "\n",
    "def coverage(q, chunk_ids):\n",
    "    words = [stem(w) for w in set(tokens(q)) if w not in GENERIC and w not in PRES and not w.isdigit()]\n",
    "    if not words:\n",
    "        return 1.0\n",
    "    seen = set(stem(w) for i in chunk_ids for w in tokens(P.chunks[i][\"text\"]))\n",
    "    return sum(w in seen for w in words) / len(words)\n",
    "\n",
    "def make_system(cov, boost, use_filter=True, k=3):\n",
    "    def system(q):\n",
    "        allowed = allowed_speeches(q) if use_filter else ALL_YEARS\n",
    "        if not allowed:\n",
    "            return \"refuse\", None, None\n",
    "        sc = P.fused(q, W_BM25)\n",
    "        for i, c in enumerate(P.chunks):\n",
    "            if c[\"year\"] not in allowed:\n",
    "                sc[i] = -1\n",
    "        order = list(np.argsort(-sc))[:k]\n",
    "        if cov and coverage(q, order) < cov:\n",
    "            return \"refuse\", None, None\n",
    "        cand = [(i, s, e) for i in order for s, e in zip(*sentences_of(i))]\n",
    "        sims = np.array([c[2] @ embedder.encode(q, normalize_embeddings=True) for c in cand])\n",
    "        if boost and QTY.search(q):\n",
    "            sims = sims + boost * np.array([bool(re.search(r\"\\d|\\$|percent|million|billion\", c[1])) for c in cand])\n",
    "        i, s, _ = cand[int(np.argmax(sims))]\n",
    "        return \"answer\", P.chunks[i][\"id\"], s\n",
    "    return system\n",
    "\n",
    "print(f\"{'coverage':>9}{'boost':>7}   dev correct (of {len(DEV)})\")\n",
    "grid = {}\n",
    "for cov in (0, 0.3, 0.4, 0.5, 0.6, 0.7):\n",
    "    for boost in (0, 0.1, 0.2):\n",
    "        grid[(cov, boost)] = report(make_system(cov, boost), DEV)[0]\n",
    "        print(f\"{cov:>9}{boost:>7}   {grid[(cov, boost)]:>3}\")\n",
    "COV, BOOST = max(grid, key=lambda k: (grid[k], k[0], -k[1]))\n",
    "print(\"chosen on the dev set:\", dict(coverage=COV, boost=BOOST))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b386f6a2",
   "metadata": {},
   "source": [
    "## 6. Antes y después, en la batería y en el conjunto retenido\n",
    "\n",
    "Las preguntas retenidas se escribieron después de diseñar los arreglos y nunca se usaron para elegir ningún parámetro. El\n",
    "intervalo bootstrap (módulo 07) nos dice cuánto de cada cambio podría ser suerte con este tamaño."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "55bca490",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:34:38.938271Z",
     "iopub.status.busy": "2026-11-17T21:34:38.937930Z",
     "iopub.status.idle": "2026-11-17T21:34:56.128199Z",
     "shell.execute_reply": "2026-11-17T21:34:56.126110Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "set        questions  first system  improved   change   95% interval of the change\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev               18          0.39      0.67    +0.28   [-0.06, +0.61]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "battery           28          0.50      0.68    +0.18   [-0.04, +0.39]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "held-out          16          0.44      0.62    +0.19   [-0.06, +0.44]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAE2CAYAAACdsHHTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQ0tJREFUeJzt3Xl4TNf/B/D3ZJtsshJEIojEkqD2IBKxLyEItbZqr6W0qkppq/pFFUVLUVrbj9qXCiKxfG1JlJYQCUWCiD37vp7fH31yvx0zSWayTUbfr+fJ87jnnnPP59zcyXzcOXOuTAghQERERESkY/S0HQARERERUWkwkSUiIiIincREloiIiIh0EhNZIiIiItJJTGSJiIiISCcxkSUiIiIincREloiIiIh0EhNZIiIiItJJTGSJiIiISCcxkSUqpR9++AENGzaEgYEBatWqVapjNGzYEEOGDCnnyCpXmzZt0Lt3b22HUSn+TWNV15twDb8uPj4eo0aNgp2dHWQyGWbPnq3tkIioCExkqVzk5ORg/fr1aNu2LWxsbGBtbY22bdti8eLFePbsmUJdd3d3+Pr6lqm/8jhGWVy8eBEzZszAF198gezsbKUx/pO2YwWA6tWro0uXLir3PXjwADKZDB9++GGlxvRvUtnXQFW45nTZ/Pnz8d///heXL1+GEAIrVqzQdkiVonHjxhg4cKC2wyDSiIG2A6A3w7Bhw3D8+HGsWrUK/v7+MDMzw8mTJzF79mxcuXIFhw8f1naI5ercuXMAgOHDh0NfX1/L0WjX1atXtR1Cpfk3jfXf7Ny5c/D09ET9+vW1HQoRlYB3ZKnMIiMjcfjwYUyZMgVTp05FzZo1YW5uDn9/f1y7dg0dO3bUdojl7tWrV9DX14eRkZG2QyGicvbq1SuYmJhoOwwiUgMTWSqzhIQEAECdOnWU9llZWWHOnDnStrm5OW7duoVjx45BJpNBJpOhYcOG0n5fX1+pXF9fH3Z2dhg6dCju3r2r9jEA4MiRI/Dy8kK1atVgamoKT09PnD59usSxCCGwdu1aNGvWDMbGxrCxsYGfnx9u3rwJAMjLy4NMJsPq1auRn58v9b9w4UKVx1MnVgCIiopC165dYWpqCgcHByxZskTl8Uo7Lk0VzntUJy5V80ZLOo+l6UcVda4XADh9+jR8fHxga2sLS0tLdOzYEXv27IEQQqM6qsZaUrvyuOY1OVcl9bdhwwbIZDLs3r27xPOrzjkppM7vUNOx3rhxA127doWZmRmmT58OAMjMzMTnn38OV1dXyOVy2NnZ4b333it2ek+hFy9eYNKkSbC3t4eRkRHq16+POXPmICMjAwCwdetWyGQyvHr1Ctu2bZNivXfvXpHHzMzMxJdffokmTZrA2NgYTk5OmD59Ol69eqV2vwAQFhYGmUyGw4cP46effoKLiwsMDAwQFhaGBQsWQCaTISUlBVOnToWdnR1q1KghtT106BA8PT1hbm4OU1NTdO7cGWfPnlWK9dChQ/D29oalpSWsrKzQq1cvXL58GQBgbGyMO3fu4MiRI9K4GzduXOz57N27t8Lvs2bNmnj77bdx//59hXr16tXD8OHDcevWLXTp0gUmJiZwdHTEsmXLlI4ZFBQEb29v2NrawsrKCp06dcL+/ful/TNmzIC1tTXy8/Olss8//xwymQwzZsxQOJa9vT3GjBmjdA5KOleF8V6/fh0+Pj4wNTWVpl6VFB9pgSAqo6SkJGFhYSEaNWok/vrrrxLru7m5iX79+pVYLysrS1y7dk34+PgIZ2dnkZqaqtYxVq9eLWQymVi4cKF4/PixePnypVi4cKHQ19cXgYGBxfb50UcfCQMDA7FmzRoRHx8voqKiRLdu3YSZmZm4ceOGVG/mzJlCX1+/xDGUFKuzs7Pw9vYWgwYNEuHh4SIpKUksW7ZMABD79u0rt3HZ2toKb29vlftiYmIEADFz5sxSxdW6dWvRq1cvhTJ1z6Mm/ZSkqOvlzp07Qi6Xi1mzZolnz56JtLQ0cfnyZTF8+HDx8OFDteuoGqu67crjmtfkXBXX3/r16wUA8euvvxYbi7pjK+3vsKSxenl5iT59+ohr166Jp0+fin379ons7GzRqVMnUadOHfHbb7+J5ORkcevWLeHp6SlcXFxESkpKkf0lJyeLhg0bigYNGojz58+L5ORk8dtvvwlbW1vRuXNnkZeXJ9W1tbUVY8aMKfb8FI7Bw8ND1KxZU+zZs0fEx8eLR48eiXXr1okVK1Zo1G9oaKgAIAYNGiTmzp0rnjx5Ii5evChu3Lgh5s+fLwCIMWPGiN27d4uEhASxadMmIYQQK1asEHp6emLRokXi8ePH4sWLF+Lzzz8X+vr6Ijg4WIr1m2++EXp6euLTTz8V9+/fF0lJSSIoKEgMHz5cqtOoUSPh5+dX4riLOhd//PGH8PLyEi4uLiI9PV3a5+TkJHx8fMTgwYPFjRs3RFJSkliyZIkAIA4dOiTVi4yMFEZGRuKTTz5RuObefvtt8fjxYyGEEL/99psAIEJCQqR2Hh4ewsTERDRq1Egqi4iIEADEjh07pDJ1z5WTk5Pw9vYWffr0EeHh4eLp06di//79asVHlY+JLJWLY8eOiZo1awqZTCbeeustMW7cOLFlyxbx/PlzpbrqvqkXunfvngAgDh8+XOIxnj9/LoyNjcX48eOV9vXu3Vu0aNGiyH5iYmKEnp6emDp1qkJ5UlKSsLS0VOivPBNZY2Nj8eTJE4XyJk2aiB49ekjbZRmXEKVLZNWJSwjl5E6T86hJP+p6/XrZunWrAKCQfL1OnTpCKI9V3Xblcc1rcq407U8VdcdW1t9hUWM1MDAQDx48UKi7bt06AUCcPXtWoTwuLk7I5XKxfPnyIvspTJwuXbqkUL5jxw6lxF7dRHb16tUCgDh58mSZ+y1MZH18fJSOUZjIrly5UqH8yZMnwsjISEyePFmpTffu3UXr1q2FEELExsYKQ0NDMW7cuGLHU5ZEttDt27cFABEQECCVOTk5CRMTE/Hs2TOFuq6urqJPnz7S9ubNmwUAERcXV+TxU1JShKGhoVi4cKEQ4u+/K/r6+mLevHkCgHj06JEQQohVq1YJAOLp06dCCPXPVWG8hoaGIjY2VqGeOvFR5ePUAioXffv2xaNHjxAYGAh/f38kJibigw8+QIMGDbBp0ya1jxMdHY13330XDg4OMDQ0VPhYtLiP9woFBwcjKysLQ4cOVdrXvXt3hIeHIzk5WWXb//73vygoKMDgwYMVyi0tLdG9e3ecOXNG5UeqZdWyZUvUrl1boczd3R3R0dHSdlnGVZFxqaLpeSxtP4B610uzZs0gk8kwfvx4nDx5UuHj3ELq1FGltO00HUOhspwrTWkyNnXj0mSsLVq0gJOTk0LZ0aNHVa7AYW9vjyZNmkhfwlTl9OnTqFWrltKcfX9/f2m/po4fP47q1aujZ8+e5dbvgAEDijzW6/uCgoKQk5NT5N+FP//8E2lpaQgODkZubi5GjhxZ4pg0cf/+fYwePVrh91k4HeH132ebNm1Qs2ZNhTI3NzeFa6R58+aQyWQYO3YsgoODkZmZqdRntWrV0L59ewQFBQEAzpw5A319fcyZMwcWFhZSeXBwMJo1ayYtjajuuSrUqlUrODg4KNRTJz6qfExkqdwYGRmhZ8+eWLBgAQ4ePIiYmBg0bdoUU6ZMQVRUVIntU1JS4OnpiaioKBw4cACJiYkQQkhz33Jzc0s8RmHdfv36wcDAAPr6+tDT04Oenp60FmR8fLzKtoXlqtaErVWrFjIzM0uVqJTk9QQAACwsLJCUlCRtl2VcAKCnp4eCggKV+wrLX199QZ24VNH0PJa2H3Wvl1atWmHfvn1ISkpCnz59YGVlBS8vL/z666/SsdSpo0pp22k6hkKlPVelocnY1IlL07GqmnP/7NkzvHr1CgYGBgqvA5lMhuvXrxf7GoiPj1d5TZqYmMDS0lJhTqu6Xrx4oTLOsvRb3PFe31d47nr16qX0d2Hu3LkQQiAhIQEvXrwo8diaSk5ORqdOnXDv3j0cPHgQSUlJEELg8ePHAEp37bZt2xZ79uxBfHw8evXqBUtLS3h7e2PPnj0K7Xr06IHff/8dycnJCA4ORocOHWBlZYUuXbogODgYOTk5OHfuHHr06CG1UfdcFVJ1rtSNjyoXE1mqMNWrV8e0adOQn5+P0NDQEuufOnUKT58+xcqVK9G+fXuYm5sDAGJiYjTqE/h7+Zy8vDzk5+ejoKAABQUFEH9PpUGDBg1UtrWxsQEAPH/+XGnf8+fPYWJiAlNTU7VjUZdMJiuxTlnGVdi+qDfqwje5wj40iUsVTc9jafvR5Hrx9/fHlStXEB8fj4MHD8LCwgIjR45U+MKTOnVUKW07TccAlP5clZa6Y1MnLk3HamhoqFRWvXp11KtXD3l5eQqvg8LXQEhISJH929jYqLwms7KykJKSonT9q6NGjRqIi4srto6m/aoad1H7CtteunSpyL8LdevWlb4YVlKsmjh58iSeP3+O7777Du3atYOZmRmAsl+7Q4cOxdWrV/Hq1SscOHAAZmZmGD58uMIXqnr06IG8vDycOXMGwcHB0h3xnj174tSpU7h06RLS09MVEll1z1Whon4P6sRHlYuJLJXZlStXsH37dpX7Cv9wWllZSWVmZmbIzs4u8nhyuVxhW9WxizpGz549IZfLS/U/ZB8fH+jp6eHQoUMK5SkpKTh16hS6du1aqkSipPGqoyzjAoAuXbrgzp07ePjwodK+kydPSnXKQ0Wdx6Koc70Usra2hq+vLw4ePAg9PT2cP3++VHU0PXZ5XPOaKI9r7p9Ke05UKctY+/fvjwcPHuD333/XuN9u3brh6dOn0rf0Cx08eBBCCHTr1k3jY/r6+uLVq1cIDg6u1H4L9erVC4aGhiX+XejZsycMDQ2xa9euYuuV5rop72u3kI2NDfr3748DBw5AJpMpXHPt2rWDpaUlfvrpJ9y7d09KWHv06IH4+HgsW7YMRkZG8PLyktqoe67KIz6qXExkqcyys7MxZswYDBkyBKGhocjMzMSLFy/wyy+/YMmSJXBzc0Pfvn2l+u7u7ggPD1f6n3vnzp1hZWWFBQsW4NGjR3j58iWWL1+u8HFPSceoXbs2li1bhnXr1mH+/PmIjo5GZmYm7ty5gw0bNhQ7R6xevXqYPn06Nm7ciHXr1iEhIQF37tzBkCFDkJOTg8WLF5fq/BQVqybKMi4AmDdvHqpXrw4/Pz+cP38eqampePLkCdauXYtly5Zh2LBh6NChQ6nj+6eKOo+vU/d6+f777/Hxxx/jjz/+QFpaGhITE/H999+joKAAPj4+atdRRd125XHNa6K4a07d5bdKe06KUh5jnTx5Mjw9PTFkyBDs27cPL1++RHJyMn7//XfMnDkTmzdvLrLttGnT0KBBA4wePRqXLl1Camoqjh07hg8//BAdO3Ys1WN2J02ahPbt2+Odd97Bvn37kJCQgMePH2P9+vVYuXJlhfVbyMHBAUuXLsWaNWvw+eefIyYmRvq78OOPP2L06NFSvUWLFmHr1q2YN28eYmJipP9YjhgxQjqeu7s7rl+/jgcPHpTYd+EyXvPnz0dsbCxevHiBb775BikpKaUez6pVqzB79mzpmktISMD3338PIYTCNaevrw8fHx8EBgbCxsYGrVu3BgC4urrCyckJJ0+eRKdOnRQ++VH3XJVHfFTJKvGLZfSGys/PF0FBQeLdd98VjRo1EsbGxsLU1FS4u7uLuXPnioSEBIX6jx49Et26dRPm5uYCgHB2dpb2hYSEiI4dOwozMzNRq1YtMXv2bPH8+XMBQCxdulStYwghRGBgoOjZs6ewtrYWxsbGonHjxmL69Oni7t27xY6loKBArFmzRri5uQkjIyNhaWkpfH19xfXr1xXqabJqQXGxOjs7C39/f6U248ePF7a2tkrlpR1XYRwTJ06UvpFramoqWrVqJVavXq2w9JCmcalafkvd86jp+F+nzvWSkpIiVq9eLdq2bSvMzc2FjY2N8PT0FHv37pWOo04dVWNVt115XPOanKvi+lN3+S11x6ZJXGUdqxB/L/O0ZMkS0axZM2FsbCysra2Fh4eHWLt2rcKST6o8ffpUjB8/XtSqVUsYGBiIunXrio8//lhh6S8h1F+1QAgh0tPTxfz580XDhg2FkZGRqFevnpg+fbp4+fKlRv0Wrlrwz+WoChWuWpCbm6syhuPHj4sePXoIKysrYWxsLJo0aSJmzJgh7t+/r1Bv//79olOnTsLMzExYW1uLXr16ibCwMGn/gwcPRNeuXYWZmZkAoLCclSoXL14UHTp0EKampqJ27dpizpw54smTJwKAwgoSTk5OYtiwYUrtx4wZI2rWrCltJycni++++0665mxtbYWXl5fYv3+/UtvCFSyGDh2qUD5hwgQBQCxevLjU56qoeDWJjyqPTIgK+Bo2EREREVEF49QCIiIiItJJTGSJiIiISCcZaDuA5ORkBAQEwNzcHH5+fmq1efz4MS5cuABjY2N07doVlpaWFRwlEREREVU1Wpsjm5+fj/fffx/Hjh2DsbExbGxscPXq1RLbbd++HVOmTEHnzp2RmJiI6OhonDhxAm3atKmEqImIiIioqtDa1AIhBNq1a4e7d+8W+0i+f3r27Bnef/99LFu2DIGBgbh8+TK6d++O9957r2KDJSIiIqIqR2uJrIGBASZOnCg9DUQdR44cAQCMHz9eKvvggw9w69Yt3Lhxo9xjJCIiIqKqS+tzZDURERGB+vXrw8TERCpzc3OT9jVv3lypTXZ2tsKTSgoKCpCQkABbW9tKf9wjERERERVPCIHU1FTY29tDT6/4e646lcimpKTA2tpaoczS0hL6+vpFPk1k6dKl+OqrryojPCIiIiIqJ7GxsXBwcCi2jk4lsiYmJkhNTVUoy8zMRH5+vsJd2n+aN28eZs2aJW0nJyejbt26iI2NhYWFRYXGS0RERESaSUlJgaOjI6pVq1ZiXZ1KZBs2bIi9e/eioKBAutUcHR0t7VNFLpdDLpcrlVtYWDCRJSIiIqqi1JkCWqUfiJCTk4PNmzfj7t27AIB+/fohKSkJgYGBUp1du3bBzs4O7dq101aYRERERKQFWr0je/DgQSQkJODWrVt49eoVNm/eDAB47733YGBggIyMDEycOBFbtmyBi4sLmjRpgpkzZ+Kdd97BjBkzkJCQgB9//BE7duyAoaGhNodCRERERJVMq4lseHg44uLi4OTkBCcnJ4SFhQEA3nnnHRgYGEAul2P8+PFwdXWV2qxatQpdunTBmTNnIJfLERISgrZt22prCERERESkJVp7spe2pKSkwNLSEsnJyZwjS0RERFTFaJKrVek5skRERERERWEiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0RERKSGjIwM3L59G48ePUJ6ejoiIiK0HdK/HhNZIiIiohLs3r0btWrVwsCBA7Fs2TKEhoaiWbNmZTpmVlYWIiIiUFBQUE5R6kbf5UmrD0QgIiIiAoDly7dVWl+ffDJG4zZffPEFli5dimnTpgEAwsLC4ObmVqY4IiIi0LZtWyQmJsLKyqpMx9KlvssT78gSERERFSE7OxsRERGIjY0F8HcCGBERAXNzc+zevVuql5GRgYiICAghkJaWhtu3byM9PV3a/+rVK+kYAJCbm4v79+8DAKKiohAREYHHjx8XGcfr7YUQiIiIQFpamlLdW7duITk5uUx9P3z4EM+ePVM69j/HmZubi+joaGRmZkr7U1JSEB0djfz8/CLHUp6YyBIREREVITY2FsOHD0d2dja+/fZbDB8+HMOHD0f//v0Vphb8+eefaNasGebOnQtHR0f4+/sjMjISz549Q4cOHeDs7AwvLy84OTkhMDAQL1++xPz58wEA48ePx/Dhw7FmzRql/otqL5PJMGrUKCxfvlyhfkhICJo1a4akpKRS9X3q1Ck0aNAAHh4eeOutt9CoUSNcvnxZaZzTpk1DnTp14OnpiVq1auHw4cOYOXMmnJ2d0alTJ9StWxfXr18v71+HEiayREREREVo2LAhIiIiYGRkhHXr1kl3ZDdt2qSy/t27d/H06VPcunULbdu2xdKlS2FiYoKXL18iJiYGoaGhiI6Ohr29vXRHNyQkBBEREUpJKYAi2wPAhAkTsHXrVoV5rr/88gu6desGJycnjfu+d+8eBg8ejJUrV+Lp06d49uwZJk2ahMGDByvd+U1LS0NcXByePHmCwYMHY8iQIcjIyMDTp0/x5MkTeHp6Yu7cueXyOygOE1kiIiKicrJo0SIYGxtL22lpabCwsICBwd9fS7K3t8fUqVPVPl5x7UePHo0XL17g9OnTAID09HTs3bsX48aNK1XfGzduhJubG5o0aYKoqChERkaiV69eSEhIwO+//65Q96uvvoKhoSEAYNCgQcjPz8eiRYtgYGAAmUwGPz8/3pElIiIi0iX169dX2P74449x48YNNGjQAJMnT8ahQ4c0WimguPbW1tbw9/fHL7/8AgDYt28f9PX1MWjQoFL1HRUVhTt37mDIkCEYOnQo3n77bQwfPhzOzs5Kd2Rr164t/dvU1FRlmar5u+WNiSwRERFROdHX11fYbtq0Ke7du4fdu3ejXr16mDVrFvr06aP28UpqP3HiRBw+fBiJiYn45ZdfMHLkSOmOsKZ9GxkZwdPTU5o+8c+fAQMGaHgmKgeX3yIiIiKqIDk5OTAyMoKHh4f007VrV6SkpEgJZ25ubqnaW1hYwNvbG46Ojli0aBEuXLiA1atXl7pvT09PLFy4EC9evICdnZ1Unp+fD5lMBj29qnf/k4ksERERUQV599134ezsjG7dusHY2Bhr166Fu7s7LCws4OTkBBMTE2zbtg29evWCtbU1HBwc1G5faPz48Zg7dy5atGiBVq1albrvyZMnY+vWrejRowcWLlyI2rVr4+bNm/jhhx9w/vz5KrnebNVLrYmIiIiqGDc3N4Xk0dzcXOGBCGZmZnBzc1O6a7l582aYm5tj8eLFmDNnDhwdHXHy5EkAQLVq1bBz504EBQVh1KhRKpffKq59oXfeeQcApC95lbZvMzMzXLx4EYMHD8aqVavw0Ucf4c8//8S+ffukJFbVOF8/FwBgYWGBpk2bqnVuy0ImhBAV3ksVkpKSAktLSyQnJytckERERES6KCAgAEOHDsXjx49ha2ur7XDKTJNcjVMLiIiIiHRQYmIi7t69iy+++ALjxo17I5JYTXFqAREREZEOOnv2LCZOnIhmzZrhm2++0XY4WsGpBURERERUZWiSq/GOLBERERHpJCayRERERKSTmMgSERERkU5iIktEREREOomJLBERERHpJCayRERERKSTmMgSERERFSMyMhIODg5ITEzUdihasXHjRvTr10/bYajEJ3sRERGR1j266FNpfdX1PKtR/ZycHMTFxSE/P7+CIqraUlNT8fz5c22HoRLvyBIREREVw83NDbGxsbCxsdF2KPQaJrJERERExbh79y48PDyQnJwMALhy5QocHBxw8OBB9O3bF66urhgwYADu37+PgIAAeHt7w9XVFWPGjFGYjlDY7uTJk+jduzdcXV0xaNAgREdHK9UJDg5Gly5dUK9ePYSGhgIADh48CC8vLzRo0ABdu3bFyZMnpXZXr15F3bp1le6cbt68GW3atEHhg1xDQkLg6+uLhg0bonPnzti4cSNef8jr1q1b0bp1a7Ro0QJTp06Vxl0VMZElIiIiKsbrUwuys7MRFxeH//znP5g7dy7279+P58+fw8fHB19//TUWL16MPXv2IDw8HHPmzJGOU9hu0qRJ+PDDD7F3714IIdCzZ0/k5uYq1Jk5cyYWLFiACxcuoFWrVjhx4gRGjBiBoUOHIiAgAD179oSvry9CQkIAAK1bt4YQAnv27FGI/eeff4anpydkMhnOnTuHvn37YsCAATh27Bjmz5+Pb775BsuXL5fq//bbb5gyZQref/997Nq1C9bW1liyZElFn+JSk4nX0/A3nCbP7yUiIqLKUZXnyF6/fh0tW7bEy5cvUb16dVy8eBGdO3fG5cuX0a5dOwDAr7/+ipEjRyI8PBzNmzcHAGzatAlLlixBTEwMAEjt9u7di6FDhwL4Oy9xdHTEunXrMHr0aKnO6dOn0bVrVykGDw8PvPXWW9iwYYNUNnToUKSlpeHEiRMAgDlz5uDcuXO4fPkyACA6OhrOzs64cuUK2rRpg86dO6NHjx744osvpGPs3bsXH374IZ48eSL10759e6xZs0aq07lzZ2RmZuLq1asanbfS0iRX4x1ZIiIiolJwc3OT/l2jRg2VZa9evVJq5+npKf3bwsICLVq0wI0bNxTqtGrVSvq3EAI3b96El5eXQp0uXbogPDxc2h49ejR+//133Lt3DwCwc+dONG7cGG3atEFBQQHCwsKwdu1a1KtXD05OTqhbty4++OADPH36FOnp6RBC4MaNGwrxAX8nslUVVy0gIiIiKgV9ff0Sy1R98G1kZKS0nZ2drVBmbGws/bugoAC5ubmQy+UKdeRyuUK75s2bo1mzZti5cye+/PJL7Nq1C6NGjQIA5OfnIy8vDwsXLsSAAQOUYjI1NZX6URVfVcU7skRERESV6ObNm9K/8/LyEBkZCRcXlyLr6+vrw9nZWeHuKwBcu3YNrq6uCmWjRo3Czp078ccff+DOnTtSImtoaAgXFxdERETAwcFB6Ucmk0FfXx8NGjRQiA+AUr9VCRNZIiIioko0b948xMfHIz8/H4sWLUJ6ejpGjhxZbJsPPvgAP/74I/744w8AwPnz57Ft2zbMmDFDod7IkSNx7949zJo1Cx07dkT9+vWlfXPmzMHmzZvx66+/oqCgAPn5+QgLC8OCBQukOlOmTMEPP/wgJbMBAQE4evRoeQ293Gl9akFERATOnj0LY2Nj9OvXD/b29iW2OXv2LCIjI2FkZIT27dtLk6qJiIiIqrrCJbuysrJgaWmJPXv2lLhG7fvvv4+HDx/Cy8tLmr7w6aefYsSIEQr1HB0d4eXlhXPnzmH9+vUK+yZMmIC8vDzMmTMH48ePh76+Plq0aKGwKsH06dMRGRmJVq1awczMDPXr18fo0aMRERFRTqMvX1pdtWDNmjWYN28eBg8ejMTERJw7dw4BAQHo0qWLyvo5OTno27cvIiMjMWjQIKSlpWH//v2YMmUKVqxYoVafXLWAiIiINJGbm4vnz5/D3t4eenp6yMnJwYsXL+Dg4CDVyc7OxsuXLxXKsrKykJCQIN2kK1yRIDMzE8bGxoiPj4e1tTX09P73AbmqY78eS2JiImxsbGBgoPp+ZHJyMlJTU1GzZk0YGhqqrBMfHw9zc3OlebeF0tPTAQBmZmZIS0tDRkYG7OzsijlL5UeTXE1riWxsbCwaNmyIn376CWPGjAEATJw4EWfPnsXdu3chk8mU2gQGBqJPnz64c+eONCdk69atGDdunHQxlISJLBEREWnD64ksqaYTy28dOXIERkZGCrfEJ0+ejPv37+PPP/9U2cbExAQAFP73YGxsDLlcXuT/SoiIiIjozaS17C8qKgr16tVTWNKhUaNG0r7WrVsrtfH29saCBQswePBg9OnTB+np6Th16hS2bduGatWqqewnOztbYWmKlJSUch4JERERUcnatWuH2NhY3o0tR1q7I5uWlgZLS0uFsmrVqkFfXx9paWkq2+Tn5yMrKwuJiYmIj49HfHw8UlNTpXkcqixduhSWlpbSj6OjY7mOg4iIiEgdRkZGRc59pdLR2h1ZMzMzpbujaWlpyM/Ph5mZmco2O3bswLp163Dnzh0pIT127BgGDBiADh06oHHjxkpt5s2bh1mzZknbhY+CIyIiIiLdprU7sq6urnj48CHy8vKkssJHqr2+uG+ha9euoWHDhgqJaJcuXVBQUIBr166pbCOXy2FhYaHwQ0RERES6T2uJ7IABA5Ceno5Dhw5JZVu2bIGDgwPatm0L4O/5rd988420KG/Dhg0RHR2NFy9eSG3CwsIAoNgnYhARERHRm0drUwsaNGiAL7/8EuPGjcP58+eRkJCA/fv34+DBg9J6apmZmZg3bx5q1aqFZs2aYfz48diyZQs8PDwwbNgwpKenY9u2bRgzZgzatGmjraEQERERkRZo9YEIAHDp0iWcOXMGcrkcgwYNUrizmpWVhYULF2Lo0KHSKgb5+fn47bffEBUVBSMjI7Rr1w5eXl5q98d1ZImIiIiqLp14IIK2MJElIiIiqrp04oEIRERERERlwUSWiIiIiHQSE1kiIiIi0klMZImIiIhIJzGRJSIiIiKdxESWiIiIiHQSE1kiIiIi0klMZImIiIhIJzGRJSIiIiKdxESWiIiIiHQSE1kiIiIi0klMZImIiIhIJzGRJSIiIiKdxESWiIiIiHQSE1kiIiIi0klMZImIiIhIJzGRJSIiIiKdxESWiIiIiHQSE1kiIiIi0klMZImIiIhIJzGRJSIiIiKdxESWiIiIiHQSE1kiIiIi0kkG2g6AiIjK7tFFH22HoBPqep7VdghEVI54R5aIiIiIdBITWSIiIiLSSZxaQFUKPx5VDz8eJSIi4h1ZIiIiItJRTGSJiIiISCcxkSUiIiIincREloiIiIh0EhNZIiIiItJJTGSJiIiISCcxkSUiIiIincREloiIiIh0EhNZIiIiItJJfLIXERER/SvxaZLqq6pPlNR6Inv+/HmcPn0axsbGGDRoEBo3blxim5SUFBw4cAAxMTFo2bIlBg4cCJlMVgnREhEREVFVodWpBQsXLoSvry+Sk5Nx+/ZttGjRAkePHi22TXh4OBo3boxt27bByMgIBw4cwLvvvltJERMRERFRVaG1O7L379/Hf/7zH+zZswf+/v4AABsbG0yZMgV9+/aFvr6+Upu8vDwMGTIEXbp0wa5du6Tyu3fvVlrcRERERFQ1aO2O7NGjR2Fubg4/Pz+p7L333kNcXByuXr2qsk1gYCDu3buHL7/8UqHcxcWlQmMlIiIioqpH4zuy+/btQ2JiIiZNmqTRvtfduXMHdevWhYHB/0JwdnYGAPz1119o3769UpsrV66gRo0asLKywsqVK5GRkYGWLVvC19e3yH6ys7ORnZ0tbaekpJQYGxERERFVfRrfkX348CH++usvlftiYmJw//59tY6TkZEBCwsLhTJzc3Po6+sjPT1dZZvk5GQAQJcuXRAXF4fMzExMnDgR/fv3hxBCZZulS5fC0tJS+nF0dFQrPiIiIiKq2tS+I5uUlIRXr14hPj4eSUlJuHfvnsL+9PR0BAUFYcCAAWodz9zcHElJSQplqampyM/PR7Vq1Yps8/LlS2zZsgX9+vUDAIwaNQru7u4IDAxEnz59lNrMmzcPs2bNkrZTUlKYzBIRERG9AdROZDdv3oxPPvlE2v7555+V6ri7u2PkyJFqHa9p06bYvn07srOzIZfLAQC3b98GADRp0kRlGzc3NwBA27ZtFcpMTEwQHR2tso1cLpeOT0RERERvDrWnFkyaNAkxMTH47LPPMHHiRMTExCj8xMfH4+bNm6hevbpax/Pz80NOTg527twplW3YsAENGzZEy5YtAQBZWVmYPXu29OWvPn36wMLCAmfOnJHahIWFITMzE82bN1d3KERERET0BlD7jqyFhQUsLCywcOFCCCFgZGRUpo4dHBywcuVKTJ8+HcHBwUhISEBoaCgCAgKkhxtkZWVh5cqVcHd3R5s2bWBlZYVNmzZhwoQJ+O2332BsbIz9+/djxowZ6Ny5c5niISIiIiLdovGqBYaGhrhw4QJsbW3RtGlTqfzBgwe4c+cOevXqpfaxpk+fjm7duuG///0v5HI5tm3bhlq1akn7TUxMsHz5coWpBG+//Tbat2+PoKAg6Onp4cMPP+TdWCIiIqJ/IY0T2bi4OEyaNAlXrlxRKK9Tpw78/f3h6OiokOCWpEmTJkXOiZXL5Zg9e7ZSuZOTEyZOnKhZ4ERERET0RtF4+a2goCB07NgR5ubmCuWGhobo1asXAgICyi04IiIiIqKiaJzIFhQUID4+XuW++Ph45OTklDkoIiIiIqKSaJzIdu3aFSdOnFC683rhwgXs2LED3bt3L7fgiIiIiIiKovEc2fr16+Prr79G//794e7uDicnJzx58gTXrl3DRx99BA8Pj4qIk4iIiIhIgcZ3ZAFgzpw5CAsLQ+/evWFtbQ1vb2+cOXMG3333XXnHR0RERESkksZ3ZAu1b98e7du3L89YiIiIiIjUVqo7sgAQGRmJX375BefOnQMAJCUl4fnz5+UWGBERERFRcUqVyH799ddo2bIlPvnkExw9ehQAkJycDB8fH+Tm5pZrgEREREREqmicyN66dQurV69GeHg4PvvsM6ncyckJzZs3x4EDB8o1QCIiIiIiVTROZMPCwjBo0CA0btxYaV/jxo1x8+bNcgmMiIiIiKg4pXogQuFDD2QymcK+Bw8ewNLSsnwiIyIiIiIqRqkeiBAQEIC7d+8qJLLHjh3Drl270KtXr3INkIiIiIhIFY2X33J2dsZnn32Gt956C3Z2dtDX18eJEycQGRmJzz77DC1atKiIOImIiIiIFJRqHdnZs2fDx8cH+/btQ1xcHGxtbbFu3Tp06dKlnMMjIiIiIlJN40T2+PHjSE5OxogRI9C6deuKiImIiIiIqEQaz5F98OABLl26VBGxEBERERGpTeNE1tPTE2fPnkV2dnZFxENEREREpBaNpxYUFBRALpejVatWGDJkCGrUqKGwv2XLlujUqVO5BUhEREREpIrGiey1a9eQkZEBANizZ4/S/nHjxjGRJaJysXz5Nm2HoDOGddB2BERElU/jRHbs2LEYO3ZsRcRCRERERKQ2jefI/vTTT/jPf/5TEbEQEREREalN40TWwMAAMTExFRELEREREZHaNE5k+/Tpg3PnzuH+/fsVEQ8RERERkVo0niN79epVAEDTpk3h4eGhtGrBgAED8O6775ZPdERERERERdA4kTUxMUH37t2L3G9mZlamgIiIiIiI1KFxItu9e/diE1kiIiIiosqg8RxZIiIiIqKqoFSJbEZGBr7++mu0a9cO9vb2aN26NT799FMkJSWVc3hERERERKppPLUgPz8fPj4+ePz4Md555x04Ojri2bNn+PXXX3H48GFcu3YNpqamFRErEREREZFE40Q2MDAQL1++REREBKytraXyzz77DB07dsSvv/6K8ePHl2uQRERERESv03hqwf3799G9e3eFJBb4ezUDX19fri9LRERERJVC40S2du3auHLlCvLy8hTKhRAICQmBvb19uQVHRERERFQUjRPZfv36ISkpCT4+Pti+fTtOnz6NXbt2oU+fPggPD8ewYcMqIk4iIiIiIgUaz5E1NTXF+fPn8emnn2L69OlITU2FqakpunfvjpCQEKUnfREREVHlWr58m7ZD0AnDOmg7AiorjRNZAHB0dMSuXbsAAMnJybC0tCzXoIiIiIiISlLmByIwiSUiIiIibdA4kS0oKEC3bt1w+fJlhfK7d+/Cw8MDWVlZ5RYcEREREVFRNE5kT548CWNjY7Rv316h3MXFBW5ubti7d69Gxzt48CCmTZuGjz/+GGFhYWq3E0Jg3rx5eO+995CYmKhRn0RERESk+0q1jmzt2rVV7qtduzb++usvtY81Y8YMTJ48WTpe586dsXPnTrXafvfdd9ixYwe2bduG9PR0tfskIiIiojeDxomsq6srTp06pZQ85ubmIiAgAA0bNlTrOJGRkVi7di3+7//+DwsWLMDKlSsxd+5cfPTRR8jNzS227dWrV7FmzRosX75c0/CJiIiI6A2hcSLbrVs32NraolOnTvj5559x6tQpbN++Hd7e3khJScHbb7+t1nGOHz8Oa2tr9OjRQyobMWIEXr58qTT/9p9SU1MxYsQIbNy4kUt9EREREf2LaZzI6uvrIzAwEE2bNsUHH3yAHj16YOLEiahRowbOnj0LU1NTtY5z7949ODo6Qk/vfyHUr18fAIp9zO3kyZPRu3dv9OnTR61+srOzkZKSovBDRERERLqvVOvI1qhRA7t27UJeXh6Sk5NhYWEBQ0NDjY6RmZkJMzMzhTITExPo6+sjMzNTZZtffvkF4eHhuHr1qtr9LF26FF999ZVGsRERERFR1VeqRFZqbGAAW1vbUrW1sLBAUlKSQllycjLy8/OLXJt28eLFqF27NqZMmQIAePr0KQBg1qxZGDhwIEaOHKnUZt68eZg1a5a0nZKSAkdHx1LFTERERERVR5kS2bJo1qwZfv75Z2RkZEjTESIiIgAA7u7uKtssW7YMaWlp0nZkZCSCgoLQoUOHIr9kJpfLIZfLyzl6IiIiItI2rSWyfn5++Oijj7Bp0ybMnDkTAPD999/D3d0dzZo1AwBkZGRg6tSpmDBhAjw9PTFkyBCFY5w6dQrLly/H0KFD4eDgUOljUBefea0+PveaiIiI1KW1RLZmzZrYuHEjJk+ejICAACQmJiI2NhYnTpyQ6uTk5GDbtm3o0qULPD09tRUqEREREVVBWktkAWD06NHo2rUrLl26BLlcDh8fH1SrVk3ab2pqii1bthSZxLq5uWHLli2wsbGprJCJiIiIqIrQaiILAPb29hg6dKjKfUZGRnjvvfeKbFu7du1i9xMRERHRm0vjdWSJiIiIiKoCJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkA212np+fj59//hmnT5+GsbEx3n77bfTr16/YNjdv3sSuXbsQHR0NR0dHjBs3Dk2bNq2kiImIiIioqtDqHdmxY8fi66+/ho+PDxo3bgx/f3+sX7++yPpbtmzBO++8A0tLSwwePBhZWVlo0aIFAgMDKzFqIiIiIqoKtHZH9tq1a9ixYwfOnz+Pzp07S+WfffYZxo4dC2NjY6U2/fr1w3vvvQeZTAYAGDZsGF68eIElS5agd+/elRY7EREREWmf1u7IBgYGokaNGvD09JTKhgwZgqSkJISFhalsY2dnJyWxhWrVqoWUlJQKjZWIiIiIqh6tJbIxMTGoU6eOQmLq6Ogo7VPH8+fPsWvXLvTt27fIOtnZ2UhJSVH4ISIiIiLdp7VENicnB6ampgplcrkc+vr6yMnJKbF9RkYGBg8eDAcHByxYsKDIekuXLoWlpaX0U5gsExEREZFu01oia2VlhYSEBIWypKQk5Ofnw8rKqti2mZmZGDBgAJKSkhAUFKSUEP/TvHnzkJycLP3ExsaWR/hEREREpGVaS2RbtGiB6OhopKamSmXXr1+X9hUlKysLfn5+ePLkCc6cOQM7O7ti+5HL5bCwsFD4ISIiIiLdp7VE1s/PD8bGxli9ejUAoKCgACtWrEDbtm3RuHFjAEB6ejp8fX1x+vRpAH/Pd/Xz80NcXBzOnj2LmjVrait8IiIiItIyrS2/ZWNjg507d2L06NE4ePAgkpOTAQDHjx+X6uTm5uLYsWMYMmQIAODbb79FUFAQ2rdvj/Hjx0v1zM3NsXv37sodABERERFplVaf7OXr64vY2FhcvXoVcrkcbdu2haGhobTf3NwcR48exVtvvQUAGDp0KFq2bKl0nH+2ISIiIqJ/B60msgBQrVo1+Pj4qNxnYGAAX19fabtx48bStAMiIiIi+nfT6iNqiYiIiIhKi4ksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6SauJbFZWFpYsWYJu3bqhX79+2L59e4W0ISIiIqI3j4E2Ox8+fDiioqKwePFiJCYmYtq0aXjy5Anmzp1brm2IiIiI6M2jtUQ2NDQUR44cwZUrV9CmTRsAQHp6OhYsWIAPPvgAZmZm5dKGiIiIiN5MWptacPr0adSqVUtKSAHAz88P6enpCAsLK7c2RERERPRm0tod2YcPH8Le3l6hrE6dOtK+8mqTnZ2N7OxsaTs5ORkAkJKSUrrASyErK7PS+tJ1qel52g5BJ1Tm9atNfO2oj68d9fC1Q//E1436KvO1U9iXEKLEulpLZHNzcyGXyxXKDA0Noaenh9zc3HJrs3TpUnz11VdK5Y6OjqWMnCrSF9oOQGdYajsAqmL42lEXXzv0P3zdaKLyXzupqamwtCy+X60lsjY2NkhISFAoS0pKQkFBAWxtbcutzbx58zBr1ixpu6CgAAkJCbC1tYVMJivjKKg8paSkwNHREbGxsbCwsNB2OEQ6g68dIs3xdVN1CSGQmpqq9Cm8KlpLZFu1aoUffvgBiYmJsLa2BgBcvnwZANCyZctyayOXy5Xu4lpZWZXHEKiCWFhY8I8KUSnwtUOkOb5uqqaS7sQW0tqXvfz8/GBlZYXFixcDAHJycvDNN9/A29sbzs7OAIC0tDR4eHjg2LFjarchIiIion8HrSWy1apVw4EDB7Br1y44OjqiVq1aSE5OVnjAQV5eHi5fvoyXL1+q3YaIiIiI/h20+kAELy8vPHr0CFFRUZDL5XB1dVXYX61aNYSGhircbS2pDekuuVyOL7/8UmkqCBEVj68dIs3xdfNmkAl11jYgIiIiIqpitDa1gIiIiIioLJjIEhEREZFOYiJLle63334r8klsRG+i0NBQXLlyRdthEFV5Bw4cQFxcXJmPc+TIkRLfZ44dO4bo6Ogy90XaxUSWKt3UqVNx4cIFbYdBVGnWrFmDjRs3lvk4ly9fRlhYWIllRLpqzJgx5fKfvsmTJ5f4PjNz5kycOXOmzH2p49KlS7h69Wql9PVvo9VVC4iISH3r169HXl4ePDw8ii0joqpl1apVqF69Otq0aaPtUN44TGSpwsXFxeHq1atwdHRE8+bNVdbJyspCaGgoMjIy4O7uDicnJ2nf6dOnYWNjo/T0tlOnTsHW1rbIp7oRVTUpKSm4du0aMjIy4O3tDVNTU2nfzZs3cevWLQB/P32wefPmCo9nvH79OmJiYlBQUIDdu3cDAGrWrKlU1r17d1SvXh0AEB4ejpiYGNStWxdvvfUW9PT+9yFcaGgoDAwM0KhRI4SGhkIIgQYNGuD27dsYMGCAQty3b9/GX3/9pVROVFGePn2K8PBwWFpaon379grXLgBkZ2cjNDQUaWlpcHNzQ/369dU65u+//446deqgRYsWaseSm5uL0NBQxMfHw83NTWnZz1OnTsHe3h5NmzaVyi5fvgwhBDw8PHD16lU8fvwYycnJ0uu0d+/efMpoOWEiSxVq9+7dGDt2LNq0aYPc3FzIZDJkZGQo1AkJCcGQIUPg4OAAOzs7hISEYMKECfj2228B/P1HIjg4WOFjmZSUFPTv3x87d+5kIks64fr162jevDlcXV3x8OFDZGRk4MyZM3BxcQEAREVF4fDhwwCA+Ph4hISEYPHixfjwww8BALdu3cKjR48ghJDqtWnTRqmsZcuWMDQ0hL+/P+7evYsWLVogKioK1atXx9GjR6Ukd82aNbh37x6SkpLg4uICd3d3ODo6ws/PD3fu3FF4s/7oo49gZ2fHRJYqxcaNGzF79mw0adIEV65cgbu7O4KCgqRk9urVqxg0aBBq1qyJ2rVrIyQkBKNHj8aaNWuKPObBgwcxevRo6f0iJycHKSkpJcby4MED9OzZE/n5+XBxccGlS5cwYsQI/PTTT1KdBQsWwNfXVyGR/ecnJeHh4YiLi4NcLpdepx06dGAiW14EUQVJSkoS1tbWYtWqVVLZ/PnzBQCxY8cOIYQQqampws7OTmzZskWq8+jRI2FtbS1OnDghhBAiIiJCABC3b9+W6mzZskVYWVmJrKysShkLUVkMGzZMyGQycenSJSGEELm5uaJ3796ib9++RbYJCwsTRkZGIjY2ViobM2aMGDVqlEI9VWXvvvuuGDhwoMjJyRFCCJGXlyf69esnxo8frxCTkZGRuHXrlkJbDw8P8emnn0rbjx8/Fnp6euLcuXMajppIc2ZmZsLT01NkZmYKIf6+/uRyuTh+/LgQQojMzExRp04dsX79eqnNkydPRI0aNcShQ4ekspo1a0rvM2lpacLOzk4sXbpU2v/1118LAGLTpk3FxuPr6yu6du0qsrOzhRBC3LhxQxgZGYkDBw5Iddq3by++/vprhXavvy79/f3F5MmTNTkVpCZ+2YsqTHBwMLKzszF16lSpbM6cOZDJZNL28ePHkZycDDMzM+zbtw/79u1DaGgo6tevj7NnzwIA3Nzc0KJFC+zcuVNqt3PnTgwdOpRPZCGd0alTJ3Ts2BEAYGBggFmzZuH48eNIS0uT6iQlJeHcuXPYu3cvYmJiYGJigmvXrmnUT1ZWFnbv3o0mTZrgyJEj2LdvHw4cOIC6detKr6lC3bp1U7iLBAATJkzA9u3bkZ+fDwDYunUrGjRoAC8vr9IMm0hjY8eOhbGxMQCgTp06cHFxwZ07dwAAQUFBeP78OaytrbF//37s27cPFy5cQIMGDZSu70Jnz55FYmIiZs6cKZXNmjULBgbFfyidnp6OgIAAzJ49G0ZGRgCAZs2aYcCAAdIUAdI+Ti2gCvPo0SPY29tLfwAAwMLCAjY2NtL2gwcPYGhoiAMHDii0dXFxUZjzNGrUKGzYsAGLFi3C06dPcebMmSL/aBFVRfXq1VPYLry+Hz16hKZNm+LXX3/F+++/D1dXVzg4OEAulyM3NxcvXrzQqJ8nT54gJycHf/75p9LSQp07d1bYrl27tlL74cOH46OPPsKJEyfg6+uLrVu3YuzYsRrFQFQW/3yPAP5+lGxWVhaA/71nHDp0SKFOvXr1FB5n/0+PHj1CzZo1YWJiIpWZmpqiZs2a0vaNGzcQGRkpbffr1w+xsbEAgAYNGigcz9nZme8/VQgTWaowtra2SExMVCgrKChQmJdkYWGBvLw87Ny5E/r6+kUea+TIkZg7dy7CwsIQGhoKR0dHpTdloqrs9ddC4XbhnNXp06djxYoVmDhxolSnWrVqEBo+RdzCwgLA38vclTSn9Z+fjhQyMzPD8OHD8csvv6BatWqIiYnBmDFjNIqBqKJYWFggNzcX27ZtU/sTOVtbWyQlJSmV//M1eevWLRw5ckTa9vLykl6bCQkJCu0SEhKkfQCgp6eHgoIChTpZWVkl3vGl8sGpBVRhOnTogMTERFy8eFEqO3r0KHJzc6XtHj16ICcnB9u2bVNom5OTg5cvX0rbderUgbe3N3bu3ImdO3di5MiRKt+Eiaqqc+fOKbxxHjx4EC4uLrCzs0N2djYSExPRqFEjaX9gYKDCtAMAMDc3l+5MFVVWvXp1tG7dGhs2bFCKQd2F5idMmICAgAB8++236NWrF+rUqaNWO6KK1q1bNwDAzz//rFCel5dX5KcX7du3R1ZWFk6dOiWVnTx5UuGLxyNGjMDu3bulH3t7e9jZ2cHV1RUHDx6U6mVlZSEgIACenp5SWZ06dXDv3j1pu3BFhX9S9dql8sH/LlCFcXV1xcSJE+Hv7485c+YgLy8PP/zwgzT3Cfj7I5qlS5fi/fffx7Vr19CyZUs8ePAA+/fvx4YNG1CjRg2p7ujRozFz5kykpaVh+/bt2hgSUanJ5XJ07doVkyZNwr179/D9999j79690r7evXtj2rRpmDFjBp4+fYq1a9cqfBQK/L1Kwccff4wNGzbAysoK3bt3V1m2ceNG9OjRAz169MDgwYORkZGB4OBgNGvWDMuXLy8x1nbt2qFJkyY4fvw49u/fXyHng6g0HB0dsWLFCnz44Ye4efMm2rRpg9jYWOzfvx+rVq1Cjx49lNrUq1cP06ZNw/DhwzF37lwIIfD9998rLH9XlNWrV8PPzw/Z2dlwc3PDtm3bYG5ujhkzZkh13n33Xfj7+8PJyQkODg7YuXOn0uo8bdq0weLFi7Fp0yZUq1aNy2+VIyayVKHWrVuH1q1b49KlS3B0dMTp06fx/fffK8wXnDNnDry9vbF//36cP38eLi4uOHbsmNK6gP7+/ggODoadnZ3SF1SIqrKOHTuia9euqFu3LgIDA5GRkYGgoCD4+PhIdfbu3Ysff/wRFy9eRI0aNXDq1Cls2rRJYd7fO++8g9zcXPz+++9ITU1Fy5YtVZa1bt0akZGR2Lp1Ky5fvgw7OzvMnj0b3bt3V4jpn/+pfN3AgQPx5MkT9O/fv2JOCpEKhUsx/lOvXr0U/ubPnDkTnTp1wt69e3HhwgU4OzvjyJEjCq+VgQMHKrzPfPfdd2jRogXOnz8Pe3t7BAYGKr2+VOnTpw9CQ0Pxf//3fwgJCcHAgQMxadIkmJmZSXX69++Po0eP4siRI8jIyMCiRYsQFRWlMN1g8uTJ0NfXx+XLl5GWlsblt8qRTGg6AYuIiN547dq1g5eXF1asWKHtUIiIisQ7skREJDl58iROnTqFqKgohbmBRERVEb/sRUREksI1NwMDA5U+4iUiqmo4tYCIiIiIdBLvyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTmIiS0REREQ6iYksEREREekkJrJEREREpJOYyBIRERGRTvp/Yvgr65i6o+MAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 700x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rng = np.random.default_rng(0)\n",
    "def boot_diff(a, b, n=4000):\n",
    "    # a, b: per-question 0/1 correctness for the same questions; the interval of mean(b) - mean(a)\n",
    "    a, b = np.array(a, float), np.array(b, float)\n",
    "    d = [(b[s] - a[s]).mean() for s in (rng.integers(0, len(a), len(a)) for _ in range(n))]\n",
    "    return np.percentile(d, [2.5, 97.5])\n",
    "\n",
    "v1, v2 = answer_v1, make_system(COV, BOOST)\n",
    "rows = {}\n",
    "print(f\"{'set':<10}{'questions':>10}{'first system':>14}{'improved':>10}{'change':>9}   95% interval of the change\")\n",
    "for name, S in ((\"dev\", DEV), (\"battery\", BATTERY), (\"held-out\", HELDOUT)):\n",
    "    a = [g == \"ok\" for _, g in report(v1, S)[3]]\n",
    "    b = [g == \"ok\" for _, g in report(v2, S)[3]]\n",
    "    lo, hi = boot_diff(a, b)\n",
    "    rows[name] = (np.mean(a), np.mean(b))\n",
    "    print(f\"{name:<10}{len(S):>10}{np.mean(a):>14.2f}{np.mean(b):>10.2f}{np.mean(b) - np.mean(a):>+9.2f}   [{lo:+.2f}, {hi:+.2f}]\")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 3.2))\n",
    "x = np.arange(3)\n",
    "ax.bar(x - 0.18, [rows[k][0] for k in rows], 0.36, label=\"first system\", color=\"#8a8aa0\")\n",
    "ax.bar(x + 0.18, [rows[k][1] for k in rows], 0.36, label=\"improved\", color=\"#E1C340\")\n",
    "ax.set_xticks(x); ax.set_xticklabels(list(rows)); ax.set_ylim(0, 1); ax.set_ylabel(\"correct\"); ax.legend(frameon=False)\n",
    "ax.set_title(\"State of the Union assistant: share of correct answers\")\n",
    "plt.tight_layout(); plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6fed20ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:34:56.132241Z",
     "iopub.status.busy": "2026-11-17T21:34:56.131919Z",
     "iopub.status.idle": "2026-11-17T21:35:02.479974Z",
     "shell.execute_reply": "2026-11-17T21:35:02.478065Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "improved system, battery: 19 of 28   direct 6/7  trap 1/1  paraphrase 3/8  who 4/6  unanswerable 5/6\n",
      "  an unanswerable question that was still answered: u01 What did the president say about the Internet? -> 2006-042\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "improved system, held-out: 10 of 16   paraphrase 4/6  direct 2/5  who 1/1  unanswerable 3/4\n",
      "\n",
      "what still fails on the held-out set:\n",
      "  e01 [missed] How many adults cannot read well enough to function?\n",
      "        -> None \n",
      "  e03 [wrong] How many people live in the country, according to the 1974 a\n",
      "        -> 1974-079 But my colleagues, this I believe: With the help of God, who has blessed this \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  e05 [wrong] How many more police officers will be funded?\n",
      "        -> 1974-057 We have cut our military manpower over the past 5 years by more than a third, \n",
      "  e10 [missed] What happened to farm income in 1961?\n",
      "        -> None \n",
      "  e11 [missed] How many private peace talks took place in 1965?\n",
      "        -> None \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  v04 [wrong] What did Reagan say about Gorbachev?\n",
      "        -> 1982-067 There must be a real incentive for the Soviets to take these talks seriously.\n"
     ]
    }
   ],
   "source": [
    "ok, n, by, res = report(v2, BATTERY)\n",
    "print(\"improved system, battery:\", ok, \"of\", n, \" \", \"  \".join(f\"{t} {a}/{b}\" for t, (a, b) in by.items()))\n",
    "for q, g in res:\n",
    "    if q[2] is None and g != \"ok\":\n",
    "        print(\"  an unanswerable question that was still answered:\", q[0], q[1], \"->\", v2(q[1])[1])\n",
    "ok, n, by, res = report(v2, HELDOUT)\n",
    "print(\"improved system, held-out:\", ok, \"of\", n, \" \", \"  \".join(f\"{t} {a}/{b}\" for t, (a, b) in by.items()))\n",
    "print(\"\\nwhat still fails on the held-out set:\")\n",
    "for q, g in res:\n",
    "    if g != \"ok\":\n",
    "        kind, cid, sent = v2(q[1])\n",
    "        print(f\"  {q[0]} [{g}] {q[1][:60]}\\n        -> {cid} {(sent or '')[:78]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da384b1d",
   "metadata": {},
   "source": [
    "## 7. Lo que enseñó este corpus y el proyecto final no podía\n",
    "\n",
    "Resultados, en el orden de cuánto podemos fiarnos de ellos:\n",
    "\n",
    "* **Rechazar lo que nunca se dijo es la mejora real.** Las preguntas sin respuesta pasaron de 0 de 6 en la batería a 5 de 6, y de 0 de 4 a 3 de 4 en preguntas retenidas que los arreglos nunca vieron. Esta mejora es grande y generalizó. Un umbral de similitud por sí solo no podía lograrlo, porque el tema de una pregunta sin respuesta suele estar cerca de algo que **sí** se dijo, en otro discurso o en una frase cercana.\n",
    "* **La única pregunta sin respuesta que aún recibe una respuesta muestra una debilidad de la regla.** \"What did the president say about the Internet?\" solo tiene dos palabras de contenido tras quitar las genéricas, y la palabra de relleno \"about\" aparece en todos los fragmentos, así que la cobertura es del 50 %, justo en el umbral. Una lista de palabras genéricas crece con cada fallo; ponderar las palabras por IDF sería el arreglo con fundamento (ejercicio 3).\n",
    "* **La mejora global es plausible pero no está demostrada.** Las respuestas correctas pasaron de 0,50 a 0,68 en la batería y de 0,44 a 0,62 en las preguntas retenidas, pero todos los intervalos del 95 % incluyen el cero. Con 28 y 16 preguntas es lo mejor que podemos decir, y el módulo 07 ya calculó cuántas preguntas harían falta para estar seguros.\n",
    "* **El texto real es mucho más difícil que un texto escrito por ti.** El primer sistema sacó 0,75 en el corpus del proyecto final y 0,50 aquí con el mismo método. Documentos largos y retóricos, un recuperador que mezcla épocas y respuestas enterradas en frases que repiten la pregunta cuestan precisión. Una buena nota en tu propio corpus de prueba no es evidencia sobre documentos reales.\n",
    "* **Los arreglos cuestan cobertura.** La comprobación de cobertura rechaza algunas preguntas con respuesta cuya redacción está lejos del texto (\"How many private peace talks took place in 1965?\"). Toda regla de rechazo cambia respuestas erróneas por respuestas perdidas, y esta se ajustó con 14 preguntas con respuesta.\n",
    "\n",
    "**Qué cambia en el bucle de mejora.** El proyecto final reparó la documentación. Aquí el supervisor puede seguir marcando el fallo y una persona puede seguir leyendo los casos marcados, pero el arreglo va al lado de la consulta: un filtro, una regla de rechazo, una preferencia de ranking. Esos arreglos son generales, por eso se transfirieron a preguntas nuevas; un parche escrito para una pregunta que fallaba no lo habría hecho."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "75b81314",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:35:02.483585Z",
     "iopub.status.busy": "2026-11-17T21:35:02.483268Z",
     "iopub.status.idle": "2026-11-17T21:35:02.491549Z",
     "shell.execute_reply": "2026-11-17T21:35:02.489912Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                      capstone (Thermio)   real speeches\n",
      "documents you can edit                               yes              no\n",
      "passages                                              25             763\n",
      "first system, correct on battery                    0.75            0.50\n",
      "after improvements, battery                         0.96            0.68\n",
      "after improvements, held-out                           -            0.62\n"
     ]
    }
   ],
   "source": [
    "print(f\"{'':<34}{'capstone (Thermio)':>22}{'real speeches':>16}\")\n",
    "print(f\"{'documents you can edit':<34}{'yes':>22}{'no':>16}\")\n",
    "print(f\"{'passages':<34}{25:>22}{len(P.chunks):>16}\")\n",
    "print(f\"{'first system, correct on battery':<34}{'0.75':>22}{rows['battery'][0]:>16.2f}\")\n",
    "print(f\"{'after improvements, battery':<34}{'0.96':>22}{rows['battery'][1]:>16.2f}\")\n",
    "print(f\"{'after improvements, held-out':<34}{'-':>22}{rows['held-out'][1]:>16.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eac32263",
   "metadata": {},
   "source": [
    "## Ejercicios\n",
    "\n",
    "1. Cambia `CHUNK_WORDS` por el valor que prefiere la **batería** y vuelve a ejecutar la sección 5. ¿Cambia la conclusión sobre los arreglos? ¿Por qué elegir con la batería es un error aunque dé un número más bonito?\n",
    "2. Añade un discurso (cualquier otro año del corpus `state_union` está en los datos de NLTK, o usa tu propio texto) y escribe seis preguntas con claves. ¿Sigue funcionando el filtro de orador para un presidente que dio varios discursos?\n",
    "3. La comprobación de cobertura rechaza por palabras que faltan. Encuentra una pregunta con respuesta que rechaza por error y propón un arreglo más barato que bajar el umbral. Después haz lo contrario con la pregunta sobre Internet: pondera cada palabra por su IDF en lugar de contar palabras y comprueba si la palabra de relleno \"about\" deja de importar.\n",
    "4. Sustituye el selector de frases por una llamada a un modelo (el patrón de celda opcional del módulo 04) y mide, con la misma batería, si el refuerzo de cantidades sigue siendo necesario.\n",
    "5. Escribe diez preguntas nuevas sin respuesta sobre temas que **sí** aparecen en otro discurso. ¿Cuántas rechaza el sistema mejorado? ¿Qué necesitaría un supervisor para detectar el resto?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
