{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5f07e21",
   "metadata": {},
   "source": [
    "# 11 · Proyecto final: un asistente RAG supervisado\n",
    "\n",
    "Diez módulos, diez piezas. Este proyecto final las ensambla en un único sistema que funciona sobre un corpus\n",
    "**nuevo**, para que no puedas apoyarte en nada que hayas ajustado antes: la base de conocimiento de soporte de una\n",
    "empresa ficticia de termostatos inteligentes, *Thermio*, más una pequeña tabla de ventas. Después recorre todo el\n",
    "bucle del curso:\n",
    "\n",
    "> construir → responder → trazar → **juzgar** → encontrar lo que falló → **arreglar la documentación** → reindexar → volver a medir\n",
    "\n",
    "Todo es determinista y sin conexión: un generador de sustitución y un supervisor de sustitución ocupan el lugar de\n",
    "los modelos de lenguaje, exactamente como en los módulos anteriores, de modo que los números que ves los produce\n",
    "el código que puedes leer. Cambiar a modelos reales modifica la calidad de los dos sustitutos, no la arquitectura.\n",
    "\n",
    "**Aprenderás**\n",
    "1. Cómo encajan las piezas de los módulos 01 a 10 en un único pipeline, con un enrutador que manda cada pregunta a la recuperación de pasajes, a SQL o a un rechazo\n",
    "2. A elegir el peso de recuperación y el umbral de rechazo con un **conjunto de desarrollo**, y a evaluar después con una batería distinta\n",
    "3. Una respuesta tipada que no puede existir sin una cita (Pydantic)\n",
    "4. Un supervisor que juzga cada respuesta, y si fiarse de él (medimos su acuerdo con las referencias)\n",
    "5. Convertir los fallos en un arreglo de la documentación, reindexar y demostrar la mejora con intervalos de confianza\n",
    "6. Una lista de comprobación antes de poner un sistema RAG delante de los usuarios"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6460cac8",
   "metadata": {},
   "source": [
    "## 1. El sistema que vamos a construir\n",
    "\n",
    "```\n",
    " pregunta ──► enrutador ──┬─ pasaje ─► recuperación (BM25 + embeddings, RRF) ─► generador ─┐\n",
    "                          ├─ sql ────► text-to-SQL ─► guardián ─► DuckDB ──────────────────┤\n",
    "                          └─ rechazo (no se encontró nada relevante) ──────────────────────┤\n",
    "                                                                                            ▼\n",
    "                     Answer tipada (citas obligatorias) ─► registro de trazas (tokens, coste, latencia)\n",
    "                                                                  │\n",
    "                                       el supervisor juzga cada respuesta ─► resumen diario\n",
    "                                                                  │\n",
    "                    análisis de fallos de recuperación ─► arreglo de la documentación ─► reindexar ─► volver a medir\n",
    "```\n",
    "\n",
    "| Componente | Construido en el módulo |\n",
    "|---|---|\n",
    "| troceado por estructura del documento, BM25, hit@k y MRR | 01 |\n",
    "| embeddings y recuperación híbrida con Reciprocal Rank Fusion | 02 |\n",
    "| salida tipada con Pydantic | 04 |\n",
    "| un pipeline como pasos explícitos con un enrutador (LangGraph expresa lo mismo como un grafo) | 05 |\n",
    "| respuestas desde datos con un guardián SQL de solo lectura | 06 |\n",
    "| batería de preguntas, calificación, intervalos de confianza por bootstrap | 07 |\n",
    "| el supervisor, el bucle que arregla la documentación | 08 |\n",
    "| trazas, tokens, coste | 09 |\n",
    "| rechazo, hábitos de seguridad, modelo de coste | 10 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d665cf90",
   "metadata": {},
   "source": [
    "### Imagen: el bucle de mejora\n",
    "\n",
    "Todo el proyecto final es este bucle, ejecutado una vez. Las cajas 1 a 5 construyen, responden, observan y\n",
    "diagnostican; solo la caja 6 cambia algo, y lo que cambia es la documentación. La caja 8 mide el resultado sobre la\n",
    "misma batería, con un intervalo, para que la mejora no se dé por buena sin comprobarla."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0990cf15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:43.376215Z",
     "iopub.status.busy": "2026-11-17T21:23:43.375870Z",
     "iopub.status.idle": "2026-11-17T21:23:43.447276Z",
     "shell.execute_reply": "2026-11-17T21:23:43.445210Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 820 345\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"820\" height=\"345\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"410.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">El bucle de mejora: arregla la fuente de verdad y vuelve a medir</text><rect x=\"45\" y=\"50\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"125.0\" y=\"77.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">1 Construir</text><text x=\"125.0\" y=\"94.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">25 pasajes, peso híbrido</text><text x=\"125.0\" y=\"111.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">elegido en desarrollo</text><line x1=\"205\" y1=\"90\" x2=\"235\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"235\" y=\"50\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"315.0\" y=\"77.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">2 Responder</text><text x=\"315.0\" y=\"94.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">enrutador: pasaje, sql</text><text x=\"315.0\" y=\"111.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">o rechazo; tipada</text><line x1=\"395\" y1=\"90\" x2=\"425\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"425\" y=\"50\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"505.0\" y=\"77.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">3 Trazar</text><text x=\"505.0\" y=\"94.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">tokens, coste, latencia</text><text x=\"505.0\" y=\"111.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">por petición</text><line x1=\"585\" y1=\"90\" x2=\"615\" y2=\"90\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"615\" y=\"50\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"695.0\" y=\"77.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">4 Juzgar</text><text x=\"695.0\" y=\"94.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">supervisor: ¿coinciden</text><text x=\"695.0\" y=\"111.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">dos clasificadores?</text><line x1=\"695\" y1=\"130\" x2=\"695\" y2=\"180\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"615\" y=\"180\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"695.0\" y=\"207.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">5 Hallar fallos</text><text x=\"695.0\" y=\"224.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">posición del pasaje y</text><text x=\"695.0\" y=\"241.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">palabras que faltan</text><line x1=\"615\" y1=\"220\" x2=\"585\" y2=\"220\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"425\" y=\"180\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"505.0\" y=\"207.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">6 Arreglar docs</text><text x=\"505.0\" y=\"224.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">no el código ni</text><text x=\"505.0\" y=\"241.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">los umbrales</text><line x1=\"425\" y1=\"220\" x2=\"395\" y2=\"220\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"235\" y=\"180\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"315.0\" y=\"207.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">7 Reindexar</text><text x=\"315.0\" y=\"224.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">mismo código, nueva</text><text x=\"315.0\" y=\"241.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">base de conocimiento</text><line x1=\"235\" y1=\"220\" x2=\"205\" y2=\"220\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"45\" y=\"180\" width=\"160\" height=\"80\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"125.0\" y=\"207.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">8 Volver a medir</text><text x=\"125.0\" y=\"224.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">misma batería,</text><text x=\"125.0\" y=\"241.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">intervalo bootstrap</text><line x1=\"125\" y1=\"180\" x2=\"125\" y2=\"130\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"135\" y=\"160\" text-anchor=\"start\" font-size=\"12\" fill=\"#333\">siguiente ronda</text><rect x=\"45\" y=\"280\" width=\"730\" height=\"50\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/><text x=\"55\" y=\"300\" font-size=\"12\" fill=\"#444\">Medido aquí: 18 de 24 correctas antes (75%), 23 de 24 después, sin regresiones.</text><text x=\"55\" y=\"316\" font-size=\"12\" fill=\"#444\">En las 20 preguntas cuya referencia no cambió: mejora +0.14, IC 95% [0.00, 0.29].</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import SVG, display\n",
    "import html as _h\n",
    "\n",
    "_COL = {  # fill, stroke  (light cards with dark text: readable in light and dark themes)\n",
    "    \"gold\": (\"#fff4c2\", \"#b89d28\"), \"blue\": (\"#dbeafe\", \"#3b82f6\"), \"green\": (\"#dcfce7\", \"#16a34a\"),\n",
    "    \"red\": (\"#fee2e2\", \"#dc2626\"), \"grey\": (\"#f1f1f4\", \"#8a8aa0\"), \"violet\": (\"#ede9fe\", \"#7c3aed\"),\n",
    "}\n",
    "\n",
    "def diagram(width, height, items, title=None):\n",
    "    # items: (\"box\", x, y, w, h, \"text\\nmore text\", colour) | (\"arrow\", x1, y1, x2, y2, \"label\") |\n",
    "    #        (\"text\", x, y, \"text\", size, \"start|middle|end\") | (\"note\", x, y, w, h, \"text\")\n",
    "    out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 {width} {height}\" width=\"100%\" '\n",
    "           f'font-family=\"Helvetica, Arial, sans-serif\">',\n",
    "           '<defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" '\n",
    "           'orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs>',\n",
    "           f'<rect width=\"{width}\" height=\"{height}\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/>']\n",
    "    if title:\n",
    "        out.append(f'<text x=\"{width / 2}\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">{_h.escape(title)}</text>')\n",
    "    for it in items:\n",
    "        kind = it[0]\n",
    "        if kind == \"box\":\n",
    "            _, x, y, w, h, text, col = it\n",
    "            fill, stroke = _COL[col]\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"9\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"1.6\"/>')\n",
    "            lines = text.split(\"\\n\")\n",
    "            y0 = y + h / 2 - (len(lines) - 1) * 8.5 + 4.5\n",
    "            for k, ln in enumerate(lines):\n",
    "                bold = ' font-weight=\"700\"' if k == 0 and len(lines) > 1 else \"\"\n",
    "                out.append(f'<text x=\"{x + w / 2}\" y=\"{y0 + k * 17}\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\"{bold}>{_h.escape(ln)}</text>')\n",
    "        elif kind == \"arrow\":\n",
    "            _, x1, y1, x2, y2, label = it\n",
    "            out.append(f'<line x1=\"{x1}\" y1=\"{y1}\" x2=\"{x2}\" y2=\"{y2}\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/>')\n",
    "            if label:\n",
    "                out.append(f'<text x=\"{(x1 + x2) / 2}\" y=\"{(y1 + y2) / 2 - 6}\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">{_h.escape(label)}</text>')\n",
    "        elif kind == \"text\":\n",
    "            _, x, y, text, size, anchor = it\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x}\" y=\"{y + k * (size + 4)}\" text-anchor=\"{anchor}\" font-size=\"{size}\" fill=\"#333\">{_h.escape(ln)}</text>')\n",
    "        elif kind == \"note\":\n",
    "            _, x, y, w, h, text = it\n",
    "            out.append(f'<rect x=\"{x}\" y=\"{y}\" width=\"{w}\" height=\"{h}\" rx=\"6\" fill=\"none\" stroke=\"#b89d28\" stroke-dasharray=\"5 4\"/>')\n",
    "            for k, ln in enumerate(text.split(\"\\n\")):\n",
    "                out.append(f'<text x=\"{x + 10}\" y=\"{y + 20 + k * 16}\" font-size=\"12\" fill=\"#444\">{_h.escape(ln)}</text>')\n",
    "    out.append(\"</svg>\")\n",
    "    display(SVG(\"\".join(out)))\n",
    "\n",
    "\n",
    "diagram(820, 345, [\n",
    "    (\"box\", 45, 50, 160, 80, \"1 Construir\\n25 pasajes, peso híbrido\\nelegido en desarrollo\", \"grey\"),\n",
    "    (\"arrow\", 205, 90, 235, 90, \"\"),\n",
    "    (\"box\", 235, 50, 160, 80, \"2 Responder\\nenrutador: pasaje, sql\\no rechazo; tipada\", \"green\"),\n",
    "    (\"arrow\", 395, 90, 425, 90, \"\"),\n",
    "    (\"box\", 425, 50, 160, 80, \"3 Trazar\\ntokens, coste, latencia\\npor petición\", \"blue\"),\n",
    "    (\"arrow\", 585, 90, 615, 90, \"\"),\n",
    "    (\"box\", 615, 50, 160, 80, \"4 Juzgar\\nsupervisor: ¿coinciden\\ndos clasificadores?\", \"violet\"),\n",
    "    (\"arrow\", 695, 130, 695, 180, \"\"),\n",
    "    (\"box\", 615, 180, 160, 80, \"5 Hallar fallos\\nposición del pasaje y\\npalabras que faltan\", \"red\"),\n",
    "    (\"arrow\", 615, 220, 585, 220, \"\"),\n",
    "    (\"box\", 425, 180, 160, 80, \"6 Arreglar docs\\nno el código ni\\nlos umbrales\", \"gold\"),\n",
    "    (\"arrow\", 425, 220, 395, 220, \"\"),\n",
    "    (\"box\", 235, 180, 160, 80, \"7 Reindexar\\nmismo código, nueva\\nbase de conocimiento\", \"grey\"),\n",
    "    (\"arrow\", 235, 220, 205, 220, \"\"),\n",
    "    (\"box\", 45, 180, 160, 80, \"8 Volver a medir\\nmisma batería,\\nintervalo bootstrap\", \"violet\"),\n",
    "    (\"arrow\", 125, 180, 125, 130, \"\"),\n",
    "    (\"text\", 135, 160, \"siguiente ronda\", 12, \"start\"),\n",
    "    (\"note\", 45, 280, 730, 50, \"Medido aquí: 18 de 24 correctas antes (75%), 23 de 24 después, sin regresiones.\\nEn las 20 preguntas cuya referencia no cambió: mejora +0.14, IC 95% [0.00, 0.29].\"),\n",
    "], \"El bucle de mejora: arregla la fuente de verdad y vuelve a medir\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4793397c",
   "metadata": {},
   "source": [
    "### Imagen: una pregunta real, de principio a fin\n",
    "\n",
    "La pregunta es `c01` de la batería. Cada número de las cajas lo imprime una celda más abajo. Las rutas de SQL y de\n",
    "rechazo son las otras dos salidas del enrutador; el supervisor y el arreglo son el bucle de la imagen anterior\n",
    "aplicado a este único fallo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9d5398e1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:43.451158Z",
     "iopub.status.busy": "2026-11-17T21:23:43.450868Z",
     "iopub.status.idle": "2026-11-17T21:23:43.467265Z",
     "shell.execute_reply": "2026-11-17T21:23:43.465361Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 840 470\" width=\"100%\" font-family=\"Helvetica, Arial, sans-serif\"><defs><marker id=\"ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path d=\"M0,0 L10,5 L0,10 z\" fill=\"#555\"/></marker></defs><rect width=\"840\" height=\"470\" rx=\"12\" fill=\"#fbfbf8\" stroke=\"#d8d8cf\"/><text x=\"420.0\" y=\"26\" text-anchor=\"middle\" font-size=\"15\" font-weight=\"700\" fill=\"#222\">Una pregunta real, de principio a fin</text><rect x=\"20\" y=\"140\" width=\"160\" height=\"90\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"100.0\" y=\"164.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Pregunta c01</text><text x=\"100.0\" y=\"181.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">Can I connect the</text><text x=\"100.0\" y=\"198.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">thermostat to a 5 GHz</text><text x=\"100.0\" y=\"215.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">network?</text><line x1=\"180\" y1=\"185\" x2=\"210\" y2=\"185\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"210\" y=\"150\" width=\"120\" height=\"70\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"270.0\" y=\"172.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Enrutador</text><text x=\"270.0\" y=\"189.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">sin ventas,</text><text x=\"270.0\" y=\"206.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">luego: pasaje</text><line x1=\"270\" y1=\"150\" x2=\"370\" y2=\"100\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"370\" y=\"40\" width=\"190\" height=\"70\" rx=\"9\" fill=\"#f1f1f4\" stroke=\"#8a8aa0\" stroke-width=\"1.6\"/><text x=\"465.0\" y=\"62.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Ruta SQL, p. ej. s01</text><text x=\"465.0\" y=\"79.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">un SELECT vigilado</text><text x=\"465.0\" y=\"96.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">en DuckDB: 2465 [1]</text><line x1=\"330\" y1=\"185\" x2=\"370\" y2=\"185\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"370\" y=\"150\" width=\"190\" height=\"70\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"465.0\" y=\"172.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Recuperación</text><text x=\"465.0\" y=\"189.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">coseno máx. 0.35 &gt; 0.25</text><text x=\"465.0\" y=\"206.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">top 3: t07, t01, t02</text><line x1=\"465\" y1=\"220\" x2=\"465\" y2=\"270\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><text x=\"465.0\" y=\"239.0\" text-anchor=\"middle\" font-size=\"11.5\" fill=\"#555\">por debajo de 0.25</text><rect x=\"370\" y=\"270\" width=\"190\" height=\"60\" rx=\"9\" fill=\"#fee2e2\" stroke=\"#dc2626\" stroke-width=\"1.6\"/><text x=\"465.0\" y=\"287.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Ruta de rechazo</text><text x=\"465.0\" y=\"304.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">'No lo encuentro</text><text x=\"465.0\" y=\"321.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">en la documentación'</text><line x1=\"560\" y1=\"185\" x2=\"600\" y2=\"185\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"560\" y1=\"75\" x2=\"660\" y2=\"150\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><line x1=\"560\" y1=\"300\" x2=\"600\" y2=\"215\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"600\" y=\"150\" width=\"220\" height=\"70\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"710.0\" y=\"172.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Respuesta tipada</text><text x=\"710.0\" y=\"189.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">cita t07 (Offline):</text><text x=\"710.0\" y=\"206.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">restart the router [1]</text><line x1=\"710\" y1=\"220\" x2=\"710\" y2=\"260\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"600\" y=\"260\" width=\"220\" height=\"60\" rx=\"9\" fill=\"#dbeafe\" stroke=\"#3b82f6\" stroke-width=\"1.6\"/><text x=\"710.0\" y=\"277.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Traza</text><text x=\"710.0\" y=\"294.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">91 tokens de entrada,</text><text x=\"710.0\" y=\"311.5\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">25 de salida, $0.0000067, 345 ms</text><line x1=\"710\" y1=\"320\" x2=\"710\" y2=\"350\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"600\" y=\"350\" width=\"220\" height=\"80\" rx=\"9\" fill=\"#ede9fe\" stroke=\"#7c3aed\" stroke-width=\"1.6\"/><text x=\"710.0\" y=\"369.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Supervisor</text><text x=\"710.0\" y=\"386.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">el recuperador eligió t07,</text><text x=\"710.0\" y=\"403.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">BM25 prefiere t01:</text><text x=\"710.0\" y=\"420.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">confianza baja</text><line x1=\"600\" y1=\"390\" x2=\"560\" y2=\"390\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"340\" y=\"350\" width=\"220\" height=\"80\" rx=\"9\" fill=\"#fff4c2\" stroke=\"#b89d28\" stroke-width=\"1.6\"/><text x=\"450.0\" y=\"369.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Arreglo de documentación</text><text x=\"450.0\" y=\"386.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">t01 añade: It cannot</text><text x=\"450.0\" y=\"403.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">connect to 5 GHz</text><text x=\"450.0\" y=\"420.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">networks, only 2.4 GHz</text><line x1=\"340\" y1=\"390\" x2=\"300\" y2=\"390\" stroke=\"#555\" stroke-width=\"1.8\" marker-end=\"url(#ah)\"/><rect x=\"80\" y=\"350\" width=\"220\" height=\"80\" rx=\"9\" fill=\"#dcfce7\" stroke=\"#16a34a\" stroke-width=\"1.6\"/><text x=\"190.0\" y=\"369.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\" font-weight=\"700\">Tras reindexar</text><text x=\"190.0\" y=\"386.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">c01 ya es correcta</text><text x=\"190.0\" y=\"403.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">y nada más</text><text x=\"190.0\" y=\"420.0\" text-anchor=\"middle\" font-size=\"13\" fill=\"#222\">se volvió incorrecta</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diagram(840, 470, [\n",
    "    (\"box\", 20, 140, 160, 90, \"Pregunta c01\\nCan I connect the\\nthermostat to a 5 GHz\\nnetwork?\", \"gold\"),\n",
    "    (\"arrow\", 180, 185, 210, 185, \"\"),\n",
    "    (\"box\", 210, 150, 120, 70, \"Enrutador\\nsin ventas,\\nluego: pasaje\", \"blue\"),\n",
    "    (\"arrow\", 270, 150, 370, 100, \"\"),\n",
    "    (\"box\", 370, 40, 190, 70, \"Ruta SQL, p. ej. s01\\nun SELECT vigilado\\nen DuckDB: 2465 [1]\", \"grey\"),\n",
    "    (\"arrow\", 330, 185, 370, 185, \"\"),\n",
    "    (\"box\", 370, 150, 190, 70, \"Recuperación\\ncoseno máx. 0.35 > 0.25\\ntop 3: t07, t01, t02\", \"blue\"),\n",
    "    (\"arrow\", 465, 220, 465, 270, \"por debajo de 0.25\"),\n",
    "    (\"box\", 370, 270, 190, 60, \"Ruta de rechazo\\n'No lo encuentro\\nen la documentación'\", \"red\"),\n",
    "    (\"arrow\", 560, 185, 600, 185, \"\"),\n",
    "    (\"arrow\", 560, 75, 660, 150, \"\"),\n",
    "    (\"arrow\", 560, 300, 600, 215, \"\"),\n",
    "    (\"box\", 600, 150, 220, 70, \"Respuesta tipada\\ncita t07 (Offline):\\nrestart the router [1]\", \"green\"),\n",
    "    (\"arrow\", 710, 220, 710, 260, \"\"),\n",
    "    (\"box\", 600, 260, 220, 60, \"Traza\\n91 tokens de entrada,\\n25 de salida, $0.0000067, 345 ms\", \"blue\"),\n",
    "    (\"arrow\", 710, 320, 710, 350, \"\"),\n",
    "    (\"box\", 600, 350, 220, 80, \"Supervisor\\nel recuperador eligió t07,\\nBM25 prefiere t01:\\nconfianza baja\", \"violet\"),\n",
    "    (\"arrow\", 600, 390, 560, 390, \"\"),\n",
    "    (\"box\", 340, 350, 220, 80, \"Arreglo de documentación\\nt01 añade: It cannot\\nconnect to 5 GHz\\nnetworks, only 2.4 GHz\", \"gold\"),\n",
    "    (\"arrow\", 340, 390, 300, 390, \"\"),\n",
    "    (\"box\", 80, 350, 220, 80, \"Tras reindexar\\nc01 ya es correcta\\ny nada más\\nse volvió incorrecta\", \"green\"),\n",
    "], \"Una pregunta real, de principio a fin\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da68e9bb",
   "metadata": {},
   "source": [
    "## 2. Los datos: una base de conocimiento, una tabla, tres conjuntos de preguntas\n",
    "\n",
    "* **La base de conocimiento:** 25 pasajes de soporte, cada uno con un título de sección.\n",
    "* **La tabla de ventas:** unidades e ingresos por mes, producto y región, cargada en DuckDB.\n",
    "* **El conjunto de desarrollo:** 12 preguntas (8 paráfrasis y 4 que usan términos exactos como `E45`) usadas **solo** para elegir el peso de recuperación y el umbral de rechazo.\n",
    "* **La batería:** 24 preguntas con respuesta conocida, usadas **solo** para evaluar: 16 respondibles desde pasajes,\n",
    "  4 desde la tabla de ventas y 4 que nada puede responder (incluida una trampa con un código de error verosímil).\n",
    "\n",
    "Mantener separados el desarrollo y la batería es la disciplina del módulo 07: ajusta con un conjunto, informa con el otro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f1bd0168",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:43.471023Z",
     "iopub.status.busy": "2026-11-17T21:23:43.470709Z",
     "iopub.status.idle": "2026-11-17T21:23:53.918075Z",
     "shell.execute_reply": "2026-11-17T21:23:53.915872Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/patito/Developments/academy/rag-systems/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25 pasajes; 36 filas de ventas; 8245 unidades en total\n",
      "{'month': '2026-01', 'product': 'Thermio One', 'region': 'North', 'units': 124, 'revenue_eur': 15996}\n"
     ]
    }
   ],
   "source": [
    "import re, math, json, random, time, itertools\n",
    "from collections import defaultdict\n",
    "from collections import Counter\n",
    "import numpy as np\n",
    "import duckdb\n",
    "import matplotlib.pyplot as plt\n",
    "from typing import Literal\n",
    "from pydantic import BaseModel, model_validator, ValidationError\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "KB = [\n",
    "    (\"t01\", \"Setup / Wi-Fi\", \"The Thermio One connects to 2.4 GHz Wi-Fi only. Hold the ring button for 5 seconds until it blinks blue, then pick your network in the app.\"),\n",
    "    (\"t02\", \"Setup / Wiring\", \"Thermio One works with 24 V heating and cooling systems and needs a C-wire. If your system has no C-wire, install the included power adapter at the furnace.\"),\n",
    "    (\"t03\", \"Setup / Compatibility\", \"Thermio One is not compatible with 120 V or 240 V line-voltage baseboard heaters. Use Thermio Line for those systems.\"),\n",
    "    (\"t04\", \"Features / Schedules\", \"Schedules support up to 6 changes per day, with separate weekday and weekend programs. Changes apply within one minute of saving.\"),\n",
    "    (\"t05\", \"Features / Geofencing\", \"Away mode via geofencing starts when the last phone leaves a 500 m radius, and it sets the temperature back by 4 degrees.\"),\n",
    "    (\"t06\", \"Features / Eco mode\", \"Eco mode limits heating to 16 degrees and cooling to 28 degrees. The heating limit can be changed in settings between 10 and 18 degrees.\"),\n",
    "    (\"t07\", \"Troubleshooting / Offline\", \"If the thermostat shows Offline, restart the router, then hold the ring button for 10 seconds to reconnect. Settings are kept.\"),\n",
    "    (\"t08\", \"Troubleshooting / Blank display\", \"A blank display usually means no power. Check the C-wire and the furnace breaker. The battery backup keeps the display on for about 2 hours.\"),\n",
    "    (\"t09\", \"Troubleshooting / Error E21\", \"Error E21 means the temperature sensor reading is out of range. Remove the thermostat from its base for 30 seconds and refit it. If E21 returns, contact support for a replacement.\"),\n",
    "    (\"t10\", \"Troubleshooting / Error E45\", \"Error E45 means the system is short cycling. The thermostat waits 5 minutes between heating starts to protect the compressor.\"),\n",
    "    (\"t11\", \"Troubleshooting / Wrong temperature\", \"If the temperature reading seems off, avoid direct sun and drafts. A calibration offset of up to 3 degrees can be set in the app.\"),\n",
    "    (\"t12\", \"Warranty / Coverage\", \"Thermio One has a 3 year limited warranty from the date of purchase. Damage from water or from incorrect wiring is not covered.\"),\n",
    "    (\"t13\", \"Warranty / Returns\", \"Returns are accepted within 45 days of delivery if the device is undamaged. The refund is issued to the original payment method within 10 business days.\"),\n",
    "    (\"t14\", \"Warranty / Replacement\", \"A faulty unit under warranty is replaced with a refurbished or new device and shipped within 3 business days of approval.\"),\n",
    "    (\"t15\", \"Account / Password reset\", \"To reset the password, use the Forgot password link on the sign-in page. The reset email arrives within 5 minutes and the link is valid for 1 hour.\"),\n",
    "    (\"t16\", \"Account / Sharing\", \"Up to 8 people can share one home in the app. The owner can remove members at any time under Home settings.\"),\n",
    "    (\"t17\", \"Account / Data\", \"Temperature history is stored for 12 months. Deleting an account removes all history within 30 days.\"),\n",
    "    (\"t18\", \"Energy / Savings report\", \"The monthly energy report compares runtime with the previous month and with similar homes. Reports need at least 14 days of data.\"),\n",
    "    (\"t19\", \"Energy / Smart utility\", \"Utility programs can request a temporary setback of up to 2 degrees for at most 4 hours. You can opt out of any event from the app.\"),\n",
    "    (\"t20\", \"Hardware / Battery\", \"Thermio One uses a built-in rechargeable battery that charges from the C-wire. The battery is not user-replaceable.\"),\n",
    "    (\"t21\", \"Hardware / Sensors\", \"A remote room sensor (sold separately) measures temperature and occupancy up to 10 m from the thermostat. Up to 6 sensors can be paired.\"),\n",
    "    (\"t22\", \"Hardware / Dimensions\", \"The thermostat measures 86 mm in diameter and 24 mm in depth and weighs 150 g.\"),\n",
    "    (\"t23\", \"Support / Contact\", \"Support is available by chat from 08:00 to 20:00 on weekdays. Response time for email is within 24 hours.\"),\n",
    "    (\"t24\", \"Support / Firmware\", \"Firmware updates install automatically between 02:00 and 04:00 local time. An update takes about 10 minutes and the schedule keeps running.\"),\n",
    "    (\"t25\", \"Support / Privacy\", \"Thermio does not record audio or video. Location is used only for geofencing and can be switched off in the app.\"),\n",
    "]\n",
    "\n",
    "# sales table: month x product x region (deterministic synthetic numbers)\n",
    "_r = random.Random(21)\n",
    "PRICE = {\"Thermio One\": 129, \"Thermio Line\": 99, \"Thermio Sensor\": 39}\n",
    "sales = []\n",
    "for m in range(1, 7):\n",
    "    for p in PRICE:\n",
    "        for reg in (\"North\", \"South\"):\n",
    "            u = _r.randint(40, 400)\n",
    "            sales.append(dict(month=f\"2026-{m:02d}\", product=p, region=reg, units=u, revenue_eur=u * PRICE[p]))\n",
    "con = duckdb.connect()\n",
    "con.execute(\"CREATE TABLE sales (month VARCHAR, product VARCHAR, region VARCHAR, units INTEGER, revenue_eur INTEGER)\")\n",
    "con.executemany(\"INSERT INTO sales VALUES (?, ?, ?, ?, ?)\", [tuple(r.values()) for r in sales])\n",
    "print(len(KB), \"pasajes;\", len(sales), \"filas de ventas;\", sum(r[\"units\"] for r in sales), \"unidades en total\")\n",
    "print(sales[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2e92e20e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:53.923081Z",
     "iopub.status.busy": "2026-11-17T21:23:53.922290Z",
     "iopub.status.idle": "2026-11-17T21:23:53.947373Z",
     "shell.execute_reply": "2026-11-17T21:23:53.945147Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "referencias de las preguntas SQL: {'s01': '2465', 's02': 'South', 's03': '154176', 's04': '1251'}\n",
      "24 preguntas de la batería, 12 preguntas de desarrollo\n"
     ]
    }
   ],
   "source": [
    "# references for the SQL questions come from an INDEPENDENT plain-Python calculation, never from the system under test\n",
    "def tot(key, **where):\n",
    "    return sum(r[key] for r in sales if all(r[k] == v for k, v in where.items()))\n",
    "exp = dict(\n",
    "    s01=str(tot(\"units\", product=\"Thermio One\")),\n",
    "    s02=max((\"North\", \"South\"), key=lambda reg: tot(\"units\", product=\"Thermio Line\", month=\"2026-03\", region=reg)),\n",
    "    s03=str(tot(\"revenue_eur\", month=\"2026-06\")),\n",
    "    s04=str(tot(\"units\", product=\"Thermio Sensor\", region=\"North\")),\n",
    ")\n",
    "print(\"referencias de las preguntas SQL:\", exp)\n",
    "\n",
    "# (id, question, gold passage or None, the fact the answer must contain)\n",
    "BATTERY = [\n",
    "    (\"c01\", \"Can I connect the thermostat to a 5 GHz network?\", \"t01\", \"2.4 GHz\"),\n",
    "    (\"c02\", \"My heating system has no common wire, what now?\", \"t02\", \"power adapter\"),\n",
    "    (\"c03\", \"Does it work with electric baseboard heaters?\", \"t03\", \"Thermio Line\"),\n",
    "    (\"c04\", \"How many schedule changes can I set per day?\", \"t04\", \"6 changes\"),\n",
    "    (\"c05\", \"When does the house switch to away mode?\", \"t05\", \"500 m\"),\n",
    "    (\"c06\", \"What is the lowest heating limit in eco mode?\", \"t06\", \"16 degrees\"),\n",
    "    (\"c07\", \"The thermostat says Offline, what should I do?\", \"t07\", \"restart the router\"),\n",
    "    (\"c08\", \"How long does the display stay on when the power is cut?\", \"t08\", \"2 hours\"),\n",
    "    (\"c09\", \"What does error E21 mean?\", \"t09\", \"sensor\"),\n",
    "    (\"c10\", \"Why does the system wait before starting the heating again?\", \"t10\", \"5 minutes\"),\n",
    "    (\"c11\", \"How long is the warranty?\", \"t12\", \"3 year\"),\n",
    "    (\"c12\", \"Within how many days can I return it?\", \"t13\", \"45 days\"),\n",
    "    (\"c13\", \"How do I get a new password?\", \"t15\", \"Forgot password\"),\n",
    "    (\"c14\", \"How many people can share the home?\", \"t16\", \"8 people\"),\n",
    "    (\"c15\", \"How long is the temperature history kept?\", \"t17\", \"12 months\"),\n",
    "    (\"c16\", \"When do firmware updates install?\", \"t24\", \"02:00\"),\n",
    "    (\"s01\", \"How many Thermio One units were sold in total?\", \"sales\", exp[\"s01\"]),\n",
    "    (\"s02\", \"Which region sold the most Thermio Line units in March?\", \"sales\", exp[\"s02\"]),\n",
    "    (\"s03\", \"What was the total revenue in June?\", \"sales\", exp[\"s03\"]),\n",
    "    (\"s04\", \"How many Thermio Sensor units were sold in the North region?\", \"sales\", exp[\"s04\"]),\n",
    "    (\"u01\", \"Does the thermostat work with voice assistants?\", None, None),\n",
    "    (\"u02\", \"What is the price of a Thermio One?\", None, None),\n",
    "    (\"u03\", \"Who founded the company?\", None, None),\n",
    "    (\"u04\", \"How do I fix error E99?\", None, None),\n",
    "]\n",
    "# dev set: used ONLY to choose the retrieval weight and the refusal threshold\n",
    "DEV = [(\"d1\", \"Which wifi bands are supported?\", \"t01\"), (\"d2\", \"How cold can eco mode keep the house in winter?\", \"t06\"),\n",
    "       (\"d3\", \"What happens if my account is deleted?\", \"t17\"), (\"d4\", \"Can I get my money back?\", \"t13\"),\n",
    "       (\"d5\", \"How do sensors in other rooms work?\", \"t21\"), (\"d6\", \"Is my voice recorded?\", \"t25\"),\n",
    "       (\"d7\", \"What if the screen stays dark?\", \"t08\"), (\"d8\", \"How big is the device?\", \"t22\"),\n",
    "       (\"d9\", \"What does E45 mean?\", \"t10\"), (\"d10\", \"Do I need a C-wire?\", \"t02\"),\n",
    "       (\"d11\", \"Is error E21 serious?\", \"t09\"), (\"d12\", \"firmware update time\", \"t24\")]\n",
    "OFF_TOPIC = [\"What is the capital of France?\", \"How do I bake sourdough bread?\", \"Who won the football match yesterday?\", \"What is the speed of light?\"]\n",
    "print(len(BATTERY), \"preguntas de la batería,\", len(DEV), \"preguntas de desarrollo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce18fe9",
   "metadata": {},
   "source": [
    "## 3. Recuperación: trocear por estructura y elegir el peso de la fusión midiendo\n",
    "\n",
    "La base de conocimiento ya tiene estructura: un pasaje por tema, cada uno con un título de sección, así que el\n",
    "fragmento es el pasaje y el título forma parte del texto buscable (módulo 01). La recuperación puede combinar BM25\n",
    "y embeddings con RRF (módulo 02). En el módulo 02 los pesos iguales no eran los mejores; aquí **medimos** el peso\n",
    "de BM25 sobre el conjunto de desarrollo en lugar de suponerlo, y aceptamos lo que diga."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3c78506f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:53.951921Z",
     "iopub.status.busy": "2026-11-17T21:23:53.951467Z",
     "iopub.status.idle": "2026-11-17T21:23:56.984526Z",
     "shell.execute_reply": "2026-11-17T21:23:56.982914Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights:   0%|          | 0/103 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Loading weights: 100%|██████████| 103/103 [00:00<00:00, 2822.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peso de BM25 (peso semántico = 1) frente al conjunto de DESARROLLO:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  peso 0     MRR 0.958   hit@1 0.92\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  peso 0.25  MRR 0.847   hit@1 0.75\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  peso 0.5   MRR 0.833   hit@1 0.75\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  peso 1.0   MRR 0.819   hit@1 0.75\n",
      "peso de BM25 elegido: 0\n"
     ]
    }
   ],
   "source": [
    "STOP = set(\"a an and are as at be by can do does for from how i if in is it of on or that the this to what when which who with you my me\".split())\n",
    "tokens = lambda text: [w for w in re.findall(r\"[a-z0-9]+\", text.lower()) if w not in STOP]\n",
    "embedder = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\", device=\"cpu\")\n",
    "\n",
    "class Index:\n",
    "    # chunks + BM25 + embeddings over a list of (id, section, text) passages\n",
    "    def __init__(self, passages):\n",
    "        self.ids = [p[0] for p in passages]\n",
    "        self.texts = [f\"{p[1]}. {p[2]}\" for p in passages]\n",
    "        self.tf = [defaultdict(int) for _ in self.texts]\n",
    "        for i, t in enumerate(self.texts):\n",
    "            for w in tokens(t):\n",
    "                self.tf[i][w] += 1\n",
    "        self.len = np.array([sum(d.values()) for d in self.tf], float)\n",
    "        df = defaultdict(int)\n",
    "        for d in self.tf:\n",
    "            for w in d:\n",
    "                df[w] += 1\n",
    "        n = len(self.texts)\n",
    "        self.idf = {w: math.log(1 + (n - c + 0.5) / (c + 0.5)) for w, c in df.items()}\n",
    "        self.emb = embedder.encode(self.texts, normalize_embeddings=True)\n",
    "    def bm25(self, q):\n",
    "        s = np.zeros(len(self.texts))\n",
    "        for w in set(tokens(q)):\n",
    "            if w in self.idf:\n",
    "                for i, d in enumerate(self.tf):\n",
    "                    f = d.get(w, 0)\n",
    "                    if f:\n",
    "                        s[i] += self.idf[w] * f * 2.5 / (f + 1.5 * (0.25 + 0.75 * self.len[i] / self.len.mean()))\n",
    "        return s\n",
    "    def cosine(self, q):\n",
    "        return self.emb @ embedder.encode(q, normalize_embeddings=True)\n",
    "    def fused(self, q, w_bm25):\n",
    "        tot = np.zeros(len(self.texts))\n",
    "        for sc, w in ((self.bm25(q), w_bm25), (self.cosine(q), 1.0)):\n",
    "            for rank, i in enumerate(np.argsort(-sc), 1):\n",
    "                tot[i] += w / (60 + rank)\n",
    "        return tot\n",
    "\n",
    "index = Index(KB)\n",
    "\n",
    "def mrr(idx, w):\n",
    "    r = [list(np.argsort(-idx.fused(q, w))).index(idx.ids.index(g)) + 1 for _, q, g in DEV]\n",
    "    return float(np.mean([1 / x for x in r])), float(np.mean([x == 1 for x in r]))\n",
    "\n",
    "print(\"peso de BM25 (peso semántico = 1) frente al conjunto de DESARROLLO:\")\n",
    "dev_scores = {}\n",
    "for w in (0, 0.25, 0.5, 1.0):\n",
    "    dev_scores[w] = mrr(index, w)\n",
    "    print(f\"  peso {w:<5} MRR {dev_scores[w][0]:.3f}   hit@1 {dev_scores[w][1]:.2f}\")\n",
    "W_BM25 = max(dev_scores, key=lambda w: (dev_scores[w][0], -w))          # best MRR; ties go to the smaller (simpler) weight\n",
    "print(\"peso de BM25 elegido:\", W_BM25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b066c1",
   "metadata": {},
   "source": [
    "En estas preguntas de desarrollo, añadir BM25 a la fusión **no ayudó**: el mejor peso es el que muestra la tabla, y\n",
    "con el peso en cero el \"híbrido\" es solo semántico. Es un resultado legítimo de medir (en el módulo 02 las\n",
    "preguntas con identificadores favorecían un peso pequeño de BM25; esta base de conocimiento es pequeña, sus\n",
    "pasajes son distintivos y la mayoría de las preguntas de desarrollo son paráfrasis). Sería un error mantener un\n",
    "peso solo porque \"lo híbrido es la mejor práctica\". Aun así, BM25 no se desperdicia: se queda en el sistema como el\n",
    "**segundo ranker independiente** que usa el supervisor en la sección 7. Repite la selección del peso cada vez que\n",
    "cambie el corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a4b3be",
   "metadata": {},
   "source": [
    "## 4. Enrutador y SQL seguro\n",
    "\n",
    "Algunas preguntas las responde un pasaje y otras un número. Un **enrutador** decide, y debe ser conservador: una\n",
    "pregunta va a SQL solo si pide claramente una cantidad de la tabla de ventas. El camino SQL sigue el módulo 06: el\n",
    "\"modelo\" (aquí un sustituto basado en reglas) escribe **un** `SELECT`, un **guardián** lo valida y DuckDB lo\n",
    "ejecuta sobre una vista registrada."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "359ab7ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:56.988943Z",
     "iopub.status.busy": "2026-11-17T21:23:56.988554Z",
     "iopub.status.idle": "2026-11-17T21:23:57.004654Z",
     "shell.execute_reply": "2026-11-17T21:23:57.003012Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DROP TABLE sales                              -> (False, 'solo se permite un único SELECT')\n",
      "SELECT * FROM sales; DELETE FROM sales        -> (False, 'solo se permite un único SELECT')\n",
      "SELECT * FROM read_csv('/etc/passwd')         -> (False, 'palabra clave prohibida')\n",
      "SELECT * FROM secrets                         -> (False, 'tabla desconocida')\n",
      "SELECT region, SUM(units) AS total FROM sales WHERE product = 'Thermio Line' AND month = '2026-03' GROUP BY region ORDER BY total DESC LIMIT 1\n"
     ]
    }
   ],
   "source": [
    "MONTHS = {m: f\"2026-{i:02d}\" for i, m in enumerate([\"january\", \"february\", \"march\", \"april\", \"may\", \"june\"], 1)}\n",
    "\n",
    "def route(question):\n",
    "    q = question.lower()\n",
    "    return \"sql\" if re.search(r\"\\b(sold|sales|revenue)\\b\", q) else \"passage\"\n",
    "\n",
    "def text_to_sql(question):\n",
    "    # stand-in for the model that writes the SELECT: a rule-based translation of the few shapes we support\n",
    "    q = question.lower()\n",
    "    where = []\n",
    "    for p in PRICE:\n",
    "        if p.lower() in q:\n",
    "            where.append(f\"product = '{p}'\")\n",
    "    for reg in (\"north\", \"south\"):\n",
    "        if reg in q:\n",
    "            where.append(f\"region = '{reg.title()}'\")\n",
    "    for m, key in MONTHS.items():\n",
    "        if m in q:\n",
    "            where.append(f\"month = '{key}'\")\n",
    "    metric = \"revenue_eur\" if \"revenue\" in q else \"units\"\n",
    "    clause = (\" WHERE \" + \" AND \".join(where)) if where else \"\"\n",
    "    if re.search(r\"which (region|product)\", q):\n",
    "        col = \"region\" if \"region\" in q else \"product\"\n",
    "        return f\"SELECT {col}, SUM({metric}) AS total FROM sales{clause} GROUP BY {col} ORDER BY total DESC LIMIT 1\"\n",
    "    return f\"SELECT SUM({metric}) AS total FROM sales{clause}\"\n",
    "\n",
    "FORBIDDEN = re.compile(r\"\\b(insert|update|delete|drop|alter|create|attach|copy|pragma|install|load|read_\\w+)\\b\", re.I)\n",
    "\n",
    "def guard(sql):\n",
    "    body = sql.strip().rstrip(\";\")\n",
    "    if \";\" in body or not re.match(r\"(?is)^select\\b\", body):\n",
    "        return False, \"solo se permite un único SELECT\"\n",
    "    if FORBIDDEN.search(body):\n",
    "        return False, \"palabra clave prohibida\"\n",
    "    if set(re.findall(r\"(?i)\\b(?:from|join)\\s+(\\w+)\", body)) - {\"sales\"}:\n",
    "        return False, \"tabla desconocida\"\n",
    "    return True, body\n",
    "\n",
    "for bad in [\"DROP TABLE sales\", \"SELECT * FROM sales; DELETE FROM sales\", \"SELECT * FROM read_csv('/etc/passwd')\", \"SELECT * FROM secrets\"]:\n",
    "    print(f\"{bad[:44]:46s}-> {guard(bad)}\")\n",
    "print(text_to_sql(\"Which region sold the most Thermio Line units in March?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16979d23",
   "metadata": {},
   "source": [
    "## 5. Una respuesta tipada que no puede existir sin una cita\n",
    "\n",
    "Cada respuesta sale del sistema como un objeto **Pydantic**. El propio tipo rechaza una respuesta que no sea un\n",
    "rechazo y no tenga cita, de modo que un fallo del generador no puede producir en silencio una afirmación sin\n",
    "respaldo. El generador es el sustituto determinista de los módulos anteriores: devuelve la frase del pasaje\n",
    "principal más cercana a la pregunta. La regla de rechazo se calibra con el conjunto de desarrollo y con preguntas\n",
    "fuera de tema, nunca con la batería."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f7824afd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:57.008230Z",
     "iopub.status.busy": "2026-11-17T21:23:57.007917Z",
     "iopub.status.idle": "2026-11-17T21:23:57.536922Z",
     "shell.execute_reply": "2026-11-17T21:23:57.535067Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "coseno máximo: preguntas de desarrollo 0.33 a 0.79; fuera de tema 0.07 a 0.17  ->  rechazar por debajo de 0.25\n",
      "Can I connect the thermostat to a 5 GHz network?\n",
      "   [passage] If the thermostat shows Offline, restart the router, then hold the ring button for 10 seconds to reconnect. [1]   cites ['t07']\n",
      "Which region sold the most Thermio Line units in March?\n",
      "   [sql] South [1]   cites ['sales']\n",
      "What is the capital of Peru?\n",
      "   [refuse] No lo encuentro en la documentación de Thermio.   cites []\n"
     ]
    }
   ],
   "source": [
    "class Citation(BaseModel):\n",
    "    n: int\n",
    "    source: str                      # a passage id, or \"sales\" for a database result\n",
    "\n",
    "class Answer(BaseModel):\n",
    "    question: str\n",
    "    route: Literal[\"passage\", \"sql\", \"refuse\"]\n",
    "    text: str\n",
    "    citations: list[Citation] = []\n",
    "    @model_validator(mode=\"after\")\n",
    "    def needs_citation(self):\n",
    "        if self.route != \"refuse\" and not self.citations:\n",
    "            raise ValueError(\"an answer that is not a refusal must cite a source\")\n",
    "        return self\n",
    "\n",
    "# ----- refusal threshold, from the dev set (answerable) and off-topic questions (not answerable) -----\n",
    "dev_top = [index.cosine(q).max() for _, q, _ in DEV]\n",
    "off_top = [index.cosine(q).max() for q in OFF_TOPIC]\n",
    "T_COS = (max(off_top) + min(dev_top)) / 2 if max(off_top) < min(dev_top) else max(off_top) + 0.01\n",
    "print(f\"coseno máximo: preguntas de desarrollo {min(dev_top):.2f} a {max(dev_top):.2f}; fuera de tema {min(off_top):.2f} a {max(off_top):.2f}  ->  rechazar por debajo de {T_COS:.2f}\")\n",
    "\n",
    "REFUSAL = \"No lo encuentro en la documentación de Thermio.\"\n",
    "\n",
    "def best_sentence(question, text):\n",
    "    sents = [s for s in re.split(r\"(?<=[.!?])\\s+\", text) if s]\n",
    "    sims = embedder.encode(sents, normalize_embeddings=True) @ embedder.encode(question, normalize_embeddings=True)\n",
    "    return sents[int(np.argmax(sims))]\n",
    "\n",
    "def answer_question(question, idx, w=None):\n",
    "    # returns (Answer, details); details feed the trace and the supervisor\n",
    "    w = W_BM25 if w is None else w\n",
    "    if route(question) == \"sql\":\n",
    "        ok, sql = guard(text_to_sql(question))\n",
    "        if ok:\n",
    "            row = con.execute(sql).fetchall()[0]\n",
    "            value = row[0] if len(row) == 1 else row[0]\n",
    "            return Answer(question=question, route=\"sql\", text=f\"{value} [1]\", citations=[Citation(n=1, source=\"sales\")]), dict(sql=sql, pieces=[], top_cos=1.0)\n",
    "        return Answer(question=question, route=\"refuse\", text=REFUSAL), dict(sql=sql, pieces=[], top_cos=0.0)\n",
    "    cos = idx.cosine(question)\n",
    "    order = list(np.argsort(-idx.fused(question, w)))[:3]\n",
    "    if cos.max() < T_COS:\n",
    "        return Answer(question=question, route=\"refuse\", text=REFUSAL), dict(pieces=[idx.ids[i] for i in order], top_cos=float(cos.max()))\n",
    "    top = order[0]\n",
    "    sentence = best_sentence(question, idx.texts[top].split(\". \", 1)[1])\n",
    "    try:\n",
    "        ans = Answer(question=question, route=\"passage\", text=f\"{sentence} [1]\", citations=[Citation(n=1, source=idx.ids[top])])\n",
    "    except ValidationError:\n",
    "        ans = Answer(question=question, route=\"refuse\", text=REFUSAL)\n",
    "    return ans, dict(pieces=[idx.ids[i] for i in order], top_cos=float(cos.max()))\n",
    "\n",
    "for q in [\"Can I connect the thermostat to a 5 GHz network?\", \"Which region sold the most Thermio Line units in March?\", \"What is the capital of Peru?\"]:\n",
    "    a, _ = answer_question(q, index)\n",
    "    print(f\"{q}\\n   [{a.route}] {a.text}   cites {[c.source for c in a.citations]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32ea3cc",
   "metadata": {},
   "source": [
    "## 6. Ejecutar la batería, con una traza por cada petición\n",
    "\n",
    "Ahora el sistema responde las 24 preguntas de la batería. Cada petición escribe una **traza** (módulo 09): la ruta,\n",
    "los pasajes que vio, el recuento de tokens, el coste (con precios supuestos) y una latencia simulada. La\n",
    "calificación frente a las referencias se calcula aparte y es lo que intentamos mejorar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8a7960dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:57.540345Z",
     "iopub.status.busy": "2026-11-17T21:23:57.540018Z",
     "iopub.status.idle": "2026-11-17T21:23:59.354360Z",
     "shell.execute_reply": "2026-11-17T21:23:59.352791Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "correctas en total: 18 de 24 (75%)\n",
      "  preguntas de pasaje (16)  13 de 16\n",
      "  preguntas SQL (4)         4 de 4\n",
      "  sin respuesta posible (4) 1 de 4\n",
      "\n",
      "respuestas incorrectas:\n",
      "  c01 Can I connect the thermostat to a 5 GHz network?\n",
      "      -> [passage] If the thermostat shows Offline, restart the router, then hold the ring button for 10 seconds to reconnect. [1]   (referencia '2.4 GHz')\n",
      "  c03 Does it work with electric baseboard heaters?\n",
      "      -> [passage] Thermio One is not compatible with 120 V or 240 V line-voltage baseboard heaters. [1]   (referencia 'Thermio Line')\n",
      "  c11 How long is the warranty?\n",
      "      -> [passage] A faulty unit under warranty is replaced with a refurbished or new device and shipped within 3 business days of approval. [1]   (referencia '3 year')\n",
      "  u01 Does the thermostat work with voice assistants?\n",
      "      -> [passage] A remote room sensor (sold separately) measures temperature and occupancy up to 10 m from the thermostat. [1]   (referencia None)\n",
      "  u02 What is the price of a Thermio One?\n",
      "      -> [passage] Thermio One has a 3 year limited warranty from the date of purchase. [1]   (referencia None)\n",
      "  u04 How do I fix error E99?\n",
      "      -> [passage] Error E21 means the temperature sensor reading is out of range. [1]   (referencia None)\n"
     ]
    }
   ],
   "source": [
    "PRICE_IN, PRICE_OUT = 0.035, 0.14                  # ASSUMED USD per million tokens for the generation model\n",
    "TRACE_LOG = []\n",
    "\n",
    "def grade(ans, gold, fact):\n",
    "    # correct = contains the reference fact; for an unanswerable question (gold None) correct = a refusal\n",
    "    if gold is None:\n",
    "        return int(ans.route == \"refuse\")\n",
    "    return int(ans.route != \"refuse\" and fact.lower() in ans.text.lower())\n",
    "\n",
    "def run_battery(idx, label, battery=None):\n",
    "    r = random.Random(5)\n",
    "    rows = []\n",
    "    for qid, q, gold, fact in (battery or BATTERY):\n",
    "        ans, det = answer_question(q, idx)\n",
    "        n_in = round((len(q.split()) + 60 * (ans.route == \"passage\")) * 1.3) or 1\n",
    "        n_out = round(len(ans.text.split()) * 1.3)\n",
    "        cost = (n_in * PRICE_IN + n_out * PRICE_OUT) / 1e6\n",
    "        latency = (6 if ans.route != \"sql\" else 12) + (120 + 8 * n_out + r.random() * 30 if ans.route == \"passage\" else 0)\n",
    "        TRACE_LOG.append(dict(run=label, id=qid, route=ans.route, tokens_in=n_in, tokens_out=n_out, cost=cost, latency=latency))\n",
    "        rows.append(dict(id=qid, question=q, gold=gold, fact=fact, answer=ans, details=det, correct=grade(ans, gold, fact)))\n",
    "    return rows\n",
    "\n",
    "before = run_battery(index, \"before\")\n",
    "c = np.array([r[\"correct\"] for r in before])\n",
    "groups = {\"preguntas de pasaje (16)\": [r for r in before if r[\"id\"].startswith(\"c\")], \"preguntas SQL (4)\": [r for r in before if r[\"id\"].startswith(\"s\")],\n",
    "          \"sin respuesta posible (4)\": [r for r in before if r[\"id\"].startswith(\"u\")]}\n",
    "print(f\"correctas en total: {c.sum()} de {len(c)} ({c.mean():.0%})\")\n",
    "for name, g in groups.items():\n",
    "    print(f\"  {name:26s}{sum(x['correct'] for x in g)} de {len(g)}\")\n",
    "print(\"\\nrespuestas incorrectas:\")\n",
    "for r in before:\n",
    "    if not r[\"correct\"]:\n",
    "        print(f\"  {r['id']} {r['question']}\\n      -> [{r['answer'].route}] {r['answer'].text}   (referencia {r['fact']!r})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81675b37",
   "metadata": {},
   "source": [
    "## 7. El supervisor\n",
    "\n",
    "En producción, el supervisor es un segundo modelo que lee la pregunta, los pasajes y la respuesta y devuelve un\n",
    "veredicto (módulo 08). Aquí es un **sustituto barato**, y tiene la única propiedad que también tiene el real:\n",
    "**no conoce la respuesta de referencia**. Usa una señal clásica de incertidumbre que no necesita llamar a ningún\n",
    "modelo: **¿coinciden dos rankers independientes?** La respuesta se construyó con el pasaje que más gustó al\n",
    "recuperador. A un ranker BM25 puro, que funciona de otra manera que el recuperador, se le pregunta qué pasaje\n",
    "citaría *él*. Si ambos discrepan, o BM25 no encuentra nada, la respuesta se marca con **baja confianza** y una\n",
    "persona (o un modelo más potente) debería mirarla.\n",
    "\n",
    "* un **rechazo** es *justificado* si nada del índice se parece a la pregunta, y una *respuesta omitida* si algo sí;\n",
    "* una **respuesta de la base de datos** está *respaldada* (el guardián y la consulta son la evidencia);\n",
    "* una **respuesta de pasaje** está *respaldada* si los rankers coinciden y es de *baja confianza* en caso contrario;\n",
    "* de cada marca nombra los dos pasajes candidatos, que es donde está el arreglo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5e07e930",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:23:59.357665Z",
     "iopub.status.busy": "2026-11-17T21:23:59.357347Z",
     "iopub.status.idle": "2026-11-17T21:24:00.975001Z",
     "shell.execute_reply": "2026-11-17T21:24:00.973242Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "respaldada: 19, baja confianza: 4, rechazo justificado: 1 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "en las 12 preguntas de desarrollo marca 5; 1 respuestas eran realmente incorrectas; 1 de las marcas eran errores reales, 0 respuestas incorrectas no se marcaron\n",
      "\n",
      "                                sin marcar  marcadas\n",
      "correctas (según la referencia)            18         0\n",
      "incorrectas (según la referencia)             2         4\n",
      "\n",
      "marcó 4 de 6 respuestas incorrectas (exhaustividad), 4 de 4 marcas eran errores reales (precisión)\n"
     ]
    }
   ],
   "source": [
    "def supervise(row, idx):\n",
    "    ans, q = row[\"answer\"], row[\"question\"]\n",
    "    near = idx.cosine(q)\n",
    "    j = int(np.argmax(near))\n",
    "    name = lambda k: f\"{idx.ids[k]} ({idx.texts[k].split('. ')[0]})\"\n",
    "    if ans.route == \"refuse\":\n",
    "        if near[j] >= T_COS + 0.1:                  # a passage clearly related exists: the refusal may be a miss\n",
    "            return dict(verdict=\"omitida\", missing=name(j))\n",
    "        return dict(verdict=\"rechazo justificado\", missing=\"\")\n",
    "    if ans.route == \"sql\":\n",
    "        return dict(verdict=\"respaldada\", missing=\"\")\n",
    "    b = idx.bm25(q)\n",
    "    cited = idx.ids.index(ans.citations[0].source)\n",
    "    if b.max() > 0 and int(np.argmax(b)) == cited:\n",
    "        return dict(verdict=\"respaldada\", missing=\"\")\n",
    "    return dict(verdict=\"baja confianza\", missing=f\"el recuperador eligió {name(cited)}; BM25 prefiere \" + (name(int(np.argmax(b))) if b.max() > 0 else \"nada (ninguna palabra coincide)\"))\n",
    "\n",
    "for r in before:\n",
    "    r[\"judge\"] = supervise(r, index)\n",
    "verdicts = Counter(r[\"judge\"][\"verdict\"] for r in before)\n",
    "print(\", \".join(f\"{k}: {v}\" for k, v in verdicts.most_common()), \"\\n\")\n",
    "\n",
    "# how does it behave on the DEV questions, whose answers we can check against the dev references?\n",
    "dev_rows = [dict(question=q, answer=answer_question(q, index)[0]) for _, q, _ in DEV]\n",
    "dev_flag = [supervise(r, index)[\"verdict\"] == \"baja confianza\" for r in dev_rows]\n",
    "dev_wrong = [(r[\"answer\"].citations[0].source if r[\"answer\"].citations else None) != g for r, (_, _, g) in zip(dev_rows, DEV)]\n",
    "caught = sum(f and w for f, w in zip(dev_flag, dev_wrong))\n",
    "print(f\"en las {len(DEV)} preguntas de desarrollo marca {sum(dev_flag)}; {sum(dev_wrong)} respuestas eran realmente incorrectas; {caught} de las marcas eran errores reales, {sum(dev_wrong) - caught} respuestas incorrectas no se marcaron\")\n",
    "\n",
    "# does the supervisor agree with the references on the battery? (it never saw them)\n",
    "flag_bad = np.array([r[\"judge\"][\"verdict\"] in (\"omitida\", \"baja confianza\") for r in before])\n",
    "truth_bad = np.array([not r[\"correct\"] for r in before])\n",
    "print(f\"\\n{'':28s}{'sin marcar':>14s}{'marcadas':>10s}\")\n",
    "for label, t in ((\"correctas (según la referencia)\", False), (\"incorrectas (según la referencia)\", True)):\n",
    "    print(f\"{label:28s}{int(((truth_bad == t) & ~flag_bad).sum()):>14d}{int(((truth_bad == t) & flag_bad).sum()):>10d}\")\n",
    "print(f\"\\nmarcó {int((flag_bad & truth_bad).sum())} de {truth_bad.sum()} respuestas incorrectas (exhaustividad), {int((flag_bad & truth_bad).sum())} de {flag_bad.sum()} marcas eran errores reales (precisión)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fdd0fb9",
   "metadata": {},
   "source": [
    "Lee la tabla cruzada con honestidad. En la batería, el triaje marcó las respuestas incorrectas en las que los dos\n",
    "rankers **discrepan**, y cada marca era un error real. Se le escaparon las incorrectas en las que ambos rankers\n",
    "coincidían en el pasaje correcto y nuestro generador aun así escogió de él una frase incompleta o equivocada: el\n",
    "juez mira *qué pasaje*, no *si la frase responde a la pregunta*. En las preguntas de desarrollo, parafraseadas y\n",
    "cortas, **marca de más**, porque BM25 es débil cuando los usuarios no usan las palabras del documento (mira la\n",
    "línea de desarrollo de arriba). Esas son las dos caras de un juez barato: se le escapan algunos errores reales y\n",
    "da algunas falsas alarmas. Un juez LLM real, que lea los pasajes y la respuesta, cierra buena parte de esa\n",
    "brecha, a un coste por respuesta juzgada que el presupuesto del módulo 10 permite calcular. Usa esta tabla para\n",
    "decidir *cuánto fiarte del resumen nocturno*, no como prueba de que el sistema está bien.\n",
    "\n",
    "## 8. El resumen diario\n",
    "\n",
    "La parte del supervisor que escala es el **resumen**: no 24 veredictos que leer, sino una página que dice qué ha\n",
    "cambiado y dónde mirar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "89df5402",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:24:00.978602Z",
     "iopub.status.busy": "2026-11-17T21:24:00.978271Z",
     "iopub.status.idle": "2026-11-17T21:24:00.988800Z",
     "shell.execute_reply": "2026-11-17T21:24:00.987193Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peticiones 24   coste $0.000115 en total, $0.0000048 por respuesta   latencia p50 279 ms, p95 379 ms\n",
      "rutas: passage 19, sql 4, refuse 1\n",
      "veredictos: respaldada 19, baja confianza 4, rechazo justificado 1\n",
      "\n",
      "requieren atención (vista del supervisor, sin usar referencias):\n"
     ]
    }
   ],
   "source": [
    "tl = [t for t in TRACE_LOG if t[\"run\"] == \"before\"]\n",
    "costs, lats = np.array([t[\"cost\"] for t in tl]), np.array([t[\"latency\"] for t in tl])\n",
    "print(f\"peticiones {len(tl)}   coste ${costs.sum():.6f} en total, ${costs.mean():.7f} por respuesta   latencia p50 {np.median(lats):.0f} ms, p95 {np.percentile(lats, 95):.0f} ms\")\n",
    "print(\"rutas: \" + \", \".join(f\"{k} {v}\" for k, v in Counter(t[\"route\"] for t in tl).most_common()))\n",
    "print(\"veredictos: \" + \", \".join(f\"{k} {v}\" for k, v in verdicts.most_common()))\n",
    "print(\"\\nrequieren atención (vista del supervisor, sin usar referencias):\")\n",
    "for r in before:\n",
    "    if r[\"judge\"][\"verdict\"] in (\"omitida\", \"unsupported\"):\n",
    "        print(f\"  {r['id']} [{r['judge']['verdict']}] {r['question']}\\n        closest passage in the index: {r['judge']['missing']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5184fb51",
   "metadata": {},
   "source": [
    "## 9. Análisis de fallos de recuperación y un arreglo de la documentación\n",
    "\n",
    "Ahora, los fallos que las referencias dejaron al descubierto. Para cada pregunta de pasaje incorrecta nos hacemos\n",
    "la única pregunta que decide el remedio: **¿en qué puesto del ranking estaba el pasaje que contiene la respuesta, y\n",
    "qué palabras usó la pregunta que el pasaje no usa?** El puesto 1 con la frase equivocada es un problema de\n",
    "generación; un puesto fuera de los 3 primeros es un fallo de recuperación, y la brecha de vocabulario indica cómo\n",
    "cerrarla en la documentación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "19693e0f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:24:00.992359Z",
     "iopub.status.busy": "2026-11-17T21:24:00.991920Z",
     "iopub.status.idle": "2026-11-17T21:24:01.059768Z",
     "shell.execute_reply": "2026-11-17T21:24:01.057786Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c01 Can I connect the thermostat to a 5 GHz network?\n",
      "    pasaje de referencia t01 en el puesto 2 -> entre los 3 primeros pero sin usar; palabras de la pregunta que no están en el pasaje: ['connect', 'thermostat']\n",
      "c03 Does it work with electric baseboard heaters?\n",
      "    pasaje de referencia t03 en el puesto 1 -> entre los 3 primeros pero sin usar; palabras de la pregunta que no están en el pasaje: ['electric', 'work']\n",
      "c11 How long is the warranty?\n",
      "    pasaje de referencia t12 en el puesto 2 -> entre los 3 primeros pero sin usar; palabras de la pregunta que no están en el pasaje: ['long']\n",
      "\n",
      "u01 Does the thermostat work with voice assistants?\n",
      "    respondió desde t21: A remote room sensor (sold separately) measures temperature and occupancy up to 10 m from the thermostat. [1]   (nada en la documentación responde a esto)\n",
      "\n",
      "u02 What is the price of a Thermio One?\n",
      "    respondió desde t12: Thermio One has a 3 year limited warranty from the date of purchase. [1]   (nada en la documentación responde a esto)\n",
      "\n",
      "u04 How do I fix error E99?\n",
      "    respondió desde t09: Error E21 means the temperature sensor reading is out of range. [1]   (nada en la documentación responde a esto)\n"
     ]
    }
   ],
   "source": [
    "def analyse(rows, idx):\n",
    "    for r in rows:\n",
    "        if r[\"correct\"] or r[\"gold\"] in (None, \"sales\"):\n",
    "            continue\n",
    "        order = [idx.ids[i] for i in np.argsort(-idx.fused(r[\"question\"], W_BM25))]\n",
    "        rank = order.index(r[\"gold\"]) + 1\n",
    "        gap = sorted(set(tokens(r[\"question\"])) - set(tokens(idx.texts[idx.ids.index(r[\"gold\"])])))\n",
    "        print(f\"{r['id']} {r['question']}\\n    pasaje de referencia {r['gold']} en el puesto {rank} -> {'fallo de recuperación' if rank > 3 else 'entre los 3 primeros pero sin usar'}; \"\n",
    "              f\"palabras de la pregunta que no están en el pasaje: {gap}\")\n",
    "\n",
    "analyse(before, index)\n",
    "traps = [r for r in before if r[\"gold\"] is None and not r[\"correct\"]]\n",
    "for r in traps:\n",
    "    print(f\"\\n{r['id']} {r['question']}\\n    respondió desde {r['answer'].citations[0].source if r['answer'].citations else '-'}: {r['answer'].text}   (nada en la documentación responde a esto)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f91cb658",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:24:01.063283Z",
     "iopub.status.busy": "2026-11-17T21:24:01.062958Z",
     "iopub.status.idle": "2026-11-17T21:24:01.071392Z",
     "shell.execute_reply": "2026-11-17T21:24:01.069754Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t01: añadido ' It cannot connect to 5 GHz networks, only to 2.4 GHz networks.'\n",
      "t03: añadido ' Electric baseboard heaters are supported by Thermio Line, not by Thermio One.'\n",
      "t12: añadido ' The warranty lasts 3 years.'\n",
      "t26: nuevo pasaje 'Troubleshooting / Other error codes'\n",
      "t27: nuevo pasaje 'Features / Voice assistants'\n",
      "t28: nuevo pasaje 'Support / Pricing'\n"
     ]
    }
   ],
   "source": [
    "# THE FIX, written by reading the analysis above. Everything is a change to the DOCUMENTATION, the source of truth:\n",
    "#  1) answer the question in the words users use (the answer sentence must contain the fact, in one place)\n",
    "#  2) the questions the docs never covered get a passage that says so, or says where to go\n",
    "FIXES = {\n",
    "    \"t01\": \" It cannot connect to 5 GHz networks, only to 2.4 GHz networks.\",\n",
    "    \"t03\": \" Electric baseboard heaters are supported by Thermio Line, not by Thermio One.\",\n",
    "    \"t12\": \" The warranty lasts 3 years.\",\n",
    "}\n",
    "NEW = [\n",
    "    (\"t26\", \"Troubleshooting / Other error codes\", \"Only the error codes E21 and E45 are documented. For any other error code, contact support and include the code shown on the display.\"),\n",
    "    (\"t27\", \"Features / Voice assistants\", \"Thermio does not currently support voice assistants or third-party smart-home platforms.\"),\n",
    "    (\"t28\", \"Support / Pricing\", \"The price of a Thermio One is not listed in this documentation. See the store page for the current price.\"),\n",
    "]\n",
    "KB_FIXED = [(i, s, t + FIXES.get(i, \"\")) for i, s, t in KB] + NEW\n",
    "for i in FIXES:\n",
    "    print(f\"{i}: añadido {FIXES[i]!r}\")\n",
    "for n in NEW:\n",
    "    print(f\"{n[0]}: nuevo pasaje '{n[1]}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cffd481b",
   "metadata": {},
   "source": [
    "Un arreglo en la documentación es un arreglo **para cada usuario futuro y para cada modelo futuro**, y por eso el\n",
    "módulo 08 encamina las correcciones allí en lugar de parchear prompts. Fíjate en lo que *no* hicimos: no cambiamos\n",
    "el código, los umbrales ni el peso de recuperación. El único cambio es la base de conocimiento. Ahora reindexamos y\n",
    "volvemos a medir sobre la **misma batería**.\n",
    "\n",
    "Una salvedad honesta, dicha antes de los números para que no pueda esconderse en ellos: tres de las cuatro\n",
    "preguntas \"sin respuesta posible\" (`u01`, `u02`, `u04`) eran en realidad **huecos de documentación**, y el arreglo\n",
    "da un pasaje a cada una. Para esas tres, el comportamiento *correcto* ya no es un rechazo, así que sus referencias\n",
    "cambian (ahora esperan el dato del nuevo pasaje). Mover una referencia es legítimo cuando cambia el alcance de la\n",
    "documentación, y peligroso si es lo único que se movió, así que informamos de **dos** resultados: las 24 preguntas\n",
    "con las referencias actualizadas y las **20 preguntas cuya referencia no cambió**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a055dca6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T21:24:01.074983Z",
     "iopub.status.busy": "2026-11-17T21:24:01.074599Z",
     "iopub.status.idle": "2026-11-17T21:24:04.026408Z",
     "shell.execute_reply": "2026-11-17T21:24:04.024575Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             antes          IC 95% después          IC 95%   cambio          IC 95%\n",
      "20 preguntas, referencias sin cambios         0.86    [0.71, 1.00]    1.00    [1.00, 1.00]    +0.14    [0.00, 0.29]\n",
      "24 preguntas, referencias actualizadas        0.75    [0.58, 0.92]    0.96    [0.88, 1.00]    +0.21    [0.04, 0.38]\n",
      "\n",
      "ahora correctas: ['c01', 'c03', 'c11', 'u01', 'u04']   ahora incorrectas (regresiones): []\n",
      "  siguen mal: u02 What is the price of a Thermio One? -> The price of a Thermio One is not listed in this documentation. [1]\n"
     ]
    }
   ],
   "source": [
    "UPDATED = {\"u01\": (\"t27\", \"does not currently support\"), \"u02\": (\"t28\", \"store page\"), \"u04\": (\"t26\", \"contact support\")}\n",
    "BATTERY_AFTER = [(i, q, *UPDATED[i]) if i in UPDATED else (i, q, g, f) for i, q, g, f in BATTERY]\n",
    "index2 = Index(KB_FIXED)\n",
    "after = run_battery(index2, \"after\", BATTERY_AFTER)\n",
    "\n",
    "b_same = np.array([r[\"correct\"] for r in before])                                           # old answers, old references\n",
    "b_new = np.array([grade(r[\"answer\"], g, f) for r, (_, _, g, f) in zip(before, BATTERY_AFTER)])   # old answers, UPDATED references\n",
    "a_ok = np.array([r[\"correct\"] for r in after])\n",
    "unchanged = np.array([i not in UPDATED for i, *_ in BATTERY])\n",
    "\n",
    "rng = np.random.default_rng(0)\n",
    "def boot(b, a, mask):\n",
    "    b, a = b[mask], a[mask]\n",
    "    ix = rng.integers(0, len(a), (5000, len(a)))\n",
    "    ci = lambda x: f\"[{np.percentile(x, 2.5):.2f}, {np.percentile(x, 97.5):.2f}]\"\n",
    "    return b.mean(), ci(b[ix].mean(1)), a.mean(), ci(a[ix].mean(1)), a.mean() - b.mean(), ci(a[ix].mean(1) - b[ix].mean(1))\n",
    "\n",
    "print(f\"{'':42s}{'antes':>8s}{'IC 95%':>16s}{'después':>8s}{'IC 95%':>16s}{'cambio':>9s}{'IC 95%':>16s}\")\n",
    "for label, b, mask in ((\"20 preguntas, referencias sin cambios\", b_same, unchanged), (\"24 preguntas, referencias actualizadas\", b_new, np.ones(24, bool))):\n",
    "    r = boot(b, a_ok, mask)\n",
    "    print(f\"{label:42s}{r[0]:>8.2f}{r[1]:>16s}{r[2]:>8.2f}{r[3]:>16s}{r[4]:>+9.2f}{r[5]:>16s}\")\n",
    "fixed = [r[\"id\"] for r, bb, aa in zip(after, b_new, a_ok) if aa and not bb]\n",
    "broke = [r[\"id\"] for r, bb, aa in zip(after, b_new, a_ok) if bb and not aa]\n",
    "print(\"\\nahora correctas:\", fixed, \"  ahora incorrectas (regresiones):\", broke)\n",
    "for r in after:\n",
    "    if not r[\"correct\"]:\n",
    "        print(f\"  siguen mal: {r['id']} {r['question']} -> {r['answer'].text}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898257a2",
   "metadata": {},
   "source": [
    "Dos cosas que dicen los números y una que no pueden decir. Dicen **qué preguntas reparó el arreglo** (`c01`, `c03`,\n",
    "`c11`, `u01`, `u04`) y, igual de importante, que **no rompió nada**: una comprobación de regresiones es la mitad del\n",
    "valor de una batería fija. Dicen **cuán ancha es la incertidumbre**: en las 20 preguntas cuya referencia no\n",
    "cambió, el intervalo de la ganancia va de 0,00 a 0,29, así que incluso una mejora clara apenas se distingue de no\n",
    "haber cambiado nada, y una puntuación \"después\" de 1,00 da un intervalo degenerado, señal de que la batería ya es\n",
    "demasiado fácil para distinguir sistemas mejores y necesita preguntas nuevas y más difíciles.\n",
    "\n",
    "El único fallo que queda, `u02`, es un artefacto de un evaluador estricto: la respuesta dice que el precio no\n",
    "figura en la documentación, lo cual es correcto, pero la referencia esperaba las palabras \"store page\". Las\n",
    "referencias de texto son un instrumento romo (por eso el módulo 07 califica también con un juez), y el informe\n",
    "honesto lo mantiene como fallo en lugar de relajar el evaluador después de ver el resultado.\n",
    "\n",
    "**Los números no pueden decir que el sistema sea bueno**, solo que este cambio no lo empeoró en estas preguntas. El\n",
    "tráfico real es más amplio que cualquier batería, y para eso está el resumen nocturno del supervisor (sección 8):\n",
    "convierte la producción en la siguiente batería.\n",
    "\n",
    "## 10. Antes de poner un sistema RAG delante de los usuarios\n",
    "\n",
    "Una lista de comprobación, con cada punto señalando el módulo donde lo construiste:\n",
    "\n",
    "- [ ] **Corpus:** cada documento tiene un responsable y una fecha; los fragmentos siguen la estructura del documento (01)\n",
    "- [ ] **Recuperación:** el peso de la fusión elegido con un conjunto de desarrollo que no es tu conjunto de prueba, y reelegido cuando cambia el corpus (02)\n",
    "- [ ] **Interfaces:** las llamadas a modelos pasan por una única capa de enrutado; cada entrada y salida tiene un esquema tipado (04)\n",
    "- [ ] **Flujo:** los pasos son explícitos, con un enrutador y un camino de rechazo; ningún paso puede entrar en bucle infinito (05)\n",
    "- [ ] **Números:** las cantidades salen de una consulta de solo lectura y con guardián, nunca de la memoria de un modelo (06)\n",
    "- [ ] **Calidad:** una batería de preguntas con referencias tomadas de los datos, con trampas y preguntas sin respuesta posible, ejecutada en cada cambio (07)\n",
    "- [ ] **Supervisión:** cada respuesta se juzga, los fallos se agrupan por causa y los arreglos van a la documentación (08)\n",
    "- [ ] **Observabilidad:** una traza por petición con tokens, coste, latencia y puntuaciones; percentiles en un panel (09)\n",
    "- [ ] **Presupuesto:** coste por respuesta estimado a partir de tus propias trazas; límites de uso y un tope de concurrencia (10)\n",
    "- [ ] **Seguridad:** el texto recuperado se trata como no confiable; la salida se valida en código; los secretos fuera de las URL; los datos personales ocultos (10)\n",
    "- [ ] **Honestidad:** el sistema dice \"no lo encuentro\" en lugar de adivinar, y el equipo conoce su tasa de rechazo\n",
    "- [ ] **Responsabilidad:** alguien lee el resumen nocturno, y arreglar la documentación forma parte de su trabajo\n",
    "\n",
    "## Resumen: del módulo al componente del proyecto final\n",
    "\n",
    "| Módulo | Qué construiste | En este proyecto final |\n",
    "|---|---|---|\n",
    "| 01 | troceado, BM25, hit@k, MRR, rechazo | `Index.bm25`, fragmentos por sección, MRR del conjunto de desarrollo |\n",
    "| 02 | embeddings, RRF, pesos | `Index.cosine`, `Index.fused`, el peso `W_BM25` elegido con el conjunto de desarrollo (BM25 pasó a ser la segunda opinión del supervisor) |\n",
    "| 03 | LangChain y LlamaIndex | la misma recuperación escrita a mano: lo que abstraen los frameworks |\n",
    "| 04 | LiteLLM y Pydantic | la `Answer` tipada y su validador; un único lugar para cambiar el generador |\n",
    "| 05 | LangGraph | `route` y los pasos explícitos de `answer_question` |\n",
    "| 06 | text-to-SQL seguro | `text_to_sql`, `guard`, DuckDB |\n",
    "| 07 | evaluación | `BATTERY`, `DEV`, calificación frente a referencias independientes, IC por bootstrap |\n",
    "| 08 | el supervisor | `supervise`, el resumen diario, el arreglo de la documentación y la reindexación |\n",
    "| 09 | observabilidad | `TRACE_LOG` con tokens, coste y latencia |\n",
    "| 10 | producción y seguridad | umbral de rechazo, modelo de coste, validación y los hábitos de la lista de comprobación |\n",
    "\n",
    "## Ejercicios\n",
    "\n",
    "1. Sustituye `best_sentence` por tomar como respuesta el pasaje principal completo. ¿Qué preguntas cambian, y qué\n",
    "   dice eso sobre la frontera entre recuperación y generación?\n",
    "2. Pon `W_BM25 = 1.0` después de la sección 3 y vuelve a ejecutar desde la sección 6. ¿Sigue el supervisor marcando\n",
    "   los fallos? ¿Se debilita su segunda opinión cuando BM25 también guía la recuperación?\n",
    "3. Añade una quinta trampa sin respuesta posible que comparta palabras con un pasaje real (\"What is the warranty on\n",
    "   the Thermio Line?\"). ¿Qué capa debería detectarla: la recuperación, el generador, el supervisor o la documentación?\n",
    "4. Escribe un arreglo de documentación nuevo para un fallo que tu supervisor marcó pero que este notebook no\n",
    "   arregló, aplícalo e informa del antes y el después con el intervalo. ¿Cambió el intervalo tu conclusión?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
