RAG over reports, prospectuses and KIDs, and how to evaluate it.

From the retrieved chunk to the cited answer

Lesson 6.3.2 left the retriever measured: structure-aware chunks with a context header and BM25 retrieve 87.3% of the gold passages with the top three chunks. This lesson adds the other half, the model that reads those chunks and writes, and above all the evaluation of the whole. The documents are the same: annual reports, ETF prospectuses and KIDs, and research notes, all synthetic, with 263 questions of which 17 have no answer in the documents. The example corpus is in Spanish (documents, questions and the prompt the model receives), so the fragments quoted below carry an English gloss in parentheses.

flowchart TB
  Q[Question] --> B[BM25 top-3]
  B --> C[Numbered context]
  C --> L[LLM or extraction]
  L --> R[Answer and citations]
  R --> E1[Accuracy]
  R --> E2[Faithfulness]
  R --> E3[Citation]
  B --> E4[Coverage]
  R --> E5[Abstention]

A RAG system can fail at every link, and a single overall score (“70% of the answers are right”) does not say where. That is why five things are measured separately:

All five are deterministic: there is no judge model. In financial documents almost all answers are figures, and a figure can be normalized (“13.668,9”, “13668.9”) and looked up. Free-text claims need something more (an LLM judge or a metric like Ragas faithfulness, which breaks the answer into claims and checks each one against the context); this is discussed at the end.

The retriever, in compact form

Structure-aware chunking and BM25 are those of lesson 6.3.2, in a single cell: blocks separated by blank lines, whole tables with their footnotes attached, paragraphs of the same section merged up to 800 characters and a “document > section” header in the indexed text.

import re
import unicodedata

import numpy as np
import pandas as pd
from rank_bm25 import BM25Okapi

docs = pd.read_parquet("rag6_documentos.parquet")
pas = pd.read_parquet("rag6_pasajes.parquet")
preg = pd.read_csv("rag6_preguntas.csv")
TEXTO = dict(zip(docs.id_doc, docs.texto))
TITULO = dict(zip(docs.id_doc, docs.titulo))
P = pas.set_index("id_pasaje")
NOTA = "¹²³⁴⁵⁶⁷⁸⁹"


def seccion(doc, i):
    hs = re.findall(r"^##+ (.*)$", TEXTO[doc][:i + 1], re.M)
    return hs[-1] if hs else ""


def trocear(doc, n=800):
    t = TEXTO[doc]
    bl = [(m.start(), m.end())
          for m in re.finditer(r"[^\n](?:.|\n(?!\n))*", t)]
    notas = {t[a]: (a, b) for a, b in bl if t[a] in NOTA}
    trozos = []
    for a, b in bl:
        s = t[a:b]
        if s.startswith("#") or s[0] in NOTA:
            continue
        marcas = [notas[c] for c in dict.fromkeys(s) if c in notas]
        fija = s.startswith("|") or bool(marcas)
        ult = trozos[-1] if trozos else None
        if (ult and not fija and not ult["fija"]
                and seccion(doc, a) == seccion(doc, ult["tramos"][0][0])
                and b - ult["tramos"][0][0] <= n):
            ult["tramos"][0] = (ult["tramos"][0][0], b)
            continue
        trozos.append(dict(doc=doc, tramos=[(a, b)] + marcas,
                           fija=fija))
    for c in trozos:
        cuerpo = "\n\n".join(t[i:f] for i, f in c["tramos"])
        sec = seccion(doc, c["tramos"][0][0])
        c["texto"] = f"{TITULO[doc]} > {sec}\n\n{cuerpo}"
    return trozos


VACIAS = set("""de la el en y a los las del que se por con un una
para es al lo su sus o como fue cual cuanto cuantos cuales
segun""".split())


def palabras(s):
    s = unicodedata.normalize("NFKD", s.lower())
    s = "".join(ch for ch in s if not unicodedata.combining(ch))
    return [w for w in re.findall(r"\w+", s) if w not in VACIAS]


def cubierta(trozos, pid):
    """Fraction of the passage's characters inside the chunks."""
    r = P.loc[pid]
    iv = sorted((max(i, r.inicio), min(f, r.fin)) for c in trozos
                if c["doc"] == r.id_doc for i, f in c["tramos"])
    total, hasta = 0, r.inicio
    for a, b in iv:
        a = max(a, hasta)
        if b > a:
            total, hasta = total + b - a, b
    return total / (r.fin - r.inicio)


TR = [c for d in TEXTO for c in trocear(d)]
bm = BM25Okapi([palabras(c["texto"]) for c in TR])


def buscar(pregunta, k=3):
    sc = bm.get_scores(palabras(pregunta))
    orden = np.argsort(-sc, kind="stable")[:k]
    return [TR[j] for j in orden], float(sc[orden[0]])


print(len(TR), "chunks;", len(preg), "questions,",
      int((~preg.respondible).sum()), "unanswerable")

376 chunks; 263 questions, 17 unanswerable

Accuracy, faithfulness and citation, in code

All the measures rest on normalizing figures. The documents use the Spanish number format (“13.668,9”, “2,2 %”), the model sometimes the English one (“40.3%”), and years (“2023”) are not answer figures but references from the question. The rule: with a comma, the period is a thousands separator; without a comma, a period followed by exactly three digits is one too; numbers in square brackets are citations and are removed first.

NUM = re.compile(r"\d[\d.,]*\d|\d")
CITA = re.compile(r"\[(\d+)\]")


def cifras(s):
    """Set of figures in a text, without the [n] citations."""
    out = set()
    for t in NUM.findall(CITA.sub(" ", s)):
        if "," in t:
            t = t.replace(".", "").replace(",", ".")
        elif re.fullmatch(r"\d{1,3}(\.\d{3})+", t):
            t = t.replace(".", "")
        out.add(round(float(t.rstrip(".")), 4))
    return out


def calla(resp):
    return "no consta" in resp.lower()


def exacta(resp, q):
    """Gold figures present and no foreign ones (except those of the
    question); with no figures in the gold, its words."""
    if not q.respondible:
        return calla(resp)
    if calla(resp):
        return False
    oro, dichas = cifras(q.respuesta_oro), cifras(resp)
    if oro:
        return oro <= dichas and dichas <= oro | cifras(q.pregunta)
    return set(palabras(q.respuesta_oro)) <= set(palabras(resp))


def contiene(resp, q):
    """Lenient version: the gold figures (or words) are inside."""
    oro = cifras(q.respuesta_oro)
    if oro:
        return oro <= cifras(resp)
    return set(palabras(q.respuesta_oro)) <= set(palabras(resp))


def fiel(resp, contexto, q):
    """Every figure in the answer is in the context or the question."""
    return cifras(resp) <= cifras(contexto) | cifras(q.pregunta)


def citas_correctas(citados, q):
    """Fraction of cited chunks that cover a gold passage."""
    if not citados or not q.respondible:
        return np.nan
    oro = q.pasajes_oro.split(";")
    return np.mean([any(cubierta([c], p) >= 0.5 for p in oro)
                    for c in citados])


print(sorted(cifras("Ventas de 13.668,9 millones, un 2,2 % más [2]")))
print(sorted(cifras("40.3% de 5.505 en 2023")))

[2.2, 13668.9]
[40.3, 2023.0, 5505.0]

The extractive baseline, with all 263 questions

Before bringing in an LLM it is worth knowing how far you get without one. The extractive baseline generates nothing: it returns the best retrieved chunk (and cites fragment 1), or, shorter, the line of that chunk that shares the most words with the question. It is what a search engine with highlighting would do. Since it does not generate, it cannot invent figures (its faithfulness is total by construction), and it can be evaluated on every question in a second.

def mejor_linea(trozo, pregunta):
    cuerpo = trozo["texto"].split("\n\n", 1)[1]
    lineas = [x for x in re.split(r"\n|(?<=\.) ", cuerpo) if x.strip()]
    qp = set(palabras(pregunta))
    return max(lineas, key=lambda x: len(qp & set(palabras(x))))


base = []
for q in preg.itertuples():
    top, s1 = buscar(q.pregunta)
    oro = q.pasajes_oro.split(";") if q.respondible else []
    cob = (np.mean([cubierta(top, p) >= 0.5 for p in oro])
           if oro else np.nan)
    trozo = top[0]["texto"].split("\n\n", 1)[1]
    linea = mejor_linea(top[0], q.pregunta)
    base.append(dict(
        id=q.id_pregunta, tipo=q.tipo_pregunta, resp=q.respondible,
        score=s1, cobertura=cob,
        trozo=contiene(trozo, q) if q.respondible else np.nan,
        linea=contiene(linea, q) if q.respondible else np.nan,
        linea_exacta=exacta(linea, q) if q.respondible else np.nan,
        cita=citas_correctas(top[:1], q)))
base = pd.DataFrame(base)
r = base[base.resp]
print(r[["cobertura", "trozo", "linea", "linea_exacta", "cita"]]
      .astype(float).mean().round(3).to_dict())
cols = ["cobertura", "trozo", "linea", "cita"]
r.groupby("tipo")[cols].mean().round(2)

{'cobertura': 0.873, 'trozo': 0.772, 'linea': 0.545, 'linea_exacta': 0.382, 'cita': 0.707}
              cobertura     trozo     linea  cita
tipo
calculo            0.00       0.0       0.0  0.00
comparacion        0.11       0.0       0.0  0.14
nota_pie           0.75       1.0       0.5  0.50
tabla              0.88  0.671875  0.203125  0.67
tabla_y_nota       1.00  0.923077  0.384615  0.54
texto              0.99  0.844262  0.754098  0.81
texto_y_nota       0.75       1.0       1.0  1.00

The retriever brings the gold passages in 87.3% of cases (the coverage with k = 3 from lesson 6.3.2). The best chunk (trozo) contains the gold figures or words in 77.2% of the 246 answerable questions; the best line (linea), only in 54.5%, and it is exactly the answer, with no extra figures, in 38.2%. The citation of fragment 1 (cita) is correct in 70.7%. By type, text and footnotes do well with the whole chunk. Tables suffer with the line (20.3%): the row that shares the most words with the question is not always the one with the figure, and a lone row loses the column header. Calculation and comparison stay at zero: the answer is not written in any chunk, it has to be computed or documents have to be cross-checked, and an extraction does neither.

What the baseline cannot do is abstain. Its only signal is the BM25 score of the best chunk: if it is low, maybe the question has no answer. The next figure shows why it is a poor signal.

from sklearn.metrics import roc_auc_score

print(base.groupby("resp").score.describe()[["mean", "50%", "min",
                                             "max"]].round(2))
print("AUC:", round(roc_auc_score(~base.resp, -base.score), 3))
u = base[~base.resp].score.median()        # abstains on half
calla_bien = (~base.resp & (base.score < u)).sum()
calla_mal = (base.resp & (base.score < u)).sum()
print(f"threshold {u:.2f}: abstains on {calla_bien} of 17 unanswerable"
      f" and on {calla_mal} of 246 answerable")

        mean    50%   min    max
resp
False  10.95   9.39  8.39  16.16
True   14.72  13.86  7.08  27.67
AUC: 0.742
threshold 9.39: abstains on 8 of 17 unanswerable and on 16 of 246 answerable

Unanswerable questions score lower on average (10.95 versus 14.72), but the distributions overlap: the AUC of the score as a detector is 0.742 (0.5 would be chance). With the threshold at the median of the unanswerable questions (9.39), the baseline abstains on 8 of the 17 and, in exchange, on 16 of the 246 answerable ones. It makes sense: the unanswerable questions in the set name the company, the fund and the year just like the others (“¿cuál es el rating crediticio de Busmar Hogar según su informe anual de 2024?”, what is Busmar Hogar's credit rating according to its 2024 annual report?), so BM25 finds very similar chunks. Knowing that the rating is not there requires reading the context, which is exactly what an LLM is asked to do.

The local LLM: generation with citations on a sample

The generator is Qwen2.5-0.5B-Instruct (Apache-2.0), the small model of the ia environment (lesson 6.2.1): 0.49 billion parameters, on CPU, with greedy (deterministic) decoding. It generates a few tokens per second, so it is not evaluated on all 263 questions: a stratified sample of 12 is taken (three text, three table, one footnote, one text and note, one table and note and three unanswerable), with a fixed seed. With 12 questions, any percentage has a confidence interval of about ±25 points: the sample serves to see how the model fails, not to estimate precisely how much.

The prompt follows what worked best with this model in earlier tests: short instructions in the system message, an example answer with a citation and the explicit instruction to answer “No consta” (not stated) if the context does not say. Like the corpus, the prompt is in Spanish and asks for the answer in Spanish. The context is the three numbered chunks [1], [2], [3].

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
CUOTA = {"texto": 3, "tabla": 3, "nota_pie": 1, "texto_y_nota": 1,
         "tabla_y_nota": 1, "no_respondible": 3}
muestra = pd.concat([preg[preg.tipo_pregunta == t]
                     .sample(n, random_state=0)
                     for t, n in CUOTA.items()])

SISTEMA = (
    "Eres un analista financiero. Responde en español con una frase "
    "corta, usando solo los fragmentos. Termina la frase con el número "
    "del fragmento que la apoya entre corchetes. Si los fragmentos no "
    "contienen la respuesta, responde exactamente: No consta.\n\n"
    "Ejemplo:\n[1] La plantilla media fue de 1.200 personas.\n"
    "[2] El consejo tiene 9 miembros.\n"
    "Pregunta: ¿Cuántos miembros tiene el consejo?\n"
    "Respuesta: El consejo tiene 9 miembros [2].")


def mensajes(pregunta, top):
    ctx = "\n\n".join(f"[{i + 1}] {c['texto']}"
                      for i, c in enumerate(top))
    return [{"role": "system", "content": SISTEMA},
            {"role": "user",
             "content": f"{ctx}\n\nPregunta: {pregunta}\nRespuesta:"}]


modelo = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(modelo)
llm = AutoModelForCausalLM.from_pretrained(modelo, dtype=torch.float32)
print(muestra.tipo_pregunta.value_counts().to_dict())

{'texto': 3, 'tabla': 3, 'no_respondible': 3, 'nota_pie': 1, 'texto_y_nota': 1, 'tabla_y_nota': 1}

filas = []
for q in muestra.itertuples():
    top, _ = buscar(q.pregunta)
    e = tok.apply_chat_template(mensajes(q.pregunta, top),
                                add_generation_prompt=True,
                                return_tensors="pt", return_dict=True)
    with torch.inference_mode():
        out = llm.generate(**e, max_new_tokens=40, do_sample=False,
                           pad_token_id=tok.eos_token_id)
    n = e["input_ids"].shape[1]
    resp = tok.decode(out[0, n:], skip_special_tokens=True).strip()
    citados = [top[int(i) - 1] for i in CITA.findall(resp)
               if 1 <= int(i) <= len(top)]
    contexto = "\n".join(c["texto"] for c in top)
    oro = q.pasajes_oro.split(";") if q.respondible else []
    filas.append(dict(
        id=q.id_pregunta, tipo=q.tipo_pregunta, resp=q.respondible,
        tokens=n, respuesta=resp.replace("\n", " ")[:60],
        exacta=exacta(resp, q), contiene=contiene(resp, q),
        fiel=fiel(resp, contexto, q) if not calla(resp) else np.nan,
        cita=citas_correctas(citados, q), cita_alguna=bool(citados),
        cobertura=(np.mean([cubierta(top, p) >= 0.5 for p in oro])
                   if oro else np.nan)))
llmr = pd.DataFrame(filas)
llmr[["id", "tipo", "respuesta", "exacta", "fiel", "cita"]]

       id            tipo                                          respuesta  exacta   fiel  cita
0   QA196           texto                                            [2] Yes   False   True   0.0
1   QA217           texto                                        704 valores    True   True   NaN
2   QA026           texto                                         [1]. 879,1    True   True   1.0
3   QA165           tabla        -869,8 -181,2 = -1,047 millones de USD [3].   False  False   0.0
4   QA105           tabla                                              20,5%    True   True   NaN
5   QA160           tabla                                      1.143,821 USD   False  False   NaN
6   QA015        nota_pie                                         No consta.   False    NaN   NaN
7   QA007    texto_y_nota  1,01 millones de retribución variable⁵, ligada...    True   True   0.0
8   QA151    tabla_y_nota  [1]. Zaptec Capital, S.A. — Informe anual 2023...   False   True   0.5
9   QA044  no_respondible                                         No consta.    True    NaN   NaN
10  QA154  no_respondible                                         No consta.    True    NaN   NaN
11  QA191  no_respondible                                       0,20 % anual   False   True   NaN

The retriever covers all the gold passages in the sample (100% coverage), so every failure belongs to the generator. Six answers are exact (exacta): 704 securities, 879.1 million, 20.5%, 1.01 million and two “No consta” on unanswerable questions. The failures are a catalog of what a small model does:

• Language and form. To “¿Reparte dividendos la clase A…?” (does class A pay dividends…?) it answers “[2] Yes”: in English, although the prompt asks for Spanish, without the “semestral” (semiannual) of the gold answer and citing a fragment that is not the gold one.
• Invented arithmetic. For dividends paid (181.2 million) it adds two figures from the cash flow statement and gives “−1,047”: the correct figure appears, but inside an operation nobody asked for. Faithfulness (fiel) detects it (the result is not in the context).
• Figures that do not exist. Zaptec's market capitalization at the end of 2024 (1,472.9 million) comes out as “1.143,821 USD”, a figure that is not in any fragment.
• Confusing “no” with “not stated”. To “¿Incluye la deuda financiera neta los pasivos por arrendamiento?” (does net financial debt include lease liabilities?; gold: no) it answers “No consta” (not stated). A negative answer is not an abstention, and the model mixes them up.
• Copying the header. On the question about scope 2 emissions it repeats the fragment's header and uses up the 40 tokens without getting to the figure.
• Answering what was not asked. Asked for the TER of a pound-hedged share class that does not exist, it gives “0,20 % anual” (0.20% a year): the TER of the class that does exist. It is a figure from the context, so numerical faithfulness accepts it. Faithfulness checks that what is said is in the context, not that it answers the question.

Numerical accuracy also has its blind spot: in the variable pay question an answer that gives the figure (1.01 million) counts as exact even though the gold also asks what it includes. Deterministic metrics are cheap and reproducible, but they measure what they measure.

The sample, compared: LLM versus extraction

The fair comparison is on the same 12 questions. The line baseline has to decide when to abstain: it is given the BM25 threshold rule from the previous section, the median of the scores of the unanswerable questions.

bm_ = base.set_index("id").loc[muestra.id_pregunta].reset_index()
calla_base = bm_.score < u
exacta_base = np.where(bm_.resp, ~calla_base & (bm_.linea_exacta == 1),
                       calla_base)


def ic(x):
    """Proportion with its 95% Wilson interval."""
    x = np.asarray(x, float)
    x = x[~np.isnan(x)]
    n, p, z = len(x), x.mean(), 1.96
    c = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    h = (z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
         / (1 + z**2 / n))
    return f"{p:.2f} [{c - h:.2f}, {c + h:.2f}] n={n}"


r_llm, r_base = llmr[llmr.resp], bm_[bm_.resp]
tabla = pd.DataFrame({
    "LLM 0.5B": [ic(llmr.exacta), ic(r_llm.contiene), "-",
                 ic(r_llm.fiel), ic(r_llm.cita), ic(r_llm.cita_alguna),
                 ic(llmr[~llmr.resp].exacta)],
    "extraction": [ic(exacta_base), ic(r_base.linea), ic(r_base.trozo),
                   "1.00 (no generation)", ic(r_base.cita),
                   "1.00 (always [1])", ic(calla_base[~bm_.resp])]},
    index=["exact (12)", "contains the gold", "  (whole chunk)",
           "faithful to context", "correct citations", "cites something",
           "abstains if not stated"])
print("retriever coverage on the sample:",
      round(r_llm.cobertura.mean(), 3))
tabla

retriever coverage on the sample: 1.0
                                      LLM 0.5B              extraction
exact (12)              0.50 [0.25, 0.75] n=12  0.25 [0.09, 0.53] n=12
contains the gold        0.67 [0.35, 0.88] n=9   0.22 [0.06, 0.55] n=9
  (whole chunk)                              -   0.78 [0.45, 0.94] n=9
faithful to context      0.75 [0.41, 0.93] n=8    1.00 (no generation)
correct citations        0.30 [0.07, 0.70] n=5   0.78 [0.45, 0.94] n=9
cites something          0.56 [0.27, 0.81] n=9       1.00 (always [1])
abstains if not stated   0.67 [0.21, 0.94] n=3   0.67 [0.21, 0.94] n=3

Read with its intervals, the table says little with certainty and quite a lot qualitatively:

• Accuracy. The LLM gives the exact answer in 6 of 12 questions and line extraction (with the threshold for abstaining) in 3 of 12. The 95% Wilson intervals ([0.25, 0.75] and [0.09, 0.53]) overlap: with 12 questions it cannot be claimed that the LLM is better.
• Content versus form. The whole chunk contains the gold in 7 of the 9 answerable questions, more than the LLM's answer (6 of 9). For an analyst who is going to read the evidence, the highlighted chunk is as useful as the generated sentence and invents nothing.
• Faithfulness and citations. The LLM invents figures in 2 of 8 answers with content, cites something in only 5 of 9 and, when it does cite, the mean precision of its citations is 0.30. Extraction always cites fragment 1, and it is correct in 7 of 9.
• Abstention. A tie at 2 of 3 on the unanswerable questions, but for different reasons: the BM25 threshold abstains by score and the LLM by reading (and it gets one answerable question wrong).

The honest conclusion: with a 0.5B model, generative RAG adds no measurable accuracy over an extraction with highlighting, and it adds errors of its own (invented figures, missing citations, language). A larger model changes the numbers, and that is what this evaluation makes it possible to measure: the same pipeline, with another generator, against the same gold. The lesson's header figure summarizes the table with its intervals.

Citations verifiable by offsets

Asking the model to “end with the fragment number in square brackets” is fragile: the model can omit the citation, cite a fragment it did not use or return only the number. Commercial APIs offer something better. In Anthropic's, a document sent with "citations": {"enabled": true} makes the answer arrive split into text blocks, with the cited blocks carrying a citations list; for a plain-text document, each citation is a char_location with the cited text, the document index and the offsets start_char_index (from 0) and end_char_index (exclusive) within the text sent11Note 1 · Anthropic, “Citations”, Messages API documentation, accessed on October 23, 2026: plain-text documents chunked into sentences, char_location citations with 0-based character indexes and an exclusive end, and an answer in several text blocks with their citations list.. That is exactly what is needed to verify in code: if the document text is sent as is, the citation offsets are those of rag6_pasajes.

The real code, with the Anthropic API (not run: it needs a key and network access):

import anthropic

cliente = anthropic.Anthropic()          # ANTHROPIC_API_KEY from the environment
doc = "R10"
respuesta = cliente.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "document",
         "source": {"type": "text", "media_type": "text/plain",
                    "data": TEXTO[doc]},
         "title": TITULO[doc],
         "citations": {"enabled": True}},
        {"type": "text",
         "text": "¿Cuál es el indicador resumido de riesgo?"}]}])
for bloque in respuesta.content:
    for c in getattr(bloque, "citations", None) or []:
        print(c.start_char_index, c.end_char_index, c.cited_text)

To run the same code without network access, the official client is pointed at a local simulator (httpx2.MockTransport, the test transport of the HTTP client used by Anthropic's 2.x SDK) that answers with the documented shape. The simulator's “model” is a trivial rule: it uses BM25 to find the sentence in the document most similar to the question and returns it cited with its offsets. It does not evaluate any model: it checks that the pipeline (send the document, read the citations, map them to passages) works.

import json

import anthropic
import httpx2


def simulador(request):
    """Answers like the Messages API with char_location citations."""
    cuerpo = json.loads(request.content)
    partes = cuerpo["messages"][0]["content"]
    doc, pregunta = partes[0], partes[1]["text"]
    t = doc["source"]["data"]
    frases = [(m.start(), m.end())
              for m in re.finditer(r"[^\n.]+(?:\.(?=\s|$))?", t)
              if len(m.group().strip()) > 20]
    bm_f = BM25Okapi([palabras(t[a:b]) for a, b in frases])
    a, b = frases[int(np.argmax(bm_f.get_scores(palabras(pregunta))))]
    cita = {"type": "char_location", "cited_text": t[a:b],
            "document_index": 0, "document_title": doc["title"],
            "start_char_index": a, "end_char_index": b}
    return httpx2.Response(200, json={
        "id": "msg_simulado", "type": "message", "role": "assistant",
        "model": cuerpo["model"], "stop_reason": "end_turn",
        "stop_sequence": None,
        "usage": {"input_tokens": 0, "output_tokens": 0},
        "content": [
            {"type": "text", "text": "Según el documento, "},
            {"type": "text", "text": t[a:b].strip(),
             "citations": [cita]}]})


cliente = anthropic.Anthropic(
    api_key="sk-simulado", base_url="http://simulado",
    http_client=httpx2.Client(
        transport=httpx2.MockTransport(simulador)))


def pregunta_con_citas(q):
    doc = q.id_doc
    r = cliente.messages.create(
        model="simulado", max_tokens=200,
        messages=[{"role": "user", "content": [
            {"type": "document",
             "source": {"type": "text", "media_type": "text/plain",
                        "data": TEXTO[doc]},
             "title": TITULO[doc], "citations": {"enabled": True}},
            {"type": "text", "text": q.pregunta}]}])
    citas = [c for b in r.content for c in (b.citations or [])]
    tramo = [dict(doc=doc, tramos=[(c.start_char_index,
                                    c.end_char_index)])
             for c in citas]
    pas_doc = pas[pas.id_doc == doc].id_pasaje
    tocados = [p for p in pas_doc if cubierta(tramo, p) > 0]
    return citas, tocados


Q = preg.set_index("id_pregunta")
for qid in ["QA196", "QA105", "QA230"]:
    q = Q.loc[qid]
    citas, tocados = pregunta_con_citas(q)
    c = citas[0]
    print(qid, q.pregunta[:58])
    print("   citation", c.start_char_index, c.end_char_index,
          repr(c.cited_text[:50]))
    ok = bool(set(tocados) & set(q.pasajes_oro.split(";")))
    print("   cited passages:", tocados, "| gold:", q.pasajes_oro,
          "| correct:", ok)

QA196 ¿Reparte dividendos la clase A del Sintético Bonos Gobiern
   citation 0 55 '# Folleto — Sintético Bonos Gobierno Euro 3-7 UCIT'
   cited passages: ['R11-p001'] | gold: R11-p030 | correct: False
QA105 ¿Qué margen EBITDA tuvo el segmento Internacional de Povdo
   citation 2870 2931 '| Segmento | Ventas 2023 (millones de USD) | Marge'
   cited passages: ['R05-p025'] | gold: R05-p025 | correct: True
QA230 ¿Cuáles son los costes de operación anuales del Sintético
   citation 0 73 '# Documento de datos fundamentales — Sintético Div'
   cited passages: ['R16-p001'] | gold: R16-p016 | correct: False

The pipeline works end to end: the official client sends the document with citations enabled, the simulator answers with the documented shape and the code turns each char_location into an offset span that is cross-checked with the canonical passages. The simulator's rule, on the other hand, is bad, and the verification catches it: for the EBITDA margin it cites the segment table (the gold passage), but in the other two it cites the document title, which repeats the fund's name and wins in BM25. With a real model the verification is the same: a citation that does not touch the gold passage is wrong, with no room for interpretation.

Common mistakes

• Evaluating with an overall score. “70% is right” mixes retriever, generator, citations and abstention. Each link is measured separately; if coverage is low, changing the model fixes nothing.
• Accepting “contains the figure” as accuracy. An answer can contain the correct figure inside an invented calculation. Accuracy also requires that there be no foreign figures; faithfulness, that all of them be in the context.
• Having no unanswerable questions. An evaluation set without questions the documents do not answer does not measure the most expensive hallucination: inventing a data point that does not exist (a rating, a figure for a year that is not there).
• Drawing conclusions from samples of ten questions. With 12 questions, there is no statistical difference between 33% and 50%. Report the intervals, evaluate more questions, or use a faster model for the evaluation.
• Not comparing with a baseline. Without the simple extraction you do not know what the LLM adds. In documents with literal answers, the best highlighted chunk can be more useful than a generated sentence.
• Trusting citations by fragment number. They get omitted or invented. Offset citations over the original document are verified in code.
• An uncalibrated LLM judge. A model that scores another one (Ragas, LLM-as-a-judge) also makes mistakes: before using it, measure it against a gold set like this one, just like the system.

Exercises

1. Context is paid for in tokens (and, with the local model, in seconds). For k = 1, 2, 3 and 5 chunks, measure the retriever's mean coverage on the 246 answerable questions and the tokens of the user message with the Qwen tokenizer. Where is the break-even point?

resp_ = preg[preg.respondible]
for k in (1, 2, 3, 5):
    cob, ntok = [], []
    for q in resp_.itertuples():
        top, _ = buscar(q.pregunta, k)
        oro = q.pasajes_oro.split(";")
        cob.append(np.mean([cubierta(top, p) >= 0.5 for p in oro]))
        texto = mensajes(q.pregunta, top)[1]["content"]
        ntok.append(len(tok(texto)["input_ids"]))
    print(f"k={k}: coverage {np.mean(cob):.3f},"
          f" {np.mean(ntok):4.0f} tokens on average")

k=1: coverage 0.688,  168 tokens on average
k=2: coverage 0.787,  294 tokens on average
k=3: coverage 0.873,  431 tokens on average
k=5: coverage 0.892,  687 tokens on average

2. The line baseline returns a single line from the best chunk. Write a variant that, if the question asks for a figure (it starts with “¿Cuánto” (how much), “¿Cuál” (what/which) or “¿Qué porcentaje” (what percentage)), returns only the first figure of that line. Measure its strict accuracy on the 246 answerable questions.

def primera_cifra(q):
    top, _ = buscar(q.pregunta)
    linea = mejor_linea(top[0], q.pregunta)
    linea = linea.split(":", 1)[-1] if "|" not in linea else linea
    pide = re.match(r"¿(Cuánto|Cuál|Qué porcentaje)", q.pregunta)
    m = NUM.findall(CITA.sub(" ", linea))
    if pide and m:
        no_anios = [x for x in m if not re.fullmatch(r"20\d\d", x)]
        return (no_anios or m)[0]
    return linea


ok = [exacta(primera_cifra(q), q) for q in resp_.itertuples()]
print("exact, first figure:", round(np.mean(ok), 3),
      "| whole line:", round(base[base.resp].linea_exacta.mean(), 3))

exact, first figure: 0.337 | whole line: 0.382

3. How would you evaluate a RAG system over the level's conference call transcripts (6.2.3), where the answers are mostly text (“what did management say about margins?”) and there are no gold passages?