--- title: "6.3.2 Chunking financial documents" description: "Fixed, recursive and structure-aware chunking of reports with tables and footnotes: chunks mapped to passages by offsets, recall@k measured, and XBRL." url: "https://pythonparatrading.com/en/python-trading-course/level-6/chunking-financial-documents/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/chunking-financial-documents.png" --- # 6.3.2 · Chunking financial documents: tables, footnotes and XBRL > A RAG system does not retrieve documents: it retrieves chunks. If the chunking splits a table or separates a figure from the footnote that qualifies it, no model will answer well. This lesson compares three chunking strategies on 21 synthetic annual reports, prospectuses and KIDs, measures how many reference passages each one retrieves, and separates what should go through text from what should go through structured data (XBRL). - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.3 · Financial RAG - Duration: 65 min - Prerequisites: [6.3.1 · Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant](https://pythonparatrading.com/en/python-trading-course/level-6/embeddings-and-vector-databases/), [6.1.2 · Text sources: EDGAR, CNMV, transcripts and social media](https://pythonparatrading.com/en/python-trading-course/level-6/text-sources-edgar-cnmv-transcripts-social-media/) ## By the end of this lesson - Implement fixed-size chunking, recursive chunking by separators and structure-aware chunking that respects headings, tables and footnotes. - Map any chunk to the canonical passages of a document by character offsets and measure recall@k against the questions with gold passages. - Compare strategies at the same context budget and by question type, and measure the effect of the context header (document and section). - Turn HTML and Markdown tables into retrievable text and decide when a figure should come from point-in-time XBRL facts instead of the text. ## Why chunking decides what gets retrieved A RAG system (*retrieval-augmented generation*) has two halves: a retriever that picks a few pieces of the documents and a model that writes the answer with them. Lesson 6.3.1 covers the retriever half (embeddings, vector indexes, BM25); this one covers what comes before: **how each document is split into pieces** (*chunks*). It is the least glamorous decision in the system and one of the most consequential, because the model only sees what the retriever hands it. ```mermaid flowchart TB D[Documents] --> T[Chunking] T --> I[Index] P[Question] --> B[Top-k search] I --> B B --> C[Context] C --> L[LLM] L --> R[Answer] ``` Financial documents punish bad chunking in particular: - **Tables.** The key figures (sales, debt, the ongoing charges of an ETF) live in tables. Cut in half, a table loses the header row that says what each column is, or the row the question asked for ends up in the neighboring chunk. - **Footnotes.** “Adjusted EBITDA¹” means nothing without “¹ excludes restructuring costs of 74.3 million”. In a report the note can sit right below the table or several pages away. - **Templates.** Two years of reports from the same company, or the prospectuses of five ETFs from the same manager, are almost identical. A chunk “Ventas | 13.668,9 | 13.378,6” (sales, in Spanish number format) does not say which company or year it belongs to. - **Data that is already structured.** The financial statement figures of US companies exist as tagged XBRL facts, with their period and filing date. Looking for them in the text is a detour. The lesson compares three chunking strategies with an objective measure (recall@k against reference passages) and ends with the structured alternative: parsed HTML and XBRL facts. ## The material: documents, canonical passages and gold questions The level 6 data include 21 synthetic documents in Spanish and in Markdown: the 2023 and 2024 annual reports of four fictitious Spanish issuers, the prospectus and KID of five fictitious UCITS ETFs, and three research notes. The example corpus is in Spanish, and so are the questions, so the fragments quoted below carry an English gloss in parentheses. What makes the documents useful for measuring is the **canonical segmentation**: 974 passages with their type (heading, text, table, note, list) and their exact position in characters, and 263 questions with the passages that must be cited to answer them. ```python import re import unicodedata import numpy as np import pandas as pd docs = pd.read_parquet("rag6_documentos.parquet") pas = pd.read_parquet("rag6_pasajes.parquet") preg = pd.read_csv("rag6_preguntas.csv") TEXTO = dict(zip(docs.id_doc, docs.texto)) TITULO = dict(zip(docs.id_doc, docs.titulo)) P = pas.set_index("id_pasaje") print(docs.tipo.value_counts().to_dict()) print(pas.tipo.value_counts().to_dict()) print(preg.tipo_pregunta.value_counts().to_dict()) r = P.loc["R01-p013"] print(r.id_doc, r.seccion, r.tipo, r.inicio, r.fin) print(TEXTO["R01"][r.inicio:r.fin][:150]) ``` ```output {'informe_anual': 8, 'folleto_etf': 5, 'kid': 5, 'research': 3} {'texto': 431, 'titulo': 252, 'lista': 119, 'tabla': 105, 'nota': 67} {'texto': 122, 'tabla': 64, 'no_respondible': 17, 'texto_y_nota': 16, 'nota_pie': 16, 'tabla_y_nota': 13, 'calculo': 8, 'comparacion': 7} R01 3. Magnitudes principales tabla 1577 1914 | (millones de USD) | 2023 | 2022 | |---|---|---| | Ventas | 13.668,9 | 13.378,6 | | EBITDA ajustado¹ | 3.769,6 | 3.222,8 | | Resultado neto | 2.442,8 ``` `texto[inicio:fin]` is the passage: with those offsets you can check, for any chunk of your own, which reference passages it contains and in what proportion. Questions of type `texto_y_nota` (text and note) and `tabla_y_nota` (table and note) need two passages, and some comparison questions need five (one per ETF). Question QA002 shows it (how much in lease liabilities Busmar Hogar had at the end of 2023 and whether they are included in net debt; gold answer: 369.2 million dollars, not included): the figure is in note 1 of the financial statements and the qualification in note ² of the key-figures table, many lines earlier. ```python q = preg.set_index("id_pregunta").loc["QA002"] print(q.pregunta, "|", q.respuesta_oro) for pid in q.pasajes_oro.split(";"): r = P.loc[pid] print(pid, r.inicio, r.seccion[:28], "|", TEXTO[r.id_doc][r.inicio:r.fin][:70]) ``` ```output ¿Cuántos pasivos por arrendamiento tiene Busmar Hogar al cierre de 2023 y se incluyen en la deuda neta? | 369,2 millones de dólares; no se incluyen R01-p044 5658 Notas seleccionadas a los es | Nota 1. Arrendamientos. El grupo reconoce pasivos por arrendamiento de R01-p015 2010 3. Magnitudes principales | ² Deuda financiera neta = deuda financiera bruta − caja y equivalentes ``` ## Three ways to chunk The three strategies produce the same thing: a list of chunks, each with its document and its offset **spans** `(inicio, fin)` (start, end). Storing the offsets, and not only the text, is what makes it possible to measure and, later, to cite. **Fixed size.** Windows of 500 characters with an overlap of 50. It is the simplest option and knows nothing about the document: it cuts wherever the boundary falls, including in the middle of a row or a figure. ```python def fijo(doc, n=500, solape=50): """Windows of n characters that overlap by `solape`.""" t, trozos, i = TEXTO[doc], [], 0 while True: f = min(i + n, len(t)) trozos.append(dict(doc=doc, tramos=[(i, f)])) if f == len(t): return trozos i = f - solape t = TEXTO["R01"] r = P.loc["R01-p013"] # the key-figures table for c in fijo("R01"): (i, f), = c["tramos"] if i < r.fin and f > r.inicio: print((i, f), repr(t[max(i, r.inicio):max(i, r.inicio) + 40])) ``` ```output (1350, 1850) '| (millones de USD) | 2023 | 2022 |\n|---' (1800, 2300) 'neto | 13.069,0 | 11.890,0 |\n| Dividendo' ``` The key-figures table ends up split between two windows, and the second one starts in the middle of a row: that chunk has figures without the column header. **Recursive.** The idea behind the “recursive” splitters of LangChain or LlamaIndex: try to split on the coarsest separator (a blank line); if a piece is still larger than the maximum, split it on the next one (line break, period, space), and then merge neighboring pieces back while they fit. It respects paragraphs and rows when it can, but it does not know what a table is or which section each chunk belongs to. ```python SEPARADORES = ["\n\n", "\n", ". ", " "] def cortes(t, i, f, seps, n): """Spans of at most n characters, using the coarsest separator that works.""" if f - i <= n: return [(i, f)] if not seps: return [(k, min(k + n, f)) for k in range(i, f, n)] s, bordes = seps[0], [i] k = t.find(s, i, f) while k != -1: bordes.append(k + len(s)) k = t.find(s, k + len(s), f) bordes.append(f) piezas = [] for a, b in zip(bordes[:-1], bordes[1:]): if b > a: piezas += cortes(t, a, b, seps[1:], n) unidos = [] # merge neighbors while they fit for a, b in piezas: if unidos and b - unidos[-1][0] <= n: unidos[-1] = (unidos[-1][0], b) else: unidos.append((a, b)) return unidos def recursivo(doc, n=500): t, trozos = TEXTO[doc], [] for a, b in cortes(t, 0, len(t), SEPARADORES, n): a += len(t[a:b]) - len(t[a:b].lstrip()) # trim whitespace b -= len(t[a:b]) - len(t[a:b].rstrip()) if b > a: trozos.append(dict(doc=doc, tramos=[(a, b)])) return trozos print(len(fijo("R01")), len(recursivo("R01"))) ``` ```output 23 25 ``` **Structure-aware.** It uses what the document says about itself: the blocks separated by blank lines, the `##` headings, the tables (lines starting with `|`) and the note marks (¹ ² ³…). Rules: 1. Headings are not chunks: they set the section of what follows. 2. A table is always one whole chunk, even if it exceeds the maximum. 3. A block that cites a note (¹, ²…) is joined to the note: the chunk has **two offset spans**. 4. Consecutive paragraphs of the same section are merged up to 800 characters. ```mermaid flowchart TB B[Block] --> H{Heading?} H -->|yes| S[Sets the section] H -->|no| N{Is it a note?} N -->|yes| X[Waits for its citation] N -->|no| T{Table or citation?} T -->|yes| U[Chunk with its note] T -->|no| J[Merged with the previous one] ``` ```python NOTA = "¹²³⁴⁵⁶⁷⁸⁹" def bloques(t): """Paragraphs, tables, headings and notes: whatever is separated by blank lines.""" return [(m.start(), m.end()) for m in re.finditer(r"[^\n](?:.|\n(?!\n))*", t)] def seccion(doc, i): """Last ## heading before position i.""" hs = re.findall(r"^##+ (.*)$", TEXTO[doc][:i + 1], re.M) return hs[-1] if hs else "" def estructura(doc, n=800): t = TEXTO[doc] bl = bloques(t) notas = {t[a]: (a, b) for a, b in bl if t[a] in NOTA} trozos = [] for a, b in bl: s = t[a:b] if s.startswith("#") or s[0] in NOTA: continue marcas = [notas[c] for c in dict.fromkeys(s) if c in notas] fija = s.startswith("|") or bool(marcas) ult = trozos[-1] if trozos else None if (ult and not fija and not ult["fija"] and seccion(doc, a) == seccion(doc, ult["tramos"][0][0]) and b - ult["tramos"][0][0] <= n): ult["tramos"][0] = (ult["tramos"][0][0], b) continue trozos.append(dict(doc=doc, tramos=[(a, b)] + marcas, fija=fija)) return trozos tabla = next(c for c in estructura("R01") if c["tramos"][0][0] == P.loc["R01-p013"].inicio) print(len(estructura("R01")), tabla["tramos"]) ``` ```output 29 [(1577, 1914), (1916, 2008), (2010, 2119)] ``` The key-figures table chunk carries three spans: the table and its two notes. The next figure draws, with the real offsets of that area of the 2023 Busmar Hogar report, the canonical passages and where each strategy cuts. *Fig. — Busmar Hogar 2023 annual report, characters 1,300 to 2,400: canonical passages (text, headings, the key-figures table and its two notes) and each strategy's chunks; alternating shades = different chunks.* Four rows over the same stretch of text. Top, canonical passages: the letter's text, the business-model heading and text, the key-figures heading, the table from 1,577 to 1,914, note 1 and note 2 up to 2,119, and the start of the stock-information table. Fixed chunking lays 500-character overlapping windows: one ends at 1,850, mid-table (highlighted line), and the next starts at 1,800. Recursive cuts at block boundaries but joins the table with note 1 and leaves note 2 with the next table. Structure makes one chunk of the table and both notes. The last piece is the **indexed text**. A chunk can be preceded by a context header with the document title and the section (“Busmar Hogar, S.A. — Informe anual 2023 > 3. Magnitudes principales”, that is, annual report 2023 > 3. Key figures). The header can be added to any strategy: for fixed and recursive chunks the last heading before their start is used. That separates the effect of the cuts from the effect of the header. ```python def con_texto(trozos, cabecera): """Adds to each chunk the text that will be indexed.""" for c in trozos: t = TEXTO[c["doc"]] cuerpo = "\n\n".join(t[i:f] for i, f in c["tramos"]) sec = seccion(c["doc"], c["tramos"][0][0]) cab = f"{TITULO[c['doc']]} > {sec}\n\n" if cabecera else "" c["texto"] = cab + cuerpo return trozos ESTRATEGIAS = {"fixed": fijo, "recursive": recursivo, "structure": estructura} TROZOS = {} for nombre, f in ESTRATEGIAS.items(): for cab in (False, True): trozos = [c for d in TEXTO for c in f(d)] TROZOS[(nombre, cab)] = con_texto(trozos, cab) for (nombre, cab), tr in TROZOS.items(): largo = np.mean([len(c["texto"]) for c in tr]) print(f"{nombre:10s} header={cab!s:5s} {len(tr):4d} chunks," f" {largo:5.0f} characters on average") print(TROZOS[("structure", True)][12]["texto"][:200]) ``` ```output fixed header=False 270 chunks, 487 characters on average fixed header=True 270 chunks, 561 characters on average recursive header=False 293 chunks, 404 characters on average recursive header=True 293 chunks, 480 characters on average structure header=False 376 chunks, 293 characters on average structure header=True 376 chunks, 372 characters on average Busmar Hogar, S.A. — Informe anual 2023 > Estados financieros resumidos | Balance (millones de USD) | 31-12-2023 | |---|---| | Activo no corriente | 19.543,5 | | Activo corriente (sin caja) | 3.417,2 ``` ## From a chunk to the passages: offsets and recall@k To know whether a chunk “contains” a gold passage there is no need to compare texts: intersecting intervals is enough. The `cubierta` (covered) function computes what fraction of a passage's characters falls inside a set of chunks (the union of their spans, so the overlap of the fixed windows is not counted twice). A passage counts as **retrieved** if the top $k$ chunks cover at least half of its characters, and the recall@k of a question is the fraction of its gold passages that were retrieved: $$ \text{recall@}k = \frac{1}{|O|}\sum_{p \in O} \mathbf{1}\left[\frac{|p \cap (c_1 \cup \dots \cup c_k)|}{|p|} \ge 0.5\right] $$ with $O$ the gold passages and $c_1, \dots, c_k$ the spans of the $k$ highest-scoring chunks. We also measure whether a question gets **all** of its passages (full coverage), which is what the table-and-note questions require. ```python def cubierta(trozos, pid): """Fraction of the characters of passage pid inside the union of the spans of `trozos`.""" r = P.loc[pid] iv = sorted((max(i, r.inicio), min(f, r.fin)) for c in trozos if c["doc"] == r.id_doc for i, f in c["tramos"]) total, hasta = 0, r.inicio for a, b in iv: a = max(a, hasta) if b > a: total, hasta = total + b - a, b return total / (r.fin - r.inicio) for nombre in ESTRATEGIAS: tr = [c for c in TROZOS[(nombre, True)] if c["doc"] == "R01"] cob = [float(round(cubierta([c], "R01-p013"), 2)) for c in tr] print(f"{nombre:10s} chunks touching the table:", [x for x in cob if x > 0], "| note ²:", max(cubierta([c], "R01-p015") for c in tr)) ``` ```output fixed chunks touching the table: [0.81, 0.34] | note ²: 1.0 recursive chunks touching the table: [1.0] | note ²: 1.0 structure chunks touching the table: [1.0] | note ²: 1.0 ``` With fixed-size chunking the key-figures table is split: one window covers 81% of its characters and the next one 34% (the two overlap by 50 characters). In the other two, a single chunk covers it entirely, and note ² is complete in some chunk under all three strategies, although with recursive chunking not in the table's chunk (previous figure). With or without the header, the offsets are the same: the header is added to the indexed text, not to the document. The retriever is still missing. We use **BM25** (lesson 6.3.1), which scores by word overlap weighted by rarity and normalized by chunk length. It is deterministic, fast and, as will be seen, very competitive on these texts full of proper names and figures. Normalization strips accents and a few stop words, the same for chunks and questions. The 17 questions the documents do not answer have no gold passages and are left out of the recall. ```python from rank_bm25 import BM25Okapi VACIAS = set("""de la el en y a los las del que se por con un una para es al lo su sus o como fue cual cuanto cuantos cuales segun""".split()) def palabras(s): s = unicodedata.normalize("NFKD", s.lower()) s = "".join(ch for ch in s if not unicodedata.combining(ch)) return [w for w in re.findall(r"\w+", s) if w not in VACIAS] resp = preg[preg.respondible].reset_index(drop=True) def ranking(trozos, kmax=10): """The kmax highest-scoring chunks for each question.""" bm = BM25Okapi([palabras(c["texto"]) for c in trozos]) return [np.argsort(-bm.get_scores(palabras(q)), kind="stable")[:kmax] for q in resp.pregunta] def medir(trozos, rk, k): filas = [] for q, orden in zip(resp.itertuples(), rk): top = [trozos[j] for j in orden[:k]] ok = [cubierta(top, p) >= 0.5 for p in q.pasajes_oro.split(";")] car = sum(len(c["texto"]) for c in top) filas.append((q.tipo_pregunta, np.mean(ok), all(ok), car)) return pd.DataFrame(filas, columns=["tipo", "recall", "completa", "car"]) RK = {clave: ranking(tr) for clave, tr in TROZOS.items()} filas = [] for (nombre, cab), tr in TROZOS.items(): for k in (3, 5): m = medir(tr, RK[(nombre, cab)], k) filas.append((nombre, cab, k, m.recall.mean(), m.completa.mean(), m.car.mean())) tabla = pd.DataFrame(filas, columns=[ "chunking", "header", "k", "recall", "complete", "characters"]) print(len(resp), "answerable questions") tabla.pivot_table(index=["chunking", "header"], columns="k", values=["recall", "characters"]).round(3) ``` ```output 246 answerable questions characters recall k 3 5 3 5 chunking header fixed False 1495.577 2495.577 0.228 0.408 True 1699.585 2815.459 0.798 0.862 recursive False 1172.317 1975.874 0.217 0.391 True 1452.394 2407.220 0.829 0.871 structure False 846.776 1561.736 0.254 0.476 True 1189.667 1959.715 0.873 0.892 ``` Two readings. The first, and the biggest: **without a context header no strategy works** (between 21.7% and 25.4% recall with k = 3), and with it all three exceed 79%. It is not a property of BM25: the eight annual reports share a template and the questions always say which company and which year they are about. A chunk with “Ventas | 13.668,9” (sales) and no header is as good a match for Busmar Hogar 2023 as for Celra 2024. In practice, the alternative or the complement is **filtering by metadata** (issuer, fiscal year, document type) before searching; exercise 2 measures it. The second: with a header, structure-aware chunking retrieves more with less text (87.3% with k = 3 and about 1,190 characters, versus 79.8% for fixed-size chunking with 1,700). But comparing at the same $k$ is not fair to anyone: each strategy delivers chunks of a different size. ## At the same context, and question by question An LLM's context costs: tokens that are paid for, seconds of computation (with the local model of 6.3.3, several per hundred tokens) and noise that distracts the model. The honest comparison puts the characters delivered on the horizontal axis and recall on the vertical axis, varying $k$ from 1 to 10. That is the lesson's header figure. ```python curvas = [] for nombre in ESTRATEGIAS: tr, rk = TROZOS[(nombre, True)], RK[(nombre, True)] for k in range(1, 11): m = medir(tr, rk, k) curvas.append((nombre, k, round(m.car.mean()), round(m.recall.mean(), 3))) curvas = pd.DataFrame(curvas, columns=["chunking", "k", "chars", "recall"]) curvas.pivot(index="k", columns="chunking", values=["chars", "recall"]) ``` ```output chars recall chunking fixed recursive structure fixed recursive structure k 1 569.0 490.0 408.0 0.569 0.668 0.688 2 1135.0 975.0 791.0 0.687 0.778 0.787 3 1700.0 1452.0 1190.0 0.798 0.829 0.873 4 2261.0 1928.0 1558.0 0.841 0.837 0.890 5 2815.0 2407.0 1960.0 0.862 0.871 0.892 6 3372.0 2881.0 2360.0 0.885 0.875 0.893 7 3927.0 3359.0 2755.0 0.909 0.878 0.907 8 4488.0 3820.0 3133.0 0.937 0.883 0.909 9 5045.0 4294.0 3492.0 0.952 0.904 0.917 10 5604.0 4772.0 3851.0 0.957 0.915 0.919 ``` Up to about 2,000 characters, structure-aware chunking is on top: with k = 3 (1,190 characters) it retrieves 87.3%, more than fixed-size chunking with k = 4 (2,261 characters, 84.1%). With a lot of context the advantage disappears and fixed-size chunking ends up ahead (95.7% with k = 10), because its large, overlapping windows end up covering almost everything around the answer. Structure-aware chunking plateaus around 92%: what it does not retrieve is not a matter of how many chunks, but of which questions, as the breakdown by type shows. ```python por_tipo = pd.DataFrame({ nombre: medir(TROZOS[(nombre, True)], RK[(nombre, True)], 5) .groupby("tipo").recall.mean() for nombre in ESTRATEGIAS}).round(3) por_tipo.assign(questions=resp.tipo_pregunta.value_counts()) ``` ```output fixed recursive structure questions tipo calculo 0.500 0.000 0.000 8 comparacion 0.571 0.814 0.200 7 nota_pie 1.000 1.000 1.000 16 tabla 0.812 0.875 0.875 64 tabla_y_nota 1.000 0.885 1.000 13 texto 0.918 0.926 0.992 122 texto_y_nota 0.688 0.750 0.750 16 ``` *Fig. — Recall@5 by question type (number of questions in brackets) for the three chunkings with a context header: square, fixed; triangle, recursive; highlighted circle, structure.* Seven rows. Text: fixed 92%, recursive 93%, structure 99%. Table: 81%, 88% and 88%. Footnote: all three 100%. Text and note: 69%, 75% and 75%. Table and note: 100%, 89% and 100%. Calculation: 50%, 0% and 0%. Comparison: 57%, 81% and 20%. - **Footnotes and table with note.** Structure-aware chunking gets 100%: the note travels with the table. Fixed-size chunking does too, by spillover: its 500-character windows usually take in the short table and the note below it. Recursive chunking loses the tables with notes that its cut separates (88.5%). - **Text and note.** None goes beyond 75%: the figure is in a paragraph (a note to the financial statements) and the qualification in the note of a table in another section. Joining the note to the table is not enough when the question comes in through the paragraph; the retriever would have to bring both, or the note would have to be indexed on its own as well. - **Calculation.** Growth questions (“how much did sales grow in 2023?”) have the key-figures table as gold, but the chairman's letter literally says “un 2,2 % más que en 2022” (2.2% more than in 2022). The retriever finds the answer in another passage and the passage metric counts it as a miss (fixed-size chunking gets half right because some of its windows mix the letter with the start of the table). A reminder that the gold is also a convention. - **Comparison.** “Which ETF has the lowest ongoing charges?” needs five prospectuses at once, and the result jumps around with no logic from one strategy to another (from 20% to 81.4%): with k = 5 and five documents, it is enough for two chunks of the same fund to take the place of another. There are only 7 questions and nothing should be read into that ranking. It is a question for a structured table (the five TERs) or for splitting it into five searches, one per fund. ## Tables: Markdown, linearized and embeddings A common recommendation is to **linearize** tables before indexing them: turn each row into a sentence that repeats the column header (“Ventas: 2023 = 13.668,9; 2022 = 13.378,6.”). The idea is that an embedding model understands a sentence better than a grid with bars. It can be measured: same strategy, same questions, tables in Markdown or linearized, with BM25 and with the small embedding model of the environment (all-MiniLM-L6-v2, 384 dimensions, trained mostly on English text; lesson 6.3.1). ```python import copy from sentence_transformers import SentenceTransformer def linealiza(tabla): """One sentence per row, with the header of each column.""" filas = [[x.strip() for x in linea.strip("|").split("|")] for linea in tabla.strip().split("\n")] cab = filas[0] frases = [f"{f[0]}: " + "; ".join(f"{h} = {v}" for h, v in zip(cab[1:], f[1:])) + "." for f in filas[2:]] return f"{cab[0]}. " + " ".join(frases) md = TROZOS[("structure", True)] lin = copy.deepcopy(md) for c in lin: t = TEXTO[c["doc"]] (i, f), *notas = c["tramos"] if t[i] == "|": cuerpo = [linealiza(t[i:f])] + [t[a:b] for a, b in notas] cab = f"{TITULO[c['doc']]} > {seccion(c['doc'], i)}\n\n" c["texto"] = cab + "\n\n".join(cuerpo) cambia = [j for j, (a, b) in enumerate(zip(md, lin)) if a["texto"] != b["texto"]] print(len(cambia), "chunks with a table:") print(lin[cambia[0]]["texto"].split("\n\n")[1][:150]) st = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu", local_files_only=True) cod = dict(normalize_embeddings=True, show_progress_bar=False, batch_size=64) Q = st.encode(resp.pregunta.tolist(), **cod) X_md = st.encode([c["texto"] for c in md], **cod) X_lin = X_md.copy() X_lin[cambia] = st.encode([lin[j]["texto"] for j in cambia], **cod) filas = [] for nombre, tr, X in [("markdown", md, X_md), ("linearized", lin, X_lin)]: densa = [np.argsort(-(X @ q), kind="stable")[:10] for q in Q] for buscador, rk in [("BM25", ranking(tr)), ("MiniLM", densa)]: m = medir(tr, rk, 5) filas.append((buscador, nombre, m.recall.mean(), m[m.tipo == "tabla"].recall.mean())) pd.DataFrame(filas, columns=["retriever", "tables", "recall@5", "recall@5 table"]).round(3) ``` ```output 105 chunks with a table: (millones de USD). Ventas: 2023 = 13.668,9; 2022 = 13.378,6. EBITDA ajustado¹: 2023 = 3.769,6; 2022 = 3.222,8. Resultado neto: 2023 = 2.442,8; 2022 = retriever tables recall@5 recall@5 table 0 BM25 markdown 0.892 0.875 1 MiniLM markdown 0.442 0.406 2 BM25 linearized 0.890 0.844 3 MiniLM linearized 0.422 0.344 ``` In these documents **linearizing does not help**: with BM25 the recall of the table questions drops from 87.5% to 84.4%, and with MiniLM from 40.6% to 34.4%. Repeating “2023 =” and “2022 =” in every row dilutes the words that distinguish one row from another. And the small dense model falls far behind BM25 across the board (44% versus 89%): it is an English model, and here the questions and texts are in Spanish and are told apart by proper names, years and figures, exactly what lexical search matches best. It is not a general conclusion about embeddings (a larger multilingual model, or hybrid search, changes the result; lesson 6.3.1), but it is a warning: **a chunking recommendation is measured on your own documents and questions**. ## SEC HTML and XBRL: when the data is already structured The SEC-style documents in the level 6 data are in HTML, like the real ones: `

` headings per *Item*, `` tables and `` notes. With BeautifulSoup structure-aware chunking is straightforward: one chunk per `

`, with the tables linearized (there is no Markdown to preserve here) and the context header with the issuer, the form and the period. ```python from bs4 import BeautifulSoup fil = pd.read_parquet("filings6_documentos.parquet") d = fil.set_index("id_doc").loc["D000162"] def tabla_html(tb): filas = [[x.get_text(" ", strip=True) for x in tr.find_all(["th", "td"])] for tr in tb.find_all("tr")] cab = filas[0] return " ".join(f"{f[0]}: " + "; ".join( f"{h} = {v or '-'}" for h, v in zip(cab[1:], f[1:])) + "." for f in filas[1:]) sopa = BeautifulSoup(d.texto, "html.parser") for h2 in sopa.find_all("h2"): partes = [] for el in h2.find_next_siblings(): if el.name == "h2": break partes.append(tabla_html(el) if el.name == "table" else el.get_text(" ", strip=True)) print(f"{d.emisor} > {d.tipo} {d.periodo_fin:%Y-%m-%d} >", h2.get_text(strip=True)) print(" ", " ".join(partes)[:110]) ``` ```output Rapsa Insurance Inc. > 10-Q 2019-03-31 > Part I, Item 1. Financial Statements Revenue: Three months = 8,983.5; 3 months year to date = 8,983.5. Net income (loss) (1): Three months = 21.9; Rapsa Insurance Inc. > 10-Q 2019-03-31 > Part I, Item 2. Management's Discussion and Analysis Revenue for the three months ended March 31, 2019 was $8,983.5 million. Assets under management at quarter end Rapsa Insurance Inc. > 10-Q 2019-03-31 > Part II, Item 1A. Risk Factors There have been no material changes to the risk factors disclosed in our most recent Annual Report on Form 10- ``` But for a question like “what was Rapsa's revenue in the fourth quarter of 2019?” the text is the wrong path. US companies file their financial statements with **XBRL** tags: each figure is a fact with its concept (`us-gaap:Revenues`), its unit, its period and, in the course data, the date and time of the filing. The SEC serves them through an API (`companyconcept`, `companyfacts` and `frames`)[^1]. With facts, a numerical question is a filter, not a search. Two details the text does not resolve and the facts do: - **The fourth quarter is not filed.** The 10-K gives the year and the September 10-Q the nine-month cumulative figure: the quarter is the difference. - **Amendments rewrite figures.** A 10-Q/A changes the earnings of a quarter that was already published. A backtest has to use the value known on each date (*point-in-time*), filtering by filing date. ```python xb = pd.read_parquet("filings6_xbrl.parquet").merge( fil[["id_doc", "ticker"]], on="id_doc") def hecho(ticker, concepto, inicio, fin, a_fecha): """Latest value of a fact filed up to a_fecha (UTC).""" m = ((xb.ticker == ticker) & (xb.concepto == concepto) & (xb.periodo_inicio == inicio) & (xb.periodo_fin == fin) & (xb.ts_presentacion <= pd.Timestamp(a_fecha, tz="UTC"))) s = xb[m].sort_values("ts_presentacion") return None if s.empty else s.valor.iloc[-1] / 1e6 ing = "us-gaap:Revenues" anual = hecho("RAPA", ing, "2019-01-01", "2019-12-31", "2020-06-30") nueve = hecho("RAPA", ing, "2019-01-01", "2019-09-30", "2020-06-30") ventas = pd.read_csv("acciones6_ventas.csv") oficial = ventas.query("ticker == 'RAPA' and " "fin_trimestre == '2019-12-31'").ventas.item() print(f"RAPA Q4 2019: {anual:.1f} - {nueve:.1f} = {anual - nueve:.1f}" f" (quarterly sales: {oficial})") bn = "us-gaap:NetIncomeLoss" for fecha in ["2020-09-30", "2020-12-31"]: print("BEBA, Q2 2020 net income known on", fecha, ":", hecho("BEBA", bn, "2020-04-01", "2020-06-30", fecha)) ``` ```output RAPA Q4 2019: 36104.6 - 26958.9 = 9145.7 (quarterly sales: 9145.7) BEBA, Q2 2020 net income known on 2020-09-30 : 224.5 BEBA, Q2 2020 net income known on 2020-12-31 : 187.1 ``` The subtraction gives exactly the quarter's sales. And BEBA's second-quarter 2020 net income was 224.5 million until a November amendment set it at 187.1. A RAG over the text of today's documents would answer 187.1 to a question asked as of September 2020: **look-ahead bias** (lesson 6.2.5) through the index. The practical rule is to split the work: | Question | Best path | Why | |---|---|---| | Figure from a US financial statement | XBRL facts | Exact, with period and filing date | | Derived figure (4th quarter, margins) | XBRL facts and calculation in code | Arithmetic is not delegated to the model | | Qualifications, policies, risks, guidance | Text chunked by structure | There is no XBRL tag, or there is one without the qualification | | Comparing many documents | Structured table or one search per document | The top-k of a single search does not return five documents | | Issuers without XBRL (prospectuses, KIDs, PDF reports) | Chunked text, whole tables | The only path | [^1]: SEC, “EDGAR Application Programming Interfaces”, accessed on October 23, 2026. According to that page, `frames` aggregates one fact per entity using the most recent filing that best fits the requested period: it is convenient for a cross-section, but it is not point-in-time. ## Common mistakes - **Measuring by eye.** Trying five questions and picking the strategy “that looks best”. Without reference passages with offsets there is no way to know whether a change helps or hurts; with them, every change is measured in seconds. - **Comparing at the same k.** Larger chunks gain recall through size. The comparison is at the same context budget. - **Chunking without a header.** In template documents (reports from several years, prospectuses from the same manager) the chunk must say which document and section it belongs to, or be filtered by metadata. - **Splitting tables and separating notes.** The row without the column header and the figure without its note are the two most common forms of “correct” answers that are not. - **Losing the offsets.** If the index only stores the chunk text, you can neither cite the original document nor map to passages; with the offsets, every answer points to a verifiable span (lesson 6.3.3). - **Asking the text for what is in XBRL.** And, conversely, forgetting the filing date: an index built today with amended documents mixes the future into the past. - **Generalizing a recipe.** “Linearize the tables”, “use 512-token chunks”, “use dense embeddings”: in these documents, all three make things worse or do not help. Measure with your own corpus and questions. ## Exercises 1. Change the size of fixed-size chunking to 300 and to 1,000 characters (10% overlap) and compare its recall with that of structure-aware chunking at about 1,200 characters delivered. Does any fixed size catch up with the structure-aware one? **Solution.** ```python for n in (300, 500, 1000): tr = con_texto([c for d in TEXTO for c in fijo(d, n, n // 10)], True) rk = ranking(tr) mejor = max(((k, medir(tr, rk, k)) for k in range(1, 11)), key=lambda km: (km[1].car.mean() <= 1300, km[1].car.mean())) k, m = mejor print(f"fixed {n:4d}: k={k:2d} {m.car.mean():6.0f} characters," f" recall {m.recall.mean():.3f}") ``` ```output fixed 300: k= 3 1114 characters, recall 0.772 fixed 500: k= 2 1135 characters, recall 0.687 fixed 1000: k= 1 1019 characters, recall 0.559 ``` With at most 1,300 characters delivered, no fixed window reaches the 87.3% of structure-aware chunking with k = 3. Small windows cut more tables and rows; large ones fit only a few into the budget. The optimal size of fixed-size chunking depends on the document, which is why it pays to measure before settling on one. 2. Filter by metadata before searching: for each question, keep only the chunks of its document (the question's `id_doc` column; with several, all of them) and measure the recall@3 of the three strategies **without** a header. How much of the header's effect does the filter recover? **Solution.** ```python def ranking_filtrado(trozos, k=3): rk = [] for q in resp.itertuples(): ids = set(q.id_doc.split(";")) sub = [j for j, c in enumerate(trozos) if c["doc"] in ids] bm = BM25Okapi([palabras(trozos[j]["texto"]) for j in sub]) orden = np.argsort(-bm.get_scores(palabras(q.pregunta)), kind="stable")[:k] rk.append([sub[j] for j in orden]) return rk for nombre in ESTRATEGIAS: tr = TROZOS[(nombre, False)] m = medir(tr, ranking_filtrado(tr), 3) print(f"{nombre:10s} no header, filtered: recall@3" f" {m.recall.mean():.3f}") ``` ```output fixed no header, filtered: recall@3 0.763 recursive no header, filtered: recall@3 0.822 structure no header, filtered: recall@3 0.837 ``` Filtering by document recovers almost all of the header's effect, and structure-aware chunking stays ahead. In a real system the filter needs to know which document the question is about (issuer and fiscal year), which is extracted from the question itself with rules or with an LLM (lesson 6.2.2). The header is more robust when the question does not say. 3. Using the XBRL facts, compute the fourth-quarter 2022 revenue of all companies with a 10-K for that year, as known on 2023-06-30, and compare it with `acciones6_ventas.csv`. How many match to a tenth of a million? **Solution.** ```python tick = sorted(xb[(xb.forma == "10-K") & (xb.concepto == ing) & (xb.periodo_fin == "2022-12-31")].ticker.unique()) dif = [] for tk in tick: a = hecho(tk, ing, "2022-01-01", "2022-12-31", "2023-06-30") n9 = hecho(tk, ing, "2022-01-01", "2022-09-30", "2023-06-30") v = ventas.query("ticker == @tk and fin_trimestre == '2022-12-31'") dif.append(round(a - n9, 1) - v.ventas.item()) dif = np.round(dif, 1) print(len(tick), "companies;", int((dif == 0).sum()), "match;", "max difference", np.abs(dif).max()) ``` ```output 60 companies; 60 match; max difference 0.0 ``` They all match: in the course data the XBRL facts come from the same sales as the sales file. With real data the subtraction is also the standard approach, but you should check that the 10-K and the 10-Q use the same fiscal period (not every company closes on December 31) and that no amendment filed after the query date slips into the calculation. ## Self-assessment 1. With BM25 and k = 3, structure-aware chunking without a header retrieves 25.4% of the gold passages, and with the “document > section” header, 87.3%. Why such a large difference? - A. Because the header makes the chunk longer and BM25 rewards long texts - B. Because without a header a table chunk does not say which company or year it belongs to, and the questions do: there are eight nearly identical reports - C. Because the header translates the text into English - D. Chance: with 246 questions the difference is not significant *Answer B: Because without a header a table chunk does not say which company or year it belongs to, and the questions do: there are eight nearly identical reports.* The 2023 and 2024 reports of the four companies share a template. A row “Ventas | 13.668,9” (sales) is indistinguishable across eight documents if the chunk does not carry the company name and the year. BM25 normalizes by length, so the header does not help by making the chunk longer, but by supplying the words the question uses to say which document it is about. 2. Why must the strategies be compared at the same number of retrieved characters and not only at the same k? - A. Because with larger chunks each one covers more passages: at the same k, chunking with large chunks gains recall at the cost of putting more text into the model's context - B. Because BM25 only works with chunks of the same size - C. Because recall does not depend on k - D. There is no need: at the same k the comparison is fair *Answer A: Because with larger chunks each one covers more passages: at the same k, chunking with large chunks gains recall at the cost of putting more text into the model's context.* With k = 10 fixed-size chunking reaches 95.7% recall, more than structure-aware chunking (91.9%), but it delivers 5,604 characters on average versus 3,851. At about 1,200 characters, structure-aware chunking is at 87.3% and fixed-size chunking between 68.7% and 79.8%. Context costs tokens, time and noise for the model. 3. A fixed-size chunk covers 60% of the characters of the key-figures table. What does that say about whether it contains the answer to “what was the net financial debt?” - A. That it contains it with 60% probability - B. That it certainly contains it, because the threshold is 50% - C. That it does not contain it - D. Nothing certain: overlap-based recall says what part of the gold passage is in the context, not whether the row that matters is there; it has to be checked against the answer as well *Answer D: Nothing certain: overlap-based recall says what part of the gold passage is in the context, not whether the row that matters is there; it has to be checked against the answer as well.* The 50% threshold is a convention for counting a passage as retrieved. The debt row may be in the 40% that ended up in the next chunk. That is why chunking should not split tables and why the end-to-end evaluation (6.3.3) looks at the answer, not only at the passages. 4. To answer “what were the sales in the fourth quarter of 2019?” for a US company, what is more reliable? - A. Search the 10-K paragraph that talks about the fourth quarter with BM25 - B. Ask the LLM without context - C. Subtract the nine-month figure of the third-quarter 10-Q from the annual revenue of the 10-K (XBRL), using the facts known on the query date - D. Look for the fourth-quarter 10-Q *Answer C: Subtract the nine-month figure of the third-quarter 10-Q from the annual revenue of the 10-K (XBRL), using the facts known on the query date.* The fourth quarter is not filed: the 10-K gives the year and the September 10-Q the nine-month cumulative figure. With XBRL facts the subtraction is exact (9,145.7 million for RAPA, the same as the quarter's sales figure) and it can be done point-in-time, filtering by filing date. 5. What does this lesson's structure-aware chunking do with the note “² Deuda financiera neta = …” (net financial debt)? - A. It discards it, because notes carry no information - B. It attaches it to the chunk of each block that cites the ² mark, while keeping its original position in the offsets - C. It turns it into a chunk of its own without a header - D. It translates it into XBRL *Answer B: It attaches it to the chunk of each block that cites the ² mark, while keeping its original position in the offsets.* The note qualifies a figure in the table (here, that net debt does not include leases). Separated from the table, a question about net debt retrieves the figure without the qualification. The chunk stores two offset spans, the table and the note, so it can be mapped to both canonical passages. ## Key points - The retriever returns chunks, not documents: a split table or a figure without its footnote cannot be fixed later with a better model. - Measuring requires a reference: canonical passages with offsets make it possible to map any chunk to the gold passages of each question and compute recall@k without eyeballing. - In these documents the context header (document and section) matters most: with BM25 and k = 3, from 22–25% to 80–87% recall across the three strategies. - At the same context budget structure-aware chunking wins (87.3% with about 1,200 characters); with a lot of context fixed-size chunking catches up, but at the cost of delivering 45% more text. - Linearizing tables does not improve retrieval here, neither with BM25 nor with a small English embedding model; questions that span several documents (comparisons) give unstable results with any chunking. - Figures that exist as XBRL facts are taken from the structured data, with their period and filing date (point-in-time), not from the text. ## Resources ### Documentation - [rank_bm25: BM25 implementations in Python (GitHub)](https://github.com/dorianbrown/rank_bm25) · github.com - [LlamaIndex: node parser modules (sentence, Markdown, HTML splitters…)](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/) · docs.llamaindex.ai - [SEC: EDGAR application programming interfaces (XBRL companyconcept, companyfacts and frames)](https://www.sec.gov/search-filings/edgar-application-programming-interfaces) · sec.gov - [XBRL International: an introduction to XBRL](https://www.xbrl.org/the-standard/what/an-introduction-to-xbrl/) · xbrl.org - [Beautiful Soup 4: documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) · crummy.com ### Data - [Level 6 data: documents for RAG with their canonical passages (Parquet)](https://pythonparatrading.com/curso/datos/nivel-6/rag6_pasajes.parquet) · Download · rag6_pasajes.parquet - [Level 6 data: evaluation questions with gold passages (CSV)](https://pythonparatrading.com/curso/datos/nivel-6/rag6_preguntas.csv) · Download · rag6_preguntas.csv Previous lesson: [6.3.1 · Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant](https://pythonparatrading.com/en/python-trading-course/level-6/embeddings-and-vector-databases/) Next lesson: [6.3.3 · RAG over reports, prospectuses and KIDs, and how to evaluate it](https://pythonparatrading.com/en/python-trading-course/level-6/rag-on-financial-reports-and-evaluation/)