--- title: "6.2.3 Extracting guidance, KPIs and tone with LLMs" description: "Guidance, KPIs and tone from filings, conference calls and news: rules, a local LLM, FinBERT and a dictionary measured against known ground truth." url: "https://pythonparatrading.com/en/python-trading-course/level-6/extracting-guidance-kpis-and-tone-with-llms/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/extracting-guidance-kpis-and-tone-with-llms.png" --- # 6.2.3 · Guidance, KPIs and tone: what an LLM extracts well > The information that moves a price on earnings day is in sentences: next quarter's guidance, the number of stores, the tone of the analysts' questions. This lesson extracts it with rules, with a 0.5B-parameter local LLM, with FinBERT and with a dictionary, and measures each method against the published ground truth of the data: where each one wins and what a larger model would change. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.2 · LLMs for research and trading - Duration: 75 min - Prerequisites: [6.2.2 · Structured prompting: JSON you can validate](https://pythonparatrading.com/en/python-trading-course/level-6/structured-prompting-json-with-pydantic/), [6.2.1 · How an LLM works: from the next token to the API](https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/), [6.1.1 · Tokenization, embeddings, financial dictionaries and FinBERT](https://pythonparatrading.com/en/python-trading-course/level-6/tokenization-embeddings-loughran-mcdonald-finbert/) ## By the end of this lesson - Measure an extraction with precision, coverage (recall) and numerical accuracy against a known ground truth. - Build a rules baseline (regular expressions) for guidance and KPIs, and understand why it wins on template texts. - Use a local LLM to extract data and to classify tone from the next-token probabilities, without generating text. - Compare an LLM's sentiment with FinBERT and with a dictionary on news, and the tone of the analysts' questions in the calls. - Separate what is due to model size from what is due to the method, citing the published evidence. ## What gets extracted and what it is for On earnings day, a listed company publishes a lot of information in the form of text. This lesson deals with three pieces of known value: - **Sales guidance** for the next quarter. Management knows the trend of its business better than the analyst consensus; in the course data, the distance between the midpoint of the guidance and the consensus anticipates part of the next quarter's sales surprise (README of the level 6 data). - **The operating KPI** of the sector (stores, subscribers, production, order backlog), which helps read sales before they are published. - **Tone**: that of the analysts' questions in the conference call, which in the data carries a planted drift of 2% per unit of tone over the following 20 sessions, and that of the news, which moves the price on the reaction day (lessons 6.1.1 and 6.1.3). There are three families of methods: **rules** (regular expressions and dictionaries), **discriminative models** fine-tuned for a task (FinBERT) and **generative LLMs**. The question is not which one is “best”, but which one gets more right on each task, at what cost and with what risk of failing silently. The course data make it possible to measure it, because the answer for each document is published: ```mermaid flowchart TB A[Filings and calls] --> B[Rules] A --> C[LLM] N[News] --> D[FinBERT] N --> C N --> E[Dictionary] B --> F[Against the truth] C --> F D --> F E --> F ``` All the texts are synthetic and come from templates (data README): the results say how each method behaves in a controlled case, not how much it would get right with real filings. The corpus mixes texts in English and in Spanish (SEC-style filings, CNMV-style announcements, CNMV being Spain's securities regulator, and calls in both languages); they are processed as they are, and the prompts given to the LLM are in Spanish too. ## The ground truth and how to measure `documentos6_extraccion_verdad.csv` says what each document (`id_doc`) and each call (`id_llamada`) contains: a `guidance` row with the period and the two ends in millions of USD, a `kpi` row with the value and its unit, and the literal sentence they come from. ```python import json import re import numpy as np import pandas as pd from bs4 import BeautifulSoup fil = pd.read_parquet("filings6_documentos.parquet") ll = pd.read_parquet("llamadas6_transcripciones.parquet") verdad = pd.read_csv("documentos6_extraccion_verdad.csv") verdad["fuente"] = np.where(verdad["id_doc"].notna(), "doc", "call") verdad["id"] = verdad["id_doc"].fillna(verdad["id_llamada"]) print(verdad.groupby(["fuente", "tipo"]).size().unstack()) verdad.loc[[0, 1, 6990], ["id", "tipo", "periodo", "bajo", "alto", "valor", "unidad"]] ``` ```output tipo guidance kpi fuente call 710 840 doc 3210 3780 id tipo periodo bajo alto valor unidad 0 D000007 guidance 2019T1 2440.0 2569.0 NaN millones de USD 1 D000007 kpi 2018T4 NaN NaN 1142.0 tiendas 6990 C00001 guidance 2019T1 2440.0 2569.0 NaN millones de USD ``` The columns are `tipo` (type), `periodo` (period, written `2019T1` for Q1 2019), `bajo` and `alto` (low and high end), `valor` (value) and `unidad` (unit, such as “millones de USD” or “tiendas”, stores). An extraction is evaluated like a classifier, with numbers. With $E$ the extracted data points, $V$ the true ones and $E \cap V$ the extracted ones that match a true one: $$ \text{precision} = \frac{|E \cap V|}{|E|} \qquad \text{recall} = \frac{|E \cap V|}{|V|} $$ “Matching” requires the same period and the same numbers within a 0.5% tolerance: a guidance of “\$7.23 billion” is 7,230 million, and the ground truth, 7,228; the text has no more precision than that. **Accuracy to the digit** is measured separately. An extraction with high precision and low coverage loses information silently; one with high coverage and low precision feeds false data into the model. ## Baseline: rules The first tool, and often the last, is a regular expression. The ones in this cell were written by reading the sentences in the `frase` (sentence) column of the ground truth: there are five ways of writing a guidance and a few KPI patterns. Numbers are read according to the language (in Spanish, the dot separates thousands and the comma, decimals). ```python ORD = {"first": 1, "second": 2, "third": 3, "fourth": 4, "primer": 1, "segundo": 2, "tercer": 3, "cuarto": 4} N = r"\d[\d.,]*\d|\d" GUIA = [ rf"range of \$(?P{N}) million to \$(?P{N}) million", rf"between \$(?P{N}) (?Pmillion|billion) and " rf"\$(?P{N})", rf"expected to be \$(?P{N})-(?P{N}) million", rf"entre (?P{N}) y (?P{N}) (?Pmil millones|millones)", rf"horquilla de (?P{N})-(?P{N}) millones", ] TRIM = (r"(first|second|third|fourth|primer|segundo|tercer|cuarto)" r" (?:quarter|trimestre) (?:of |de )?(\d{4})") KPI = rf"(?:at quarter end|al cierre del trimestre): (?P{N})" def numero(s, es): s = s.replace(".", "").replace(",", ".") if es else ( s.replace(",", "")) return float(s) def es_espanol(t): return bool(re.search(r"\b(trimestre|millones)\b", t)) def reglas(texto): """Guidance and KPI of a text, or None if not found.""" es = es_espanol(texto) guia = None for patron in GUIA: m = re.search(patron, texto) if m: k = 1000 if m.groupdict().get("u") in ( "billion", "mil millones") else 1 q = re.search(TRIM, texto[max(0, m.start() - 120): m.end()]) per = f"{q.group(2)}T{ORD[q.group(1)]}" if q else None guia = (per, round(numero(m["a"], es) * k, 4), round(numero(m["b"], es) * k, 4)) break m = re.search(KPI, texto) kpi = numero(m["v"], es) if m else None return guia, kpi def plano(html): return BeautifulSoup(html, "html.parser").get_text(" ") textos = pd.concat([ pd.Series([plano(t) for t in fil["texto"]], index=fil["id_doc"]), ll.groupby("id_llamada")["texto"].apply(" ".join)]) extr = {i: reglas(t) for i, t in textos.items()} print(len(extr), "texts;", reglas(textos["D000010"])) ``` ```output 4860 texts; (('2019T1', 19710.0, 20550.0), 441.0) ``` The evaluation function works for any method: it receives, per text, the extracted guidance `(periodo, bajo, alto)` and KPI (or `None`), and compares them with the ground truth. ```python G = verdad[verdad["tipo"] == "guidance"].set_index("id") K = verdad[verdad["tipo"] == "kpi"].set_index("id") def cerca(a, b, tol=0.005): return a is not None and abs(a / b - 1) < tol def evaluar(extr, ids): filas = {} for i in ids: guia, kpi = extr[i] hay_g, hay_k = i in G.index, i in K.index imp_g = hay_g and guia is not None and ( cerca(guia[1], G.loc[i, "bajo"]) and cerca(guia[2], G.loc[i, "alto"])) ok_g = imp_g and guia[0] == G.loc[i, "periodo"] dig_g = ok_g and (round(guia[1]) == G.loc[i, "bajo"] and round(guia[2]) == G.loc[i, "alto"]) ok_k = hay_k and cerca(kpi, K.loc[i, "valor"]) filas[i] = dict(g_ext=guia is not None, g_ver=hay_g, g_imp=imp_g, g_ok=ok_g, g_dig=dig_g, k_ext=kpi is not None, k_ver=hay_k, k_ok=ok_k) d = pd.DataFrame(filas).T.astype(int).sum() return pd.Series({ "guidance: precision": d.g_ok / max(d.g_ext, 1), "guidance: recall": d.g_ok / d.g_ver, "guidance: amounts": d.g_imp / d.g_ver, "guidance: to the digit": d.g_dig / max(d.g_ok, 1), "KPI: precision": d.k_ok / max(d.k_ext, 1), "KPI: recall": d.k_ok / d.k_ver}).round(3) pd.DataFrame({"documents": evaluar(extr, fil["id_doc"]), "calls": evaluar(extr, ll["id_llamada"].unique())}) ``` ```output documents calls guidance: precision 1.000 1.000 guidance: recall 1.000 1.000 guidance: amounts 1.000 1.000 guidance: to the digit 0.735 0.765 KPI: precision 0.969 1.000 KPI: recall 1.000 1.000 ``` The rules find all the guidance in the 4,020 documents and the 840 calls, with the right period and amounts (precision and recall of 1.000). To the digit they get 73.5% right in documents and 76.5% in calls: the rest are guidance figures written in billions with two decimals (“\$20.55 billion”, “2,57 mil millones”, that is, 2.57 billion), where the text does not have the precision of the ground truth: in D000010, 20,550 versus 20,545. No rule can recover what the text does not say; that is why the 0.5% tolerance is the fair measure and accuracy to the digit, a reminder of what is lost. The only blemish is the KPI precision in documents, 0.969. The 120 “false positives” are the amendments (96 10-Q/A and 24 10-K/A): they repeat the KPI of the original document and the ground truth does not count them as documents with a KPI. It is not an error of the rule but of the definition of the ground truth, and it only shows up when you read the errors one by one. Before believing a metric, look at its failures. A warning that weighs more than the table: **these rules were written by looking at the sentences**. The synthetic data use five guidance templates and a few KPI ones, and a rule written for them finds all of them. In real filings, each company words things its own way, changes its wording over the years and mixes figures across tables, notes and paragraphs; a rule needs continuous maintenance and a coverage measure that warns when it stops finding things. *Fig. — The 3,920 sales guidance statements in the level 6 data (documents and calls) come from eight sentence templates.* Eight horizontal bars with the number of guidance statements per template: in English, “range of … million” 1,124, “outlook … $N-N million” 1,090, “between … billion” 855 (highlighted: two-decimal billions that lose precision) and “between … million” 274; in Spanish, “entre … millones” 212, “horquilla de N-N” 188, “entre … mil millones” 171 and “facturar … millones” 6. The figure counts the guidance sentences by template, with figures and quarters replaced by placeholders: eight ways of writing them, four per language, and the “between … billion” one, in billions with two decimals, is the one that makes accuracy to the digit impossible. ## The LLM on a sample The model is the Qwen2.5-0.5B-Instruct from lessons 6.2.1 and 6.2.2, with a structured prompt that asks for the guidance and the KPI at the same time. An LLM of this size takes on the CPU about as long as a human takes to read, so the measurement uses a sample: 14 documents (8-K, 10-Q and CNMV-style announcements, trimmed to their text without tables) and 6 calls (management's turns in the presentation). The 20 requests are sorted by length before being grouped into batches: that way each batch carries less padding. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer torch.manual_seed(0) torch.use_deterministic_algorithms(True) NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct" tok = AutoTokenizer.from_pretrained(NOMBRE) tok.padding_side = "left" llm = AutoModelForCausalLM.from_pretrained( NOMBRE, dtype=torch.float32).eval() def recorte(html): t = re.sub(r".*?
", "", html, flags=re.S) t = "\n".join(l for l in t.splitlines() if not l.startswith("|") and "ynthetic" not in l and "intético" not in l) t = BeautifulSoup(t, "html.parser").get_text("\n") return re.sub(r"\n\s*\n+", "\n", t.replace("*", "")).strip() items = fil["items"].fillna("").astype(str) cand = fil[fil["tipo"].isin(["8-K", "10-Q", "OIR", "HR"]) & ~items.str.contains("4.02")] m_docs = cand.sample(14, random_state=5) pres = ll[(ll["rol"] == "directivo") & (ll["seccion"] == "presentacion")] pres = pres.groupby("id_llamada")["texto"].apply(" ".join) m_llam = pres.sample(6, random_state=5) muestra = pd.concat([ pd.Series([recorte(t) for t in m_docs["texto"]], index=m_docs["id_doc"]), m_llam]) SISTEMA = ( "Extrae del texto (1) la previsión de ventas del trimestre " "siguiente en millones de USD (1 billion = 1000) y (2) el KPI " "operativo al cierre del trimestre, tal como está escrito. " "Responde solo con JSON: {\"periodo\": \"AAAAT1-4\" o null, " "\"bajo\": number o null, \"alto\": number o null, " "\"kpi\": number o null}") def generar(textos_usuario, sistema, max_tokens=48, lote=10): prompts = [tok.apply_chat_template( [{"role": "system", "content": sistema}, {"role": "user", "content": t}], tokenize=False, add_generation_prompt=True) for t in textos_usuario] orden = np.argsort([len(p) for p in prompts]) salida = [None] * len(prompts) for k in range(0, len(orden), lote): sel = orden[k:k + lote] b = tok([prompts[j] for j in sel], return_tensors="pt", padding=True) with torch.inference_mode(): o = llm.generate(**b, do_sample=False, max_new_tokens=max_tokens, pad_token_id=tok.eos_token_id) n = b["input_ids"].shape[1] for j, fila in zip(sel, o): salida[j] = tok.decode(fila[n:], skip_special_tokens=True) return salida resp = generar(list(muestra), SISTEMA) print(len(resp), "responses;", resp[0].replace("`", "'")[:70]) ``` ```output 20 responses; {"periodo": "AAAAT1-4", "bajo": null, "alto": 11743, "kpi": 173} ``` The system prompt says: “Extract from the text (1) next quarter's sales guidance in millions of USD (1 billion = 1000) and (2) the operating KPI at quarter end, as written. Answer only with JSON”, followed by the format, in which `"AAAAT1-4"` is the Spanish equivalent of `YYYYQ1-4` (*año*, year; *trimestre*, quarter). The responses are read with the same tolerance as in lesson 6.2.2: first JSON object, numbers written as text converted and periods like `2024Q4` normalized to `2024T4`. Anything that cannot be read counts as “not extracted”. ```python def a_numero(v): if isinstance(v, (int, float)): return float(v) if isinstance(v, str): s = re.sub(r"[^\d.]", "", v.replace(",", ".")) return float(s) if s.count(".") <= 1 and s else None return None def leer(texto): m = re.search(r"\{.*?\}", texto, flags=re.S) try: d = json.loads(m.group(0)) if m else {} except json.JSONDecodeError: d = {} q = re.search(r"(\d{4})\s*[TQ]\s*0?([1-4])", str(d.get("periodo"))) per = f"{q.group(1)}T{q.group(2)}" if q else None bajo, alto = a_numero(d.get("bajo")), a_numero(d.get("alto")) guia = (per, bajo, alto) if bajo and alto else None return guia, a_numero(d.get("kpi")) extr_llm = {i: leer(r) for i, r in zip(muestra.index, resp)} pd.DataFrame({"rules": evaluar(extr, muestra.index), "LLM 0.5B": evaluar(extr_llm, muestra.index)}) ``` ```output rules LLM 0.5B guidance: precision 1.000 0.000 guidance: recall 1.000 0.000 guidance: amounts 1.000 0.053 guidance: to the digit 0.737 0.000 KPI: precision 1.000 0.000 KPI: recall 1.000 0.000 ``` The first response already shows the main problem: the model has literally copied the format placeholder, `"periodo": "AAAAT1-4"`, instead of writing the quarter. Small models copy the examples and templates in the prompt easily; a placeholder like that is replaced by a concrete example or, better, by structured outputs (lesson 6.2.2). In the sample, the rules get everything right and the LLM nothing: not one period, not one KPI, and the guidance amounts only in 1 of the 19 texts that contain it (5.3%). With the period copied from the placeholder, no complete guidance can be correct; that is why the table has a separate “amounts” row, which does not require the period, to see whether it at least reads the figures. It does not. ```python filas = [] for i in muestra.index[[0, 1, 2, 3, 14, 15]]: guia, kpi = extr_llm[i] filas.append({ "id": i, "LLM guidance": guia, "truth": (tuple(G.loc[i, ["periodo", "bajo", "alto"]]) if i in G.index else None), "LLM KPI": kpi, "true KPI": K.loc[i, "valor"] if i in K.index else None}) pd.DataFrame(filas).set_index("id") ``` ```output LLM guidance truth LLM KPI true KPI id D001138 None (2020T4, 11570.0, 12027.0) 173.00 1462.00 D001430 None (2021T2, 12093.0, 12472.0) 12093.00 1327.00 D002985 None (2024T1, 28516.0, 30074.0) 0.88 633.00 D002444 None (2023T2, 6587.0, 6986.0) 699000.00 3625.00 C00801 (None, 7297.7, 7437.0) (2025T3, 7064.0, 7437.0) 1007.70 14.28 C00468 None (2022T4, 3753.0, 3948.0) NaN 1253.00 ``` The table shows six cases. In four documents the model returns no readable guidance and puts any figure as the KPI: 173, the low end of the guidance (12,093), 0.88 or 699,000, when the KPI is written as is in the text. In call C00801 it reads the high end (7,437), makes up the low one (7,297.7 instead of 7,064) and writes 1,007.7 as the KPI where the text says 14.28 million. These are not format errors: the model does not locate the requested figure. ## Sentences the rule has not seen The rules get it right because they know the templates. To see what happens with new wording, the next cell uses eight sentences written for this lesson, four in English and four in Spanish, with their answer, that follow none of the data templates; the last one contains no guidance. ```python nuevas = [ ("We now see third-quarter 2024 sales landing somewhere " "around $1.2bn to $1.3bn.", ("2024T3", 1200, 1300)), ("Management guided Q2 2025 net sales to USD 845 to 870 " "million.", ("2025T2", 845, 870)), ("For Q1 2025 we expect revenue of $3.40 to $3.55 billion.", ("2025T1", 3400, 3550)), ("Q4 2024 outlook: top line between 12.5 and 13.1 billion " "dollars.", ("2024T4", 12500, 13100)), ("La dirección estima que la cifra de negocio del tercer " "trimestre de 2024 rondará los 950 a 1.000 millones de " "dólares.", ("2024T3", 950, 1000)), ("Esperamos ingresos de 2.100 millones de dólares como mínimo " "y de 2.250 millones como máximo en el cuarto trimestre de " "2023.", ("2023T4", 2100, 2250)), ("Prevemos unas ventas de 1,15 a 1,22 mil millones de dólares " "para el segundo trimestre de 2025.", ("2025T2", 1150, 1220)), ("No damos previsiones para el próximo trimestre.", None), ] r_nuevas = generar([f for f, _ in nuevas], SISTEMA) def importes(guia, esperado): if esperado is None: return guia is None return guia is not None and cerca(guia[1], esperado[1]) and ( cerca(guia[2], esperado[2])) def completo(guia, esperado): return importes(guia, esperado) and ( esperado is None or guia[0] == esperado[0]) def si(x): return "yes" if x else "no" for (frase, esp), r in zip(nuevas, r_nuevas): g_reg, g_llm = reglas(frase)[0], leer(r)[0] print(f"rules {si(completo(g_reg, esp))} · LLM " f"{si(completo(g_llm, esp))} (amounts " f"{si(importes(g_llm, esp))}) · {g_llm}") ``` ```output rules no · LLM no (amounts no) · (None, 1.2, 1.3) rules no · LLM no (amounts no) · (None, 839.0, 870.0) rules no · LLM no (amounts no) · (None, 3.4, 3.55) rules no · LLM no (amounts no) · (None, 12.5, 13.1) rules no · LLM no (amounts yes) · (None, 950.0, 1000.0) rules no · LLM no (amounts yes) · (None, 2100.0, 2250.0) rules no · LLM no (amounts yes) · (None, 1150.0, 1220.0) rules yes · LLM yes (amounts yes) · None ``` The Spanish sentences say, in order: management estimates that third-quarter 2024 revenue will be around 950 to 1,000 million dollars; we expect revenue of at least 2,100 and at most 2,250 million dollars in the fourth quarter of 2023; we forecast sales of 1.15 to 1.22 billion dollars for the second quarter of 2025; and we do not give guidance for next quarter. The rules find none of the seven new guidance sentences, because none follows their templates; they only get the eighth one right, which contains no guidance, by finding nothing. The LLM gets none completely right (it leaves the period as `None` again), but it reads the amounts of the three Spanish sentences correctly, including the one with “1,15 a 1,22 mil millones” (1.15 to 1.22 billion), which it converts to 1,150 and 1,220. In the English ones it gets the units wrong (1.2 and 1.3 instead of 1,200 and 1,300 million; 3.4 and 12.5 billion left unconverted) and in one it reads 839 where the text says 845. That is the argument for LLMs in extraction: they **generalize** to wording nobody anticipated. A 0.5B model does it halfway; with a large one, unit conversion and the period are the kind of instruction it follows well, although it has to be measured just as here. *Fig. — Sales guidance with both amounts correct (error < 0.5%): rules against the 0.5B LLM, on template texts from the sample and on seven sentences written for the lesson.* Four horizontal bars. On template texts, the rules get 100% right and the LLM 5.3% (1 of 19). On new phrasing, the rules 0% (0 of 7) and the LLM 43% (3 of 7, highlighted), all three in Spanish. The figure summarizes the guidance extraction: on the template texts the rules get everything right and the LLM almost nothing; on new sentences, the rules find nothing and the LLM reads the amounts of three of the seven correctly. ## The tone of the questions in conference calls Classifying does not require generating text. With a prompt that asks for a one-word answer (`positive`, `negative` or `neutral`), a single forward pass is enough, looking at the probability the model gives to each of the three words as the next token. The difference $p_{pos} - p_{neg}$, normalized by the mass of the three, is a continuous score between −1 and 1; and the output cannot go off format, because it is read directly from the logits (the idea of constrained decoding, lesson 6.2.2). $$ s = \frac{p_{\text{positive}} - p_{\text{negative}}} {p_{\text{positive}} + p_{\text{negative}} + p_{\text{neutral}}} $$ ```python ETIQUETAS = ["positive", "negative", "neutral"] IDS = [tok(e)["input_ids"][0] for e in ETIQUETAS] print([tok.decode([i]) for i in IDS]) def tono_llm(textos, sistema, lote=10): prompts = [tok.apply_chat_template( [{"role": "system", "content": sistema}, {"role": "user", "content": t}], tokenize=False, add_generation_prompt=True) for t in textos] orden = np.argsort([len(p) for p in prompts]) P = np.zeros((len(textos), 3)) for k in range(0, len(orden), lote): sel = orden[k:k + lote] b = tok([prompts[j] for j in sel], return_tensors="pt", padding=True) with torch.inference_mode(): z = llm(**b).logits[:, -1].double() P[sel] = torch.softmax(z, -1)[:, IDS].numpy() return (P[:, 0] - P[:, 1]) / P.sum(1), P.sum(1) ``` ```output ['positive', 'negative', 'neutral'] ``` The three words are encoded as a single token each, so the probability of the first token of the answer is enough. The analysts' questions in each call are joined into one text. The dictionary is the course's fictitious financial lexicon (`texto6_lexico.csv`, with the Loughran and McDonald categories, lesson 6.1.1) and its score is $(pos - neg)/(pos + neg + 1)$: ```python lex = pd.read_csv("texto6_lexico.csv") lex = lex[lex["lista"] == "financiera"] POS = set(lex.loc[lex["categoria"] == "positiva", "palabra"]) NEG = set(lex.loc[lex["categoria"] == "negativa", "palabra"]) def diccionario(t): w = re.findall(r"[a-záéíóúñü]+", t.lower()) p, n = sum(x in POS for x in w), sum(x in NEG for x in w) return (p - n) / (p + n + 1) lv = pd.read_csv("llamadas6_verdad.csv").set_index("id_llamada") preg = (ll[ll["rol"] == "analista"].groupby("id_llamada") ["texto"].apply(" ".join)) llam = lv[["tono_analistas"]].join(preg) llam["dicc"] = llam["texto"].map(diccionario) m_tono = llam.sample(20, random_state=0) SIS_TONO = ("These are analysts' questions in an earnings call. Is " "their tone about the company positive, negative or " "neutral? Answer with one word.") m_tono["llm"], masa = tono_llm(list(m_tono["texto"]), SIS_TONO) print(f"mass of the three labels: {masa.mean():.3f}") print("840 calls, dictionary:", round(llam[["dicc", "tono_analistas"]].corr().iloc[0, 1], 3)) m_tono[["llm", "dicc", "tono_analistas"]].corr().round(3) ``` ```output mass of the three labels: 0.343 840 calls, dictionary: 0.887 llm dicc tono_analistas llm 1.000 0.755 0.759 dicc 0.755 1.000 0.924 tono_analistas 0.759 0.924 1.000 ``` The dictionary (`dicc`) recovers the analysts' tone (`tono_analistas`) with a correlation of 0.887 over the 840 calls and 0.924 in the sample of 20; the LLM, 0.759 in the same sample. The three words get only 34.3% of the next-token probability (the rest is spread among other possible starts of the answer), and the normalization in the formula corrects that scale. The dictionary winning is not proof that dictionaries beat LLMs on conference calls: the data README warns that the questions reuse template phrases written with the words of the lexicon, so the dictionary has a much easier time than with real transcripts. On news, with phrases outside the lexicon and negations, the comparison is more realistic. ## News sentiment: LLM, FinBERT and dictionary FinBERT (ProsusAI/finbert) is a BERT fine-tuned on the Financial PhraseBank to classify sentences as positive, negative and neutral (Araci, 2019); its weights are published under a **CC-BY-NC-SA-4.0** license, which does not allow commercial use (lesson 6.1.1). It only understands English, so the comparison uses original news in English: 40 for the three methods and 500 for FinBERT and the dictionary, which are cheap. ```python from transformers import pipeline noti = pd.read_parquet("noticias6_corpus.parquet").merge( pd.read_csv("noticias6_verdad.csv"), on="id_noticia") en = noti[(noti["idioma"] == "en") & (noti["tipo_noticia"] == "original")] grande = en.sample(500, random_state=0) txt = list(grande["titular"] + ". " + grande["cuerpo"]) verd = grande["sentimiento_verdadero"].to_numpy() finbert = pipeline("text-classification", model="ProsusAI/finbert", device="cpu") def puntua_finbert(textos): out = finbert(textos, top_k=None, batch_size=32) d = [{x["label"]: x["score"] for x in o} for o in out] return np.array([x["positive"] - x["negative"] for x in d]) s_fb = puntua_finbert(txt) s_dic = np.array([diccionario(t) for t in txt]) SIS_NOT = ("Classify the news for the company's shareholders. " "Answer with one word: positive, negative or neutral.") s_llm, _ = tono_llm(txt[:40], SIS_NOT) def clase(s, umbral=0.0): return np.sign(np.where(np.abs(s) <= umbral, 0, s)) filas = {} for nombre, s, n in [("LLM 0.5B", s_llm, 40), ("FinBERT", s_fb[:40], 40), ("dictionary", s_dic[:40], 40), ("FinBERT", s_fb, 500), ("dictionary", s_dic, 500)]: filas[(nombre, n)] = { "correlation": np.corrcoef(s, verd[:n])[0, 1], "3-class accuracy": np.mean( clase(s, 0.2) == np.sign(verd[:n]))} pd.DataFrame(filas).T.round(3) ``` ```output correlation 3-class accuracy LLM 0.5B 40 0.868 0.725 FinBERT 40 0.917 0.875 dictionary 40 0.592 0.675 FinBERT 500 0.881 0.860 dictionary 500 0.702 0.712 ``` On the 40 English news items, FinBERT has the highest correlation with the true sentiment (0.917) and gets the class right in 87.5% of cases; the 0.5B LLM comes close in correlation (0.868) and does worse on classes (72.5%); the dictionary is far behind (0.592 and 67.5%). The header figure brings together these correlations and those of the calls. With 500 news items, FinBERT gives 0.881 and the dictionary 0.702, in line with the 0.71 correlation the README measures on all the news. The classes use a 0.2 threshold on the score (below it, neutral). The contrast with the extraction is the most useful lesson. The same 0.5B model that cannot locate a figure classifies the tone of a news item almost as well as a model fine-tuned for that. Classifying the tone of a short text is a task that pretraining covers well and that does not require following instructions precisely; locating a specific figure, converting its units and not confusing it with another one does. Model size weighs much more in the second. ## What a larger model would change This lesson cannot run a large model, so what follows are published results, with their source: - **Size matters on knowledge benchmarks.** In the Qwen2.5 technical report, the 0.5B-parameter base model scores 47.5 on MMLU and the 72B one, 86.1 (tables 5 and 2); in the instruct version, MMLU-Pro goes from 15.0 to 71.1 (tables 6 and 10)[^1]. - **And in finance.** Lopez-Lira and Tang (version 6, October 2025) write in their abstract that “forecasting ability generally increases with model size”; in the body of the paper, the basic models (GPT-1, GPT-2, BERT) give accuracy below 65% when scoring headlines, and their scores do not correlate significantly with the subsequent return drift. - **Some prompting techniques only work at scale.** Wei et al. (2022) observe that asking for step-by-step reasoning “only yields performance gains when used with models of ∼100B parameters”[^2]: it makes no sense to try it with a 0.5B model and conclude that it does not work. - **A small, specialized model may be enough.** Huang, Wang and Yang (2023) show that their FinBERT beats the Loughran-McDonald dictionary and other machine learning algorithms at sentiment classification: for a narrow task, a fine-tuned BERT-type model is a tough baseline. With a commercial model, the prompting levers from lesson 6.2.2 change the extraction result: the full schema with structured outputs, worked examples in the prompt (above all for unit conversions), a text field for the model to copy the sentence it takes the guidance from (which also makes it possible to check it) and separating the search for the sentence from the reading of the numbers. What does not change is this lesson's method: measure against a ground truth, with precision, coverage and accuracy, before using the data point in a signal. ## Exercises 1. With the guidance the rules extract from the documents, compute for each one the logarithm of the ratio between the midpoint and the sales consensus for that quarter (`acciones6_ventas.csv`) and its correlation with the sales surprise, $\log(\text{ventas}/\text{consenso})$ (sales over consensus). Does it match the one the README gives (0.18)? **Solution.** ```python ventas = pd.read_csv("acciones6_ventas.csv") tk = pd.read_csv("acciones6_tickers.csv") emp = pd.read_csv("acciones6_empresas.csv") actual = (tk.merge(emp[["id_empresa", "ticker"]], on="id_empresa", suffixes=("", "_actual")) .drop_duplicates("ticker") .set_index("ticker")["ticker_actual"]) tick_doc = fil.set_index("id_doc")["ticker"].map(actual) filas = [] for i, (guia, _) in extr.items(): if i in tick_doc.index and guia is not None and guia[0]: fin = pd.Period(guia[0].replace("T", "Q")).end_time filas.append({"ticker": tick_doc[i], "fin_trimestre": fin.strftime("%Y-%m-%d"), "medio": (guia[1] + guia[2]) / 2}) pg = (pd.DataFrame(filas) .drop_duplicates(["ticker", "fin_trimestre"]) .merge(ventas, on=["ticker", "fin_trimestre"])) x = np.log(pg["medio"] / pg["consenso_ventas"]) y = np.log(pg["ventas"] / pg["consenso_ventas"]) print(len(pg), "quarters · correlation", round(np.corrcoef(x, y)[0, 1], 3)) ``` ```output 1713 quarters · correlation 0.177 ``` It matches: 1,713 quarters and a correlation of 0.177, the README's figure rounded (0.18). Management's guidance carries information the consensus does not have, but not much (the correlation with the earnings-day return is practically zero, 0.02, according to the same README). The rules reproduce the result because they extract all the guidance; with the lesson's 0.5B LLM there would not be a single complete guidance to compute it with. 2. Repeat the news comparison with the class threshold at 0.1 and at 0.4 for FinBERT and the dictionary on the 500 news items. Which method is more sensitive to the threshold, and why? **Solution.** ```python pd.DataFrame({u: {"FinBERT": np.mean(clase(s_fb, u) == np.sign(verd)), "dictionary": np.mean(clase(s_dic, u) == np.sign(verd))} for u in [0.1, 0.2, 0.4]}).round(3) ``` ```output 0.1 0.2 0.4 FinBERT 0.866 0.860 0.868 dictionary 0.712 0.712 0.678 ``` FinBERT barely changes (between 0.860 and 0.868): its $p_{pos} - p_{neg}$ scores tend to be close to −1, 0 or 1, because the softmax of a fine-tuned classifier is very confident, and few fall between 0.1 and 0.4. The dictionary drops at 0.4 (from 0.712 to 0.678): its score takes discrete values ($\pm 1/4$, $\pm 1/2$, $\pm 2/3$…) and at 0.4 the news items at $\pm 1/4$ become neutral. A threshold is chosen with validation data, not with the evaluation data. 3. Write, without running it, the prompt you would use with a commercial model to extract the operating KPI from the calls, and say how you would check each response without reading it by hand. **Solution.** A system prompt with the role (“You extract data from earnings call transcripts”), the definition of each sector's KPI (stores, subscribers, production…), the unit as written and the rule not to convert it, the no-data case (`null`) and two or three worked examples, one in each language and one without a KPI. The format, with structured outputs and a Pydantic model with `metrica`, `valor`, `unidad` and `frase` (metric, value, unit and the literal sentence the number comes from). The check without reading: that `frase` appears verbatim in the transcript, that `valor` appears written in `frase` (with the language's number convention), that the `metrica` is the one for the company's sector, and that the value is plausible against the same company's previous quarter (a 1,000-fold jump gives away a unit error). Whatever fails is reviewed by hand and counted: that rate is the system's real precision. [^1]: Qwen Team (2024): “Qwen2.5 Technical Report”, arXiv:2412.15115 (https://arxiv.org/abs/2412.15115). [^2]: Wei, J. et al. (2022): “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, arXiv:2201.11903 (https://arxiv.org/abs/2201.11903), section 3.2. ## Self-assessment 1. The lesson's regular expressions get practically 100% of the guidance in 4,020 documents right. Which conclusion is correct? - A. That rules are better than any LLM for extracting guidance - B. That the LLM adds nothing in finance - C. That the data are easy - D. That the rules were written by looking at the few templates the sentences were generated from; with new wording their coverage drops to zero, and in real filings the variety is enormous *Answer D: That the rules were written by looking at the few templates the sentences were generated from; with new wording their coverage drops to zero, and in real filings the variety is enormous.* The synthetic sentences come from eight templates. A rule written for them finds all of them and none of the lesson's handwritten sentences. With real text, a rule needs continuous maintenance and a coverage measure; an LLM generalizes better, if it is large enough. 2. Why does the lesson classify sentiment from the probabilities of the tokens “positive”, “negative” and “neutral” instead of letting the model write the answer? - A. Because a single forward pass gives a continuous score (p. positive − p. negative), without generating text, without off-format answers and much faster - B. Because the model cannot write - C. Because FinBERT requires it - D. Because that way the model cannot be wrong *Answer A: Because a single forward pass gives a continuous score (p. positive − p. negative), without generating text, without off-format answers and much faster.* The output is restricted to three options (the idea of constrained decoding, lesson 6.2.2) and the difference in probabilities serves as a score to correlate with the ground truth or to build a signal. It can still be wrong all the same. 3. On the sample's English news, the 0.5B LLM has a correlation with the true sentiment close to FinBERT's (0.87 versus 0.92) and much higher than the dictionary's (0.59); in guidance extraction the same LLM gets almost nothing right. How is that explained? - A. The LLM is better than FinBERT - B. Classifying the tone of a short sentence is a task a small model solves with what it learned in pretraining; extracting a figure, converting units and telling the guidance apart from the quarter's sales requires following instructions precisely, which is what improves most with size - C. The news items are longer than the filings - D. It is chance: with another seed it would be the other way round *Answer B: Classifying the tone of a short sentence is a task a small model solves with what it learned in pretraining; extracting a figure, converting units and telling the guidance apart from the quarter's sales requires following instructions precisely, which is what improves most with size.* The two tasks require different capabilities. Model technical reports (for example, Qwen2.5's) show that knowledge and instruction-following benchmarks improve a lot with size; a 0.5B model is fine for a simple classification and not for a reliable extraction. 4. The dictionary recovers the tone of the analysts' questions with a correlation of almost 0.9, better than the LLM. What does the data README warn about? - A. That the Loughran-McDonald dictionary is perfect - B. That the calls are in English - C. That the questions reuse template phrases written with the words of the lexicon, so the dictionary has a much easier time than with real texts; on news, with phrases outside the lexicon and negations, the comparison is more realistic - D. That the LLM is misconfigured *Answer C: That the questions reuse template phrases written with the words of the lexicon, so the dictionary has a much easier time than with real texts; on news, with phrases outside the lexicon and negations, the comparison is more realistic.* The tone of the questions is written with phrases from the same fictitious lexicon the dictionary uses. It is a result of the data, not proof that dictionaries beat models on real calls. 5. A widely cited study on financial statement analysis with LLMs (Kim, Muhn and Nikolaev, arXiv:2407.17866) appears as withdrawn on arXiv. What should you do with its result? - A. Cite it anyway: it was widely circulated - B. Not rely on it while it is withdrawn: the authors withdrew it after detecting inconsistencies in the data and analyses when replicating it; always check the status of the version you cite - C. Use only the first version, which is not withdrawn - D. Replace it with a press article that summarizes it *Answer B: Not rely on it while it is withdrawn: the authors withdrew it after detecting inconsistencies in the data and analyses when replicating it; always check the status of the version you cite.* Version 3 (February 20, 2025) carries the notice of temporary withdrawal due to inconsistencies detected when replicating the analyses. A figure from a withdrawn paper is not evidence; neither is version 1, because it is the same paper. ## Key points - An extraction is measured like a classifier with numbers: precision (what is extracted is correct), coverage (everything there is gets extracted) and numerical accuracy (within a tolerance and to the digit). - On template texts, a few regular expressions extract almost everything; with new wording their coverage drops to zero. A rule measures its own coverage, or it fails silently. - The 0.5B LLM extracts guidance and KPIs badly: it confuses the quarter's sales with the guidance, does not convert billions and changes the quarter. It is a limit of size, not of the method. - Classifying tone takes a single pass: the probabilities of “positive”, “negative” and “neutral” give a continuous score. On news, the small LLM gets close to FinBERT (0.87 versus 0.92 correlation) and beats the dictionary (0.59). - In the synthetic calls, the dictionary wins because the questions reuse phrases from the lexicon; on real texts the ranking may change. - Published evidence points to the ability to extract and reason over financial text growing with model size; every claim is checked at its source (and its version: there are withdrawn papers). ## Resources ### Documentation - [ProsusAI/finbert model card on Hugging Face (CC-BY-NC-SA-4.0 license)](https://huggingface.co/ProsusAI/finbert) · huggingface.co ### Books - [Lopez-Lira, A. and Tang, Y. (2023-2025): Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models (arXiv:2304.07619)](https://arxiv.org/abs/2304.07619) · arxiv.org - [Huang, A. H., Wang, H. and Yang, Y. (2023): FinBERT: A Large Language Model for Extracting Information from Financial Text. Contemporary Accounting Research 40(2), 806-841](https://doi.org/10.1111/1911-3846.12832) · doi.org - [Loughran, T. and McDonald, B. (2011): When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. The Journal of Finance 66(1), 35-65](https://doi.org/10.1111/j.1540-6261.2010.01625.x) · doi.org - [Qwen Team (2024): Qwen2.5 Technical Report (arXiv:2412.15115)](https://arxiv.org/abs/2412.15115) · arxiv.org ### Data - [documentos6_extraccion_verdad.csv: guidance and KPIs of each document and call (synthetic data)](https://pythonparatrading.com/curso/datos/nivel-6/documentos6_extraccion_verdad.csv) · Download · documentos6_extraccion_verdad.csv - [llamadas6_transcripciones.parquet: 840 synthetic conference calls](https://pythonparatrading.com/curso/datos/nivel-6/llamadas6_transcripciones.parquet) · Download · llamadas6_transcripciones.parquet Previous lesson: [6.2.2 · Structured prompting: JSON you can validate](https://pythonparatrading.com/en/python-trading-course/level-6/structured-prompting-json-with-pydantic/) Next lesson: [6.2.4 · Lightweight fine-tuning for financial tasks: LoRA and QLoRA](https://pythonparatrading.com/en/python-trading-course/level-6/lightweight-fine-tuning-lora-qlora/)