Guidance, KPIs and tone: what an LLM extracts well.

What gets extracted and what it is for

On earnings day, a listed company publishes a lot of information in the form of text. This lesson deals with three pieces of known value:

• Sales guidance for the next quarter. Management knows the trend of its business better than the analyst consensus; in the course data, the distance between the midpoint of the guidance and the consensus anticipates part of the next quarter's sales surprise (README of the level 6 data).
• The operating KPI of the sector (stores, subscribers, production, order backlog), which helps read sales before they are published.
• Tone: that of the analysts' questions in the conference call, which in the data carries a planted drift of 2% per unit of tone over the following 20 sessions, and that of the news, which moves the price on the reaction day (lessons 6.1.1 and 6.1.3).

There are three families of methods: rules (regular expressions and dictionaries), discriminative models fine-tuned for a task (FinBERT) and generative LLMs. The question is not which one is “best”, but which one gets more right on each task, at what cost and with what risk of failing silently. The course data make it possible to measure it, because the answer for each document is published:

flowchart TB
  A[Filings and calls] --> B[Rules]
  A --> C[LLM]
  N[News] --> D[FinBERT]
  N --> C
  N --> E[Dictionary]
  B --> F[Against the truth]
  C --> F
  D --> F
  E --> F

All the texts are synthetic and come from templates (data README): the results say how each method behaves in a controlled case, not how much it would get right with real filings. The corpus mixes texts in English and in Spanish (SEC-style filings, CNMV-style announcements, CNMV being Spain's securities regulator, and calls in both languages); they are processed as they are, and the prompts given to the LLM are in Spanish too.

The ground truth and how to measure

documentos6_extraccion_verdad.csv says what each document (id_doc) and each call (id_llamada) contains: a guidance row with the period and the two ends in millions of USD, a kpi row with the value and its unit, and the literal sentence they come from.

import json
import re
import numpy as np
import pandas as pd
from bs4 import BeautifulSoup

fil = pd.read_parquet("filings6_documentos.parquet")
ll = pd.read_parquet("llamadas6_transcripciones.parquet")
verdad = pd.read_csv("documentos6_extraccion_verdad.csv")
verdad["fuente"] = np.where(verdad["id_doc"].notna(), "doc",
                            "call")
verdad["id"] = verdad["id_doc"].fillna(verdad["id_llamada"])
print(verdad.groupby(["fuente", "tipo"]).size().unstack())
verdad.loc[[0, 1, 6990], ["id", "tipo", "periodo", "bajo",
                          "alto", "valor", "unidad"]]

tipo    guidance   kpi
fuente
call         710   840
doc         3210  3780
           id      tipo periodo    bajo    alto   valor           unidad
0     D000007  guidance  2019T1  2440.0  2569.0     NaN  millones de USD
1     D000007       kpi  2018T4     NaN     NaN  1142.0          tiendas
6990   C00001  guidance  2019T1  2440.0  2569.0     NaN  millones de USD

The columns are tipo (type), periodo (period, written 2019T1 for Q1 2019), bajo and alto (low and high end), valor (value) and unidad (unit, such as “millones de USD” or “tiendas”, stores). An extraction is evaluated like a classifier, with numbers. With $E$ the extracted data points, $V$ the true ones and $E \cap V$ the extracted ones that match a true one:

“Matching” requires the same period and the same numbers within a 0.5% tolerance: a guidance of “$7.23 billion” is 7,230 million, and the ground truth, 7,228; the text has no more precision than that. Accuracy to the digit is measured separately. An extraction with high precision and low coverage loses information silently; one with high coverage and low precision feeds false data into the model.

Baseline: rules

The first tool, and often the last, is a regular expression. The ones in this cell were written by reading the sentences in the frase (sentence) column of the ground truth: there are five ways of writing a guidance and a few KPI patterns. Numbers are read according to the language (in Spanish, the dot separates thousands and the comma, decimals).

ORD = {"first": 1, "second": 2, "third": 3, "fourth": 4,
       "primer": 1, "segundo": 2, "tercer": 3, "cuarto": 4}
N = r"\d[\d.,]*\d|\d"
GUIA = [
    rf"range of \$(?P<a>{N}) million to \$(?P<b>{N}) million",
    rf"between \$(?P<a>{N}) (?P<u>million|billion) and "
    rf"\$(?P<b>{N})",
    rf"expected to be \$(?P<a>{N})-(?P<b>{N}) million",
    rf"entre (?P<a>{N}) y (?P<b>{N}) (?P<u>mil millones|millones)",
    rf"horquilla de (?P<a>{N})-(?P<b>{N}) millones",
]
TRIM = (r"(first|second|third|fourth|primer|segundo|tercer|cuarto)"
        r" (?:quarter|trimestre) (?:of |de )?(\d{4})")
KPI = rf"(?:at quarter end|al cierre del trimestre): (?P<v>{N})"

def numero(s, es):
    s = s.replace(".", "").replace(",", ".") if es else (
        s.replace(",", ""))
    return float(s)

def es_espanol(t):
    return bool(re.search(r"\b(trimestre|millones)\b", t))

def reglas(texto):
    """Guidance and KPI of a text, or None if not found."""
    es = es_espanol(texto)
    guia = None
    for patron in GUIA:
        m = re.search(patron, texto)
        if m:
            k = 1000 if m.groupdict().get("u") in (
                "billion", "mil millones") else 1
            q = re.search(TRIM, texto[max(0, m.start() - 120):
                                      m.end()])
            per = f"{q.group(2)}T{ORD[q.group(1)]}" if q else None
            guia = (per, round(numero(m["a"], es) * k, 4),
                    round(numero(m["b"], es) * k, 4))
            break
    m = re.search(KPI, texto)
    kpi = numero(m["v"], es) if m else None
    return guia, kpi

def plano(html):
    return BeautifulSoup(html, "html.parser").get_text(" ")

textos = pd.concat([
    pd.Series([plano(t) for t in fil["texto"]],
              index=fil["id_doc"]),
    ll.groupby("id_llamada")["texto"].apply(" ".join)])
extr = {i: reglas(t) for i, t in textos.items()}
print(len(extr), "texts;", reglas(textos["D000010"]))

4860 texts; (('2019T1', 19710.0, 20550.0), 441.0)

The evaluation function works for any method: it receives, per text, the extracted guidance (periodo, bajo, alto) and KPI (or None), and compares them with the ground truth.

G = verdad[verdad["tipo"] == "guidance"].set_index("id")
K = verdad[verdad["tipo"] == "kpi"].set_index("id")

def cerca(a, b, tol=0.005):
    return a is not None and abs(a / b - 1) < tol

def evaluar(extr, ids):
    filas = {}
    for i in ids:
        guia, kpi = extr[i]
        hay_g, hay_k = i in G.index, i in K.index
        imp_g = hay_g and guia is not None and (
            cerca(guia[1], G.loc[i, "bajo"])
            and cerca(guia[2], G.loc[i, "alto"]))
        ok_g = imp_g and guia[0] == G.loc[i, "periodo"]
        dig_g = ok_g and (round(guia[1]) == G.loc[i, "bajo"]
                          and round(guia[2]) == G.loc[i, "alto"])
        ok_k = hay_k and cerca(kpi, K.loc[i, "valor"])
        filas[i] = dict(g_ext=guia is not None, g_ver=hay_g,
                        g_imp=imp_g, g_ok=ok_g, g_dig=dig_g,
                        k_ext=kpi is not None, k_ver=hay_k,
                        k_ok=ok_k)
    d = pd.DataFrame(filas).T.astype(int).sum()
    return pd.Series({
        "guidance: precision": d.g_ok / max(d.g_ext, 1),
        "guidance: recall": d.g_ok / d.g_ver,
        "guidance: amounts": d.g_imp / d.g_ver,
        "guidance: to the digit": d.g_dig / max(d.g_ok, 1),
        "KPI: precision": d.k_ok / max(d.k_ext, 1),
        "KPI: recall": d.k_ok / d.k_ver}).round(3)

pd.DataFrame({"documents": evaluar(extr, fil["id_doc"]),
              "calls": evaluar(extr, ll["id_llamada"].unique())})

                        documents  calls
guidance: precision         1.000  1.000
guidance: recall            1.000  1.000
guidance: amounts           1.000  1.000
guidance: to the digit      0.735  0.765
KPI: precision              0.969  1.000
KPI: recall                 1.000  1.000

The rules find all the guidance in the 4,020 documents and the 840 calls, with the right period and amounts (precision and recall of 1.000). To the digit they get 73.5% right in documents and 76.5% in calls: the rest are guidance figures written in billions with two decimals (“$20.55 billion”, “2,57 mil millones”, that is, 2.57 billion), where the text does not have the precision of the ground truth: in D000010, 20,550 versus 20,545. No rule can recover what the text does not say; that is why the 0.5% tolerance is the fair measure and accuracy to the digit, a reminder of what is lost.

The only blemish is the KPI precision in documents, 0.969. The 120 “false positives” are the amendments (96 10-Q/A and 24 10-K/A): they repeat the KPI of the original document and the ground truth does not count them as documents with a KPI. It is not an error of the rule but of the definition of the ground truth, and it only shows up when you read the errors one by one. Before believing a metric, look at its failures.

A warning that weighs more than the table: these rules were written by looking at the sentences. The synthetic data use five guidance templates and a few KPI ones, and a rule written for them finds all of them. In real filings, each company words things its own way, changes its wording over the years and mixes figures across tables, notes and paragraphs; a rule needs continuous maintenance and a coverage measure that warns when it stops finding things.

The figure counts the guidance sentences by template, with figures and quarters replaced by placeholders: eight ways of writing them, four per language, and the “between … billion” one, in billions with two decimals, is the one that makes accuracy to the digit impossible.

The LLM on a sample

The model is the Qwen2.5-0.5B-Instruct from lessons 6.2.1 and 6.2.2, with a structured prompt that asks for the guidance and the KPI at the same time. An LLM of this size takes on the CPU about as long as a human takes to read, so the measurement uses a sample: 14 documents (8-K, 10-Q and CNMV-style announcements, trimmed to their text without tables) and 6 calls (management's turns in the presentation). The 20 requests are sorted by length before being grouped into batches: that way each batch carries less padding.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(NOMBRE)
tok.padding_side = "left"
llm = AutoModelForCausalLM.from_pretrained(
    NOMBRE, dtype=torch.float32).eval()

def recorte(html):
    t = re.sub(r"<table>.*?</table>", "", html, flags=re.S)
    t = "\n".join(l for l in t.splitlines()
                  if not l.startswith("|") and "ynthetic" not in l
                  and "intético" not in l)
    t = BeautifulSoup(t, "html.parser").get_text("\n")
    return re.sub(r"\n\s*\n+", "\n", t.replace("*", "")).strip()

items = fil["items"].fillna("").astype(str)
cand = fil[fil["tipo"].isin(["8-K", "10-Q", "OIR", "HR"])
           & ~items.str.contains("4.02")]
m_docs = cand.sample(14, random_state=5)
pres = ll[(ll["rol"] == "directivo")
          & (ll["seccion"] == "presentacion")]
pres = pres.groupby("id_llamada")["texto"].apply(" ".join)
m_llam = pres.sample(6, random_state=5)
muestra = pd.concat([
    pd.Series([recorte(t) for t in m_docs["texto"]],
              index=m_docs["id_doc"]), m_llam])

SISTEMA = (
    "Extrae del texto (1) la previsión de ventas del trimestre "
    "siguiente en millones de USD (1 billion = 1000) y (2) el KPI "
    "operativo al cierre del trimestre, tal como está escrito. "
    "Responde solo con JSON: {\"periodo\": \"AAAAT1-4\" o null, "
    "\"bajo\": number o null, \"alto\": number o null, "
    "\"kpi\": number o null}")

def generar(textos_usuario, sistema, max_tokens=48, lote=10):
    prompts = [tok.apply_chat_template(
        [{"role": "system", "content": sistema},
         {"role": "user", "content": t}],
        tokenize=False, add_generation_prompt=True)
        for t in textos_usuario]
    orden = np.argsort([len(p) for p in prompts])
    salida = [None] * len(prompts)
    for k in range(0, len(orden), lote):
        sel = orden[k:k + lote]
        b = tok([prompts[j] for j in sel], return_tensors="pt",
                padding=True)
        with torch.inference_mode():
            o = llm.generate(**b, do_sample=False,
                             max_new_tokens=max_tokens,
                             pad_token_id=tok.eos_token_id)
        n = b["input_ids"].shape[1]
        for j, fila in zip(sel, o):
            salida[j] = tok.decode(fila[n:],
                                   skip_special_tokens=True)
    return salida

resp = generar(list(muestra), SISTEMA)
print(len(resp), "responses;", resp[0].replace("`", "'")[:70])

20 responses; {"periodo": "AAAAT1-4", "bajo": null, "alto": 11743, "kpi": 173}

The system prompt says: “Extract from the text (1) next quarter's sales guidance in millions of USD (1 billion = 1000) and (2) the operating KPI at quarter end, as written. Answer only with JSON”, followed by the format, in which "AAAAT1-4" is the Spanish equivalent of YYYYQ1-4 (año, year; trimestre, quarter). The responses are read with the same tolerance as in lesson 6.2.2: first JSON object, numbers written as text converted and periods like 2024Q4 normalized to 2024T4. Anything that cannot be read counts as “not extracted”.

def a_numero(v):
    if isinstance(v, (int, float)):
        return float(v)
    if isinstance(v, str):
        s = re.sub(r"[^\d.]", "", v.replace(",", "."))
        return float(s) if s.count(".") <= 1 and s else None
    return None

def leer(texto):
    m = re.search(r"\{.*?\}", texto, flags=re.S)
    try:
        d = json.loads(m.group(0)) if m else {}
    except json.JSONDecodeError:
        d = {}
    q = re.search(r"(\d{4})\s*[TQ]\s*0?([1-4])",
                  str(d.get("periodo")))
    per = f"{q.group(1)}T{q.group(2)}" if q else None
    bajo, alto = a_numero(d.get("bajo")), a_numero(d.get("alto"))
    guia = (per, bajo, alto) if bajo and alto else None
    return guia, a_numero(d.get("kpi"))

extr_llm = {i: leer(r) for i, r in zip(muestra.index, resp)}
pd.DataFrame({"rules": evaluar(extr, muestra.index),
              "LLM 0.5B": evaluar(extr_llm, muestra.index)})

                        rules  LLM 0.5B
guidance: precision     1.000     0.000
guidance: recall        1.000     0.000
guidance: amounts       1.000     0.053
guidance: to the digit  0.737     0.000
KPI: precision          1.000     0.000
KPI: recall             1.000     0.000

The first response already shows the main problem: the model has literally copied the format placeholder, "periodo": "AAAAT1-4", instead of writing the quarter. Small models copy the examples and templates in the prompt easily; a placeholder like that is replaced by a concrete example or, better, by structured outputs (lesson 6.2.2).

In the sample, the rules get everything right and the LLM nothing: not one period, not one KPI, and the guidance amounts only in 1 of the 19 texts that contain it (5.3%). With the period copied from the placeholder, no complete guidance can be correct; that is why the table has a separate “amounts” row, which does not require the period, to see whether it at least reads the figures. It does not.

filas = []
for i in muestra.index[[0, 1, 2, 3, 14, 15]]:
    guia, kpi = extr_llm[i]
    filas.append({
        "id": i,
        "LLM guidance": guia,
        "truth": (tuple(G.loc[i, ["periodo", "bajo", "alto"]])
                  if i in G.index else None),
        "LLM KPI": kpi,
        "true KPI": K.loc[i, "valor"] if i in K.index else None})
pd.DataFrame(filas).set_index("id")

                   LLM guidance                       truth    LLM KPI  true KPI
id
D001138                    None  (2020T4, 11570.0, 12027.0)     173.00   1462.00
D001430                    None  (2021T2, 12093.0, 12472.0)   12093.00   1327.00
D002985                    None  (2024T1, 28516.0, 30074.0)       0.88    633.00
D002444                    None    (2023T2, 6587.0, 6986.0)  699000.00   3625.00
C00801   (None, 7297.7, 7437.0)    (2025T3, 7064.0, 7437.0)    1007.70     14.28
C00468                     None    (2022T4, 3753.0, 3948.0)        NaN   1253.00

The table shows six cases. In four documents the model returns no readable guidance and puts any figure as the KPI: 173, the low end of the guidance (12,093), 0.88 or 699,000, when the KPI is written as is in the text. In call C00801 it reads the high end (7,437), makes up the low one (7,297.7 instead of 7,064) and writes 1,007.7 as the KPI where the text says 14.28 million. These are not format errors: the model does not locate the requested figure.

Sentences the rule has not seen

The rules get it right because they know the templates. To see what happens with new wording, the next cell uses eight sentences written for this lesson, four in English and four in Spanish, with their answer, that follow none of the data templates; the last one contains no guidance.

nuevas = [
    ("We now see third-quarter 2024 sales landing somewhere "
     "around $1.2bn to $1.3bn.", ("2024T3", 1200, 1300)),
    ("Management guided Q2 2025 net sales to USD 845 to 870 "
     "million.", ("2025T2", 845, 870)),
    ("For Q1 2025 we expect revenue of $3.40 to $3.55 billion.",
     ("2025T1", 3400, 3550)),
    ("Q4 2024 outlook: top line between 12.5 and 13.1 billion "
     "dollars.", ("2024T4", 12500, 13100)),
    ("La dirección estima que la cifra de negocio del tercer "
     "trimestre de 2024 rondará los 950 a 1.000 millones de "
     "dólares.", ("2024T3", 950, 1000)),
    ("Esperamos ingresos de 2.100 millones de dólares como mínimo "
     "y de 2.250 millones como máximo en el cuarto trimestre de "
     "2023.", ("2023T4", 2100, 2250)),
    ("Prevemos unas ventas de 1,15 a 1,22 mil millones de dólares "
     "para el segundo trimestre de 2025.", ("2025T2", 1150, 1220)),
    ("No damos previsiones para el próximo trimestre.", None),
]
r_nuevas = generar([f for f, _ in nuevas], SISTEMA)

def importes(guia, esperado):
    if esperado is None:
        return guia is None
    return guia is not None and cerca(guia[1], esperado[1]) and (
        cerca(guia[2], esperado[2]))

def completo(guia, esperado):
    return importes(guia, esperado) and (
        esperado is None or guia[0] == esperado[0])

def si(x):
    return "yes" if x else "no"

for (frase, esp), r in zip(nuevas, r_nuevas):
    g_reg, g_llm = reglas(frase)[0], leer(r)[0]
    print(f"rules {si(completo(g_reg, esp))} · LLM "
          f"{si(completo(g_llm, esp))} (amounts "
          f"{si(importes(g_llm, esp))}) · {g_llm}")

rules no · LLM no (amounts no) · (None, 1.2, 1.3)
rules no · LLM no (amounts no) · (None, 839.0, 870.0)
rules no · LLM no (amounts no) · (None, 3.4, 3.55)
rules no · LLM no (amounts no) · (None, 12.5, 13.1)
rules no · LLM no (amounts yes) · (None, 950.0, 1000.0)
rules no · LLM no (amounts yes) · (None, 2100.0, 2250.0)
rules no · LLM no (amounts yes) · (None, 1150.0, 1220.0)
rules yes · LLM yes (amounts yes) · None

The Spanish sentences say, in order: management estimates that third-quarter 2024 revenue will be around 950 to 1,000 million dollars; we expect revenue of at least 2,100 and at most 2,250 million dollars in the fourth quarter of 2023; we forecast sales of 1.15 to 1.22 billion dollars for the second quarter of 2025; and we do not give guidance for next quarter. The rules find none of the seven new guidance sentences, because none follows their templates; they only get the eighth one right, which contains no guidance, by finding nothing. The LLM gets none completely right (it leaves the period as None again), but it reads the amounts of the three Spanish sentences correctly, including the one with “1,15 a 1,22 mil millones” (1.15 to 1.22 billion), which it converts to 1,150 and 1,220. In the English ones it gets the units wrong (1.2 and 1.3 instead of 1,200 and 1,300 million; 3.4 and 12.5 billion left unconverted) and in one it reads 839 where the text says 845.

That is the argument for LLMs in extraction: they generalize to wording nobody anticipated. A 0.5B model does it halfway; with a large one, unit conversion and the period are the kind of instruction it follows well, although it has to be measured just as here.

The figure summarizes the guidance extraction: on the template texts the rules get everything right and the LLM almost nothing; on new sentences, the rules find nothing and the LLM reads the amounts of three of the seven correctly.

The tone of the questions in conference calls

Classifying does not require generating text. With a prompt that asks for a one-word answer (positive, negative or neutral), a single forward pass is enough, looking at the probability the model gives to each of the three words as the next token. The difference $p_{pos} - p_{neg}$, normalized by the mass of the three, is a continuous score between −1 and 1; and the output cannot go off format, because it is read directly from the logits (the idea of constrained decoding, lesson 6.2.2).

ETIQUETAS = ["positive", "negative", "neutral"]
IDS = [tok(e)["input_ids"][0] for e in ETIQUETAS]
print([tok.decode([i]) for i in IDS])

def tono_llm(textos, sistema, lote=10):
    prompts = [tok.apply_chat_template(
        [{"role": "system", "content": sistema},
         {"role": "user", "content": t}],
        tokenize=False, add_generation_prompt=True)
        for t in textos]
    orden = np.argsort([len(p) for p in prompts])
    P = np.zeros((len(textos), 3))
    for k in range(0, len(orden), lote):
        sel = orden[k:k + lote]
        b = tok([prompts[j] for j in sel], return_tensors="pt",
                padding=True)
        with torch.inference_mode():
            z = llm(**b).logits[:, -1].double()
        P[sel] = torch.softmax(z, -1)[:, IDS].numpy()
    return (P[:, 0] - P[:, 1]) / P.sum(1), P.sum(1)

['positive', 'negative', 'neutral']

The three words are encoded as a single token each, so the probability of the first token of the answer is enough. The analysts' questions in each call are joined into one text. The dictionary is the course's fictitious financial lexicon (texto6_lexico.csv, with the Loughran and McDonald categories, lesson 6.1.1) and its score is $(pos - neg)/(pos + neg + 1)$:

lex = pd.read_csv("texto6_lexico.csv")
lex = lex[lex["lista"] == "financiera"]
POS = set(lex.loc[lex["categoria"] == "positiva", "palabra"])
NEG = set(lex.loc[lex["categoria"] == "negativa", "palabra"])

def diccionario(t):
    w = re.findall(r"[a-záéíóúñü]+", t.lower())
    p, n = sum(x in POS for x in w), sum(x in NEG for x in w)
    return (p - n) / (p + n + 1)

lv = pd.read_csv("llamadas6_verdad.csv").set_index("id_llamada")
preg = (ll[ll["rol"] == "analista"].groupby("id_llamada")
        ["texto"].apply(" ".join))
llam = lv[["tono_analistas"]].join(preg)
llam["dicc"] = llam["texto"].map(diccionario)
m_tono = llam.sample(20, random_state=0)
SIS_TONO = ("These are analysts' questions in an earnings call. Is "
            "their tone about the company positive, negative or "
            "neutral? Answer with one word.")
m_tono["llm"], masa = tono_llm(list(m_tono["texto"]), SIS_TONO)
print(f"mass of the three labels: {masa.mean():.3f}")
print("840 calls, dictionary:",
      round(llam[["dicc", "tono_analistas"]].corr().iloc[0, 1], 3))
m_tono[["llm", "dicc", "tono_analistas"]].corr().round(3)

mass of the three labels: 0.343
840 calls, dictionary: 0.887
                  llm   dicc  tono_analistas
llm             1.000  0.755           0.759
dicc            0.755  1.000           0.924
tono_analistas  0.759  0.924           1.000

The dictionary (dicc) recovers the analysts' tone (tono_analistas) with a correlation of 0.887 over the 840 calls and 0.924 in the sample of 20; the LLM, 0.759 in the same sample. The three words get only 34.3% of the next-token probability (the rest is spread among other possible starts of the answer), and the normalization in the formula corrects that scale.

The dictionary winning is not proof that dictionaries beat LLMs on conference calls: the data README warns that the questions reuse template phrases written with the words of the lexicon, so the dictionary has a much easier time than with real transcripts. On news, with phrases outside the lexicon and negations, the comparison is more realistic.

News sentiment: LLM, FinBERT and dictionary

FinBERT (ProsusAI/finbert) is a BERT fine-tuned on the Financial PhraseBank to classify sentences as positive, negative and neutral (Araci, 2019); its weights are published under a CC-BY-NC-SA-4.0 license, which does not allow commercial use (lesson 6.1.1). It only understands English, so the comparison uses original news in English: 40 for the three methods and 500 for FinBERT and the dictionary, which are cheap.

from transformers import pipeline

noti = pd.read_parquet("noticias6_corpus.parquet").merge(
    pd.read_csv("noticias6_verdad.csv"), on="id_noticia")
en = noti[(noti["idioma"] == "en")
          & (noti["tipo_noticia"] == "original")]
grande = en.sample(500, random_state=0)
txt = list(grande["titular"] + ". " + grande["cuerpo"])
verd = grande["sentimiento_verdadero"].to_numpy()

finbert = pipeline("text-classification", model="ProsusAI/finbert",
                   device="cpu")

def puntua_finbert(textos):
    out = finbert(textos, top_k=None, batch_size=32)
    d = [{x["label"]: x["score"] for x in o} for o in out]
    return np.array([x["positive"] - x["negative"] for x in d])

s_fb = puntua_finbert(txt)
s_dic = np.array([diccionario(t) for t in txt])
SIS_NOT = ("Classify the news for the company's shareholders. "
           "Answer with one word: positive, negative or neutral.")
s_llm, _ = tono_llm(txt[:40], SIS_NOT)

def clase(s, umbral=0.0):
    return np.sign(np.where(np.abs(s) <= umbral, 0, s))

filas = {}
for nombre, s, n in [("LLM 0.5B", s_llm, 40),
                     ("FinBERT", s_fb[:40], 40),
                     ("dictionary", s_dic[:40], 40),
                     ("FinBERT", s_fb, 500),
                     ("dictionary", s_dic, 500)]:
    filas[(nombre, n)] = {
        "correlation": np.corrcoef(s, verd[:n])[0, 1],
        "3-class accuracy": np.mean(
            clase(s, 0.2) == np.sign(verd[:n]))}
pd.DataFrame(filas).T.round(3)

                correlation  3-class accuracy
LLM 0.5B   40         0.868             0.725
FinBERT    40         0.917             0.875
dictionary 40         0.592             0.675
FinBERT    500        0.881             0.860
dictionary 500        0.702             0.712

On the 40 English news items, FinBERT has the highest correlation with the true sentiment (0.917) and gets the class right in 87.5% of cases; the 0.5B LLM comes close in correlation (0.868) and does worse on classes (72.5%); the dictionary is far behind (0.592 and 67.5%). The header figure brings together these correlations and those of the calls. With 500 news items, FinBERT gives 0.881 and the dictionary 0.702, in line with the 0.71 correlation the README measures on all the news. The classes use a 0.2 threshold on the score (below it, neutral).

The contrast with the extraction is the most useful lesson. The same 0.5B model that cannot locate a figure classifies the tone of a news item almost as well as a model fine-tuned for that. Classifying the tone of a short text is a task that pretraining covers well and that does not require following instructions precisely; locating a specific figure, converting its units and not confusing it with another one does. Model size weighs much more in the second.

What a larger model would change

This lesson cannot run a large model, so what follows are published results, with their source:

• Size matters on knowledge benchmarks. In the Qwen2.5 technical report, the 0.5B-parameter base model scores 47.5 on MMLU and the 72B one, 86.1 (tables 5 and 2); in the instruct version, MMLU-Pro goes from 15.0 to 71.1 (tables 6 and 10)11Note 1 · Qwen Team (2024): “Qwen2.5 Technical Report”, arXiv:2412.15115 (https://arxiv.org/abs/2412.15115)..
• And in finance. Lopez-Lira and Tang (version 6, October 2025) write in their abstract that “forecasting ability generally increases with model size”; in the body of the paper, the basic models (GPT-1, GPT-2, BERT) give accuracy below 65% when scoring headlines, and their scores do not correlate significantly with the subsequent return drift.
• Some prompting techniques only work at scale. Wei et al. (2022) observe that asking for step-by-step reasoning “only yields performance gains when used with models of ∼100B parameters”22Note 2 · Wei, J. et al. (2022): “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, arXiv:2201.11903 (https://arxiv.org/abs/2201.11903), section 3.2.: it makes no sense to try it with a 0.5B model and conclude that it does not work.
• A small, specialized model may be enough. Huang, Wang and Yang (2023) show that their FinBERT beats the Loughran-McDonald dictionary and other machine learning algorithms at sentiment classification: for a narrow task, a fine-tuned BERT-type model is a tough baseline.

With a commercial model, the prompting levers from lesson 6.2.2 change the extraction result: the full schema with structured outputs, worked examples in the prompt (above all for unit conversions), a text field for the model to copy the sentence it takes the guidance from (which also makes it possible to check it) and separating the search for the sentence from the reading of the numbers. What does not change is this lesson's method: measure against a ground truth, with precision, coverage and accuracy, before using the data point in a signal.

Exercises

1. With the guidance the rules extract from the documents, compute for each one the logarithm of the ratio between the midpoint and the sales consensus for that quarter (acciones6_ventas.csv) and its correlation with the sales surprise, $\log(\text{ventas}/\text{consenso})$ (sales over consensus). Does it match the one the README gives (0.18)?

ventas = pd.read_csv("acciones6_ventas.csv")
tk = pd.read_csv("acciones6_tickers.csv")
emp = pd.read_csv("acciones6_empresas.csv")
actual = (tk.merge(emp[["id_empresa", "ticker"]], on="id_empresa",
                   suffixes=("", "_actual"))
          .drop_duplicates("ticker")
          .set_index("ticker")["ticker_actual"])
tick_doc = fil.set_index("id_doc")["ticker"].map(actual)

filas = []
for i, (guia, _) in extr.items():
    if i in tick_doc.index and guia is not None and guia[0]:
        fin = pd.Period(guia[0].replace("T", "Q")).end_time
        filas.append({"ticker": tick_doc[i],
                      "fin_trimestre": fin.strftime("%Y-%m-%d"),
                      "medio": (guia[1] + guia[2]) / 2})
pg = (pd.DataFrame(filas)
      .drop_duplicates(["ticker", "fin_trimestre"])
      .merge(ventas, on=["ticker", "fin_trimestre"]))
x = np.log(pg["medio"] / pg["consenso_ventas"])
y = np.log(pg["ventas"] / pg["consenso_ventas"])
print(len(pg), "quarters · correlation",
      round(np.corrcoef(x, y)[0, 1], 3))

1713 quarters · correlation 0.177

2. Repeat the news comparison with the class threshold at 0.1 and at 0.4 for FinBERT and the dictionary on the 500 news items. Which method is more sensitive to the threshold, and why?

pd.DataFrame({u: {"FinBERT": np.mean(clase(s_fb, u)
                                     == np.sign(verd)),
                  "dictionary": np.mean(clase(s_dic, u)
                                        == np.sign(verd))}
              for u in [0.1, 0.2, 0.4]}).round(3)

              0.1    0.2    0.4
FinBERT     0.866  0.860  0.868
dictionary  0.712  0.712  0.678

3. Write, without running it, the prompt you would use with a commercial model to extract the operating KPI from the calls, and say how you would check each response without reading it by hand.