---
title: "6.1.1 Tokenization, embeddings, dictionaries and FinBERT"
description: "WordPiece and BPE tokenizers, MiniLM embeddings, Loughran-McDonald-style dictionaries and FinBERT, measured against the true sentiment."
url: "https://pythonparatrading.com/en/python-trading-course/level-6/tokenization-embeddings-loughran-mcdonald-finbert/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/tokenization-embeddings-loughran-mcdonald-finbert.png"
---
# 6.1.1 · Tokenization, embeddings, financial dictionaries and FinBERT
> A model does not read words: it reads numbers. This lesson walks through the three classic ways of turning a financial news item into a sentiment score (counting the words of a dictionary, projecting it into an embedding space and classifying it with a fine-tuned model such as FinBERT) and measures them on a synthetic corpus in which the true sentiment of every news item is known.
- Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/)
- Module: 6.1 · Classical and modern financial NLP
- Duration: 70 min
- Prerequisites: [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/), [5.2.2 · Point-in-time fundamentals, macro and alternative data](https://pythonparatrading.com/en/python-trading-course/level-5/point-in-time-fundamentals-macro-alternative-data/)
## By the end of this lesson
- Explain what a subword tokenizer does (WordPiece, byte-level BPE) and measure how many tokens the same text costs in English and in Spanish with three vocabularies.
- Score news with a financial dictionary and with a general one, and find their typical failures: words with a different meaning in finance, negations and phrases outside the lexicon.
- Compute sentence embeddings with MiniLM, compare them by cosine similarity and train a linear model on top of them with a temporal split.
- Run FinBERT locally, turn its probabilities into a score and compare it with the dictionary by language and by type of phrase, knowing its license.
## From text to a number
A strategy that reads news needs, in the end, a column of numbers: one score per news item that can be aggregated,
lagged and correlated with returns. Between the text and that column there are always the same steps: split the text
into units, turn those units into numbers and summarize them in a score.
```mermaid
flowchart TB
T[Text] --> K[Tokens]
K --> D[Lexicon count]
K --> E[Embedding]
K --> F[Fine-tuned model]
D --> S[Score]
E --> R[Linear model]
R --> S
F --> S
```
The three branches are the three families of this lesson: **dictionaries** (counting positive and negative words),
**embeddings** (one vector per sentence, with a model trained on top) and **fine-tuned models** such as FinBERT, which
read the sentence and directly return the probability of each class. Generative LLMs, which also score sentiment,
arrive in lesson 6.2.3.
The data are the synthetic news of level 6: 39,570 items from five fictitious agencies about 120 companies of the
level 5 panel, with their publication timestamp. The corpus mixes news in Spanish and in English, and the Spanish texts
are kept as they are, because the lesson measures precisely what each method does with each language. What does not
exist in a real corpus is in `noticias6_verdad.csv`: the **true sentiment** of each news item, between −1 and 1, with
which the generator wrote the text and moved the price. It is the known answer against which each method is measured.
```python
import re
import textwrap
import numpy as np
import pandas as pd
noticias = pd.read_parquet("noticias6_corpus.parquet")
verdad = pd.read_csv("noticias6_verdad.csv")
df = noticias.merge(verdad, on="id_noticia")
orig = df[df["tipo_noticia"] == "original"].copy()
orig["texto"] = orig["titular"] + ". " + orig["cuerpo"]
print(len(orig), "original news items")
print(orig["idioma"].value_counts().to_dict())
fila = orig.iloc[2]
print(fila["texto"])
print("true sentiment:", fila["sentimiento_verdadero"])
```
```output
22593 original news items
{'es': 16969, 'en': 5624}
Placeholder & Co. cuts Numvia Communications price target to $19.23. Analysts at Placeholder & Co. cut their price target on Numvia Communications to $19.23. Numvia Communications issued a warning about business weakness.
true sentiment: -0.546
```
Syndicated copies (the same story at another agency, sometimes translated) and market wrap-ups are left out: a story
is scored once. How to detect the copies is part of lesson 6.1.2.
## Tokenizing: words and subwords
The simplest way to split a text is a regular expression that extracts words. It is what dictionaries use: convert to
lowercase and keep sequences of letters. NLTK adds rules for punctuation and contractions:
```python
from nltk.tokenize import word_tokenize
frase = "Numvia didn't beat estimates; revenue fell 3.5%."
def palabras(texto):
return re.findall(r"[a-záéíóúüñ]+", texto.lower())
print(palabras(frase))
print(word_tokenize(frase))
```
```output
['numvia', 'didn', 't', 'beat', 'estimates', 'revenue', 'fell']
['Numvia', 'did', "n't", 'beat', 'estimates', ';', 'revenue', 'fell', '3.5', '%', '.']
```
The regular expression loses the negation (“didn't” ends up as `didn` and `t`) and the figures; `word_tokenize`
separates `did` and `n't` and keeps `3.5` and `%`. For a dictionary the first one is enough; for a model neither works,
because a vocabulary of whole words is never complete: company names, figures and rare words would be left out.
Current models use **subwords**: a fixed vocabulary (from 30,000 to 150,000 pieces) learned on a corpus, with which any
text is broken down into known fragments. The two families that matter here:
| Algorithm | How it builds the vocabulary | Models in this lesson |
|---|---|---|
| WordPiece | Merges the pairs of pieces that most increase the likelihood of the corpus; marks continuations with `##` | FinBERT and MiniLM (BERT vocabulary, 30,522 pieces, lowercase and without accents) |
| Byte-level BPE | Starts from the 256 bytes and merges the most frequent pairs; no text falls outside the vocabulary | Qwen2.5 (about 151,000 pieces), tiktoken's `cl100k_base` |
BPE was proposed by Sennrich, Haddow and Birch (2016) for machine translation; the Hugging Face page on tokenization
algorithms explains the variants[^1]. What matters for a quant is what each vocabulary does with financial texts in two
languages:
```python
import tiktoken
from transformers import AutoTokenizer
tok_bert = AutoTokenizer.from_pretrained("ProsusAI/finbert")
tok_qwen = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tok_tik = tiktoken.get_encoding("cl100k_base")
en = "Revenue was $3,075.9 million, a weak quarter."
es = "Las ventas sumaron 3.075,9 millones: caída de la demanda."
def piezas(texto):
q = [tok_qwen.convert_tokens_to_string([t])
for t in tok_qwen.tokenize(texto)]
t = [tok_tik.decode([i]) for i in tok_tik.encode(texto)]
return {"wordpiece": tok_bert.tokenize(texto),
"qwen": q, "cl100k": t}
for texto in (en, es):
for nombre, p in piezas(texto).items():
print(f"{nombre:9} {len(p):2}", "|".join(p))
print()
```
```output
wordpiece 15 revenue|was|$|3|,|07|##5|.|9|million|,|a|weak|quarter|.
qwen 16 Revenue| was| $|3|,|0|7|5|.|9| million|,| a| weak| quarter|.
cl100k 14 Revenue| was| $|3|,|075|.|9| million|,| a| weak| quarter|.
wordpiece 22 las|vent|##as|sum|##aro|##n|3|.|07|##5|,|9|mill|##ones|:|cai|##da|de|la|demand|##a|.
qwen 21 Las| ventas| sum|aron| |3|.|0|7|5|,|9| millones|:| ca|ída| de| la| demand|a|.
cl100k 19 Las| ventas| sum|aron| |3|.|075|,|9| millones|:| ca|ída| de| la| demand|a|.
```
The Spanish sentence means “sales totaled 3,075.9 million: a fall in demand”. Three details with consequences:
- **Figures.** WordPiece splits `3,075.9` into five pieces and Qwen writes each digit separately. No language model
“sees” the number 3,075.9 as a magnitude; it sees a sequence of symbols. That is why extracting figures with an LLM
needs validation (lesson 6.2.2).
- **Accents and lowercase.** The uncased BERT vocabulary strips accents: “caída” (fall) becomes `cai` + `##da`.
- **False friends.** “demanda” (demand) becomes `demand` + `##a`: an English piece with a different meaning. A model
trained on English text will associate that piece with what *demand* means in English.
The cost also changes with the language. Measured on 400 news items of each language, in tokens per word:
```python
muestra = (orig.groupby("idioma", group_keys=False)
.sample(400, random_state=0))
n_palabras = muestra["texto"].str.split().str.len()
def n_tokens(textos, nombre):
if nombre == "cl100k":
return np.array([len(tok_tik.encode(t)) for t in textos])
tok = tok_bert if nombre == "wordpiece" else tok_qwen
return np.array([len(tok.tokenize(t)) for t in textos])
coste = {}
for nombre in ("wordpiece", "qwen", "cl100k"):
ratio = n_tokens(muestra["texto"], nombre) / n_palabras
coste[nombre] = ratio.groupby(muestra["idioma"]).mean()
pd.DataFrame(coste).round(2)
```
```output
wordpiece qwen cl100k
idioma
en 1.47 1.52 1.44
es 2.03 1.74 1.75
```
*Fig. — Tokens per word in the corpus news by tokenizer and language: BERT's vocabulary, built from English text, splits Spanish much more.* Six horizontal bars. BERT WordPiece: 1.47 tokens per word in English and 2.03 in Spanish (highlighted). Qwen2.5 BPE: 1.52 and 1.74. cl100k_base: 1.44 and 1.75.
Spanish costs more tokens than English with all three vocabularies, and the gap is larger with BERT's, which was built
only from English text. It matters for three reasons: commercial APIs charge per token (lesson 6.2.1); BERT-type models
truncate the input to 512 tokens (FinBERT) or to 256 pieces (MiniLM, according to its model card[^2]), and a long
document, such as the risk section of a 10-K, does not fit whole; and a vocabulary that chops a language into many
pieces usually indicates that the model saw little text in it.
## Dictionaries: from Harvard to Loughran-McDonald
A sentiment dictionary is a list of words with their category. The most common score counts the positive ($p$) and the
negative ($n$) words of a text:
$$
\text{tone} = \frac{p - n}{p + n + 1}
$$
The 1 in the denominator avoids dividing by zero and pulls texts with few tone words toward zero. The result lies
between −1 and 1.
For years general-purpose lists were used, such as the Harvard dictionary (General Inquirer). Loughran and McDonald
(2011) showed that in annual reports (10-K) almost three-fourths of the occurrences of “negative” words from that list
correspond to words that are not negative in a financial context: *tax*, *cost*, *liability*, *capital*, *vice* (from
*vice president*)[^3]. They built their own domain lists, which they maintain and update on their website with seven
categories: negative, positive, uncertainty, litigious, strong modal, weak modal and constraining. The lists are free
for academic research and their commercial use requires a license from the authors[^4]. They are not copied here.
Instead, the course ships `texto6_lexico.csv`, an **invented** lexicon with five of those categories in Spanish and in
English (list `financiera`, financial) and a `general` list that imitates the flaw of general-purpose lists:
```python
lex = pd.read_csv("texto6_lexico.csv")
print(lex.pivot_table(index=["lista", "categoria"], columns="idioma",
values="palabra", aggfunc="size",
fill_value=0))
gen_neg = lex.query("lista == 'general' and categoria == 'negativa'"
" and idioma == 'es'")
print(textwrap.fill(" ".join(gen_neg["palabra"]), 72))
```
```output
idioma en es
lista categoria
financiera incertidumbre 15 17
litigio 12 14
negativa 26 31
positiva 23 31
restrictiva 9 9
general negativa 17 17
positiva 9 10
cargo cargos caída coste costes crudo cáncer demanda deuda débil
impuesto impuestos pasivo pasivos pérdida quiebra vicepresidente
```
The categories are `incertidumbre` (uncertainty), `litigio` (litigious), `negativa` (negative), `positiva` (positive)
and `restrictiva` (constraining). The general list marks as negative *coste* (cost), *deuda* (debt), *impuesto* (tax),
*pasivo* (liability), *crudo* (crude), *vicepresidente* (vice president) and *demanda* (demand, but also “lawsuit”):
the same error that Loughran and McDonald found. Score all the news with the two lists and compare them with the true
sentiment:
```python
listas = {}
for (lista, idioma), g in lex.groupby(["lista", "idioma"]):
pos = set(g.loc[g["categoria"] == "positiva", "palabra"])
neg = set(g.loc[g["categoria"] == "negativa", "palabra"])
listas[(lista, idioma)] = (pos, neg)
def tono(texto, idioma, lista="financiera"):
pos, neg = listas[(lista, idioma)]
w = palabras(texto)
p = sum(x in pos for x in w)
n = sum(x in neg for x in w)
return (p - n) / (p + n + 1)
orig["dic_fin"] = [tono(t, i) for t, i in
zip(orig["texto"], orig["idioma"])]
orig["dic_gen"] = [tono(t, i, "general") for t, i in
zip(orig["texto"], orig["idioma"])]
s = orig["sentimiento_verdadero"]
print("financial:", round(orig["dic_fin"].corr(s), 3))
print("general: ", round(orig["dic_gen"].corr(s), 3))
print("neutral news with general tone < 0: "
f"{(orig.loc[s == 0, 'dic_gen'] < 0).mean():.1%}")
```
```output
financial: 0.709
general: 0.351
neutral news with general tone < 0: 7.0%
```
The financial list correlates 0.71 with the true sentiment; the general one, half that (0.35). The last line points to
a cause: 7% of the **neutral** news items (sentiment exactly zero, such as a shareholder meeting or a conference) get a
negative tone from the general list. Which words trigger those false negatives?
```python
_, neg_gen = listas[("general", "es")]
neutras = orig.loc[(s == 0) & (orig["idioma"] == "es"), "texto"]
cuenta = pd.Series([w for t in neutras for w in palabras(t)
if w in neg_gen]).value_counts()
cuenta.head(6)
```
```output
demanda 426
caída 52
Name: count, dtype: int64
```
Almost all of them come from “demanda”, in sentences such as “el impacto final dependerá de la demanda” (the final
impact will depend on demand): customer demand, not a lawsuit. The financial list classifies it as litigious (a
category that does not enter the tone) and the general one, as negative. It is the same kind of error that Loughran and
McDonald found with *liability* or *tax*: a word that is bad news in another context and descriptive in the financial
one.
## What a dictionary cannot see
Counting words ignores order. Two types of phrase break it: **negations** (“la compañía descarta pérdidas”, the company
rules out losses, contains *pérdidas*, losses) and tone phrases **without lexicon words** (“es un jarro de agua fría
para los accionistas”, it is a bucket of cold water for shareholders). The generator records in `noticias6_verdad.csv`
how many phrases of each type each news item contains, so it is possible to measure where the dictionary fails:
```python
grupo = np.select(
[orig["frases_negacion"] > 0, orig["frases_fuera_lexico"] > 0],
["with negation", "off-lexicon"], "lexicon only")
orig["grupo"] = grupo
por_grupo = orig.groupby("grupo").apply(
lambda g: pd.Series({
"noticias": len(g),
"corr_dic": g["dic_fin"].corr(g["sentimiento_verdadero"]),
}), include_groups=False)
por_grupo.round(3)
```
```output
noticias corr_dic
grupo
lexicon only 15921.0 0.784
off-lexicon 3718.0 0.833
with negation 2954.0 0.210
```
In news with a negation, the correlation drops to 0.21, a quarter of that of the rest: the dictionary reads the negative
word and gets the sign wrong. Phrases outside the lexicon, on the other hand, do not lower the correlation: in the
generator, the more intense news items carry more tone phrases (two on average, against fewer than one), and the other
phrases do use the lexicon. An example of negation:
```python
ej = orig[(orig["frases_negacion"] > 0)
& (orig["idioma"] == "es")
& (orig["sentimiento_verdadero"] > 0.4)
& (orig["dic_fin"] < 0)].iloc[0]
print(textwrap.fill(ej["cuerpo"], 72))
print("true:", ej["sentimiento_verdadero"],
"| dictionary:", round(ej["dic_fin"], 3))
```
```output
Kogsa Bancorp ha firmado un contrato de 829,7 millones de dólares con la
administración de un estado de EE. UU. para los próximos 3 años. El
grupo deja atrás las dificultades del año pasado. Las acciones cerraron
la última sesión en 201,85 dólares.
true: 0.7784 | dictionary: -0.5
```
“Deja atrás las dificultades” (leaves the difficulties behind) is good news written with a negative word from the
lexicon (*dificultades*, difficulties). The dictionary counts it as negative and the news item, clearly positive (a
contract worth 829.7 million dollars), scores −0.5.
Dictionaries still have three virtues that keep them in use in research and in production: they are **transparent**
(you know why a news item scores what it scores), **cheap** (millions of documents in seconds, without a GPU) and
**stable** (the same list gives the same score today and ten years from now, without look-ahead bias; lesson 6.2.5).
The usual improvements (negation rules within a window of a few words, tf-idf weighting, lists by sector) reduce their
errors without changing their nature.
## Embeddings: a sentence as a vector
A sentence **embedding** is a fixed-dimension vector that summarizes its meaning: similar sentences end up close
together and closeness is measured with cosine similarity (the dot product of the normalized vectors).
`all-MiniLM-L6-v2` is a small sentence-transformers model (22 million parameters, 384 dimensions, trained on English
text[^2]) that follows the idea of Sentence-BERT[^5]. Its weights are in the cache of the `ia` environment, pinned by
commit:
```python
import torch
from sentence_transformers import SentenceTransformer
torch.manual_seed(0)
st = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2",
device="cpu", local_files_only=True)
frases = [
"The company reported record profits.",
"Earnings hit an all-time high.",
"The company warned of heavy losses.",
"La empresa obtuvo beneficios récord.",
"La empresa advirtió de fuertes pérdidas.",
]
E = st.encode(frases, normalize_embeddings=True,
show_progress_bar=False)
print(E.shape)
pd.DataFrame(E @ E.T, index=range(5), columns=range(5)).round(2)
```
```output
(5, 384)
0 1 2 3 4
0 1.00 0.53 0.42 0.41 0.18
1 0.53 1.00 0.27 0.21 0.07
2 0.42 0.27 1.00 0.14 0.15
3 0.41 0.21 0.14 1.00 0.68
4 0.18 0.07 0.15 0.68 1.00
```
Sentences 0 and 1 say the same thing with different words and their similarity is the highest among the English ones
(0.53); sentence 2 shares almost all its words with sentence 0 and says the opposite, but its similarity is not much
lower (0.42). A general-purpose embedding mostly measures **what** a sentence is about, not **in which direction**. In
Spanish it is worse: sentences 3 and 4 (the Spanish versions of 0 and 2), which say opposite things, are the most
similar pair in the table (0.68), and each one is far from its English translation (0.41 and 0.15). MiniLM is not
multilingual: with Spanish text it mostly compares the form.
Turning embeddings into sentiment requires a model on top trained with labeled examples. In practice, the labels would
come from an annotated set (such as Financial PhraseBank, the one FinBERT used) or from the price reaction; here, to
measure the ceiling of the method, the true sentiment from 2019 to 2022 is used and evaluation runs from 2023 onward,
with the temporal split of level 5 (lesson 5.4.3). A sample of 1,200 news items per language is enough for the
comparison and keeps the cell at a few seconds on CPU:
```python
from sklearn.linear_model import Ridge
m = (orig.groupby("idioma", group_keys=False)
.sample(1200, random_state=1).sort_values("ts"))
X = st.encode(m["texto"].tolist(), batch_size=64,
normalize_embeddings=True, show_progress_bar=False)
y = m["sentimiento_verdadero"].to_numpy()
entreno = (m["ts"] < "2023-01-01").to_numpy()
m["ridge"] = np.nan
for idioma in ("en", "es"):
i = (m["idioma"] == idioma).to_numpy()
modelo = Ridge(alpha=1.0).fit(X[i & entreno], y[i & entreno])
prueba = i & ~entreno
m.loc[m.index[prueba], "ridge"] = modelo.predict(X[prueba])
test = m[m["ts"] >= "2023-01-01"]
test.groupby("idioma").apply(lambda g: pd.Series({
"prueba": len(g),
"corr_ridge": g["ridge"].corr(g["sentimiento_verdadero"]),
"corr_dic": g["dic_fin"].corr(g["sentimiento_verdadero"]),
}), include_groups=False).round(3)
```
```output
prueba corr_ridge corr_dic
idioma
en 524.0 0.919 0.702
es 491.0 0.826 0.729
```
With a few hundred examples per language, the regression on embeddings beats the dictionary in both languages, and by
more in English, MiniLM's language, than in Spanish. The figure must be read with care: the linear model has learned the
generator's **templates** with perfect labels, and those of 2023 are the same as those of 2019. With real news, a few
hundred noisy labels do not get that far, and language changes over the years. The lasting advantage of embeddings lies
elsewhere: the same vector serves to search for similar documents, group news by topic (lesson 6.1.3) or feed a RAG
(lesson 6.3.1).
## FinBERT: a classifier fine-tuned for finance
FinBERT (Araci, 2019)[^6] is a base-size BERT (110 million parameters) pre-trained on financial text and fine-tuned to
classify sentences into three classes (positive, negative, neutral) with Financial PhraseBank, a set of English
financial news sentences annotated by hand. The `ProsusAI/finbert` version is the most widely used.
```mermaid
flowchart LR
F[Sentence] --> W[WordPiece]
W --> B[BERT base]
B --> C[Class layer]
C --> P[Probabilities]
```
The output is three probabilities that add up to one. A common continuous score is
$P(\text{positive}) - P(\text{negative})$, between −1 and 1, comparable with the dictionary tone:
```python
from transformers import pipeline
finbert = pipeline("text-classification", model="ProsusAI/finbert",
device="cpu")
def puntua(textos, lote=32):
salidas = finbert(list(textos), batch_size=lote, top_k=None,
truncation=True)
filas = [{d["label"]: d["score"] for d in s} for s in salidas]
probs = pd.DataFrame(filas)[["positive", "negative", "neutral"]]
return probs.assign(fb=probs["positive"] - probs["negative"])
pruebas = ["The company does not expect any losses this year.",
"Revenue declined sharply and the outlook was cut.",
"The annual meeting will be held in Madrid."]
puntua(pruebas).round(3)
```
```output
positive negative neutral fb
0 0.058 0.345 0.596 -0.287
1 0.008 0.974 0.019 -0.966
2 0.027 0.026 0.947 0.001
```
The second sentence is negative beyond doubt (0.97) and the third, neutral (0.95). The first is good news written with
a negation, and FinBERT does not get it right either: it leaves it between neutral (0.60) and negative (0.35). A single
sentence proves nothing; it has to be measured over many. The news items of the test period from the previous cell are
used: the 524 English ones and 200 Spanish ones at random (FinBERT is the slowest step of the lesson: a few seconds per
hundred news items on CPU), so that the four methods are compared on the same news:
```python
sub_en = test[test["idioma"] == "en"].copy()
sub_es = test[test["idioma"] == "es"].sample(200, random_state=2)
sub_en["fb"] = puntua(sub_en["texto"])["fb"].to_numpy()
sub_es["fb"] = puntua(sub_es["texto"])["fb"].to_numpy()
def correlaciones(g):
s = g["sentimiento_verdadero"]
return pd.Series({"noticias": len(g),
"finbert": g["fb"].corr(s),
"ridge": g["ridge"].corr(s),
"dic_fin": g["dic_fin"].corr(s),
"dic_gen": g["dic_gen"].corr(s)})
pd.DataFrame({"en": correlaciones(sub_en),
"es": correlaciones(sub_es)}).T.round(3)
```
```output
noticias finbert ridge dic_fin dic_gen
en 524.0 0.876 0.919 0.702 0.459
es 200.0 -0.114 0.857 0.760 0.473
```
In English, FinBERT (0.88) clearly beats the financial dictionary (0.70) and the general list (0.46) without having
seen a single news item from the corpus. Only the ridge on embeddings (0.92) beats it, trained with the perfect labels
of the same templates: an advantage that does not exist with real data. In Spanish, FinBERT gives −0.11: the model does
not know the language and its tokenizer chops it into meaningless fragments, while the Spanish dictionary and the ridge
keep their level. The header figure summarizes the four methods on the 524 English news items. Where does FinBERT win?
Where the dictionary fails:
```python
sub_en.groupby("grupo").apply(
lambda g: pd.Series({
"noticias": len(g),
"finbert": g["fb"].corr(g["sentimiento_verdadero"]),
"dic_fin": g["dic_fin"].corr(g["sentimiento_verdadero"]),
}), include_groups=False).round(3)
```
```output
noticias finbert dic_fin
grupo
lexicon only 361.0 0.879 0.790
off-lexicon 91.0 0.886 0.841
with negation 72.0 0.870 0.100
```
*Fig. — Correlation with true sentiment by type of news (English test news): the dictionary collapses with negations and FinBERT does not.* Six horizontal bars. News using only lexicon words (361): dictionary 0.79, FinBERT 0.88. With off-lexicon phrases (91): 0.84 and 0.89. With negation (72): dictionary 0.10 (highlighted) and FinBERT 0.87.
In the news that only uses the lexicon, the dictionary (0.79) comes close to FinBERT (0.88): counting words is enough
when the text contains them. In those with a negation, the dictionary collapses (0.10) and FinBERT keeps its level
(0.87). That it failed with the single sentence of the previous cell and gets the group right is not a contradiction:
in the news the negation comes with the rest of the text, and what is measured is the correlation over many items.
With 524 news items, the small groups have a noticeable sampling error (with about 70 news items, a correlation of 0.1
has a standard error of a little more than 0.1), so it is better to read the ranking and not the decimals.
## What to choose and what it costs
| Method | Cost | Languages | Transparency | Main risk |
|---|---|---|---|---|
| Financial dictionary | Almost zero | That of the list | Full | Negations, context, new words |
| General dictionary | Almost zero | That of the list | Full | Accounting vocabulary read as negative |
| Embeddings + linear model | Low; needs labels | Those of the embedding model | Low | Expensive or noisy labels; language drift |
| FinBERT | Medium (CPU or GPU) | English | Low | Non-commercial license; 512 tokens; training domain |
| Generative LLM (6.2.3) | High | Many | Low | Cost, format, look-ahead bias (6.2.5) |
Three warnings to take into a backtest:
- **The score is not the signal.** A well-measured sentiment is only worth something if it moves the price after you can
trade. Measuring that reaction requires aligning each news item with the right session: that is lesson 6.1.3, where
the same dictionary recovers about 75 bp per unit of tone on the reaction day against 119 bp with the true sentiment
(level 6 README).
- **A fine-tuned model also has a date.** FinBERT was trained on sentences from before 2019; a classifier trained on
data after the backtest period introduces information from the future (lesson 6.2.5).
- **Language is a design decision.** With a European universe, half of the news can be in a language the model does
not understand. In this corpus, applying FinBERT to everything would have spoiled the signal of 75% of the news.
## Exercises
1. Add a negation rule to the dictionary: if one of the three words before a lexicon word is “no” (no), “descarta”
(rules out), “sin” (without) or “niega” (denies), flip its sign. Measure the correlation with the true sentiment on
the Spanish news with negation before and after.
**Solution.**
```python
NEG = {"no", "descarta", "sin", "niega"}
def tono_neg(texto, idioma="es"):
pos, neg = listas[("financiera", idioma)]
w = palabras(texto)
p = n = 0
for k, x in enumerate(w):
signo = -1 if NEG & set(w[max(0, k - 3):k]) else 1
if x in pos:
p, n = (p + 1, n) if signo > 0 else (p, n + 1)
elif x in neg:
p, n = (p, n + 1) if signo > 0 else (p + 1, n)
return (p - n) / (p + n + 1)
g = orig[(orig["idioma"] == "es") & (orig["grupo"] == "with negation")]
nuevo = g["texto"].map(tono_neg)
s_g = g["sentimiento_verdadero"]
print("before:", round(g["dic_fin"].corr(s_g), 3))
print("after: ", round(nuevo.corr(s_g), 3))
```
```output
before: 0.21
after: 0.768
```
A three-word rule recovers a good part of the signal lost in the news with negation. It is the classic improvement of
dictionaries; its limit is that you have to know the ways of negating in advance, and in a real text there are many
more than four.
2. Compute how many original English news items exceed FinBERT's 512 tokens and how many exceed MiniLM's 256. Do the
same with the 10-Ks in `filings6_documentos.parquet`. What would you do with the documents that do not fit?
**Solution.**
```python
from transformers.utils import logging as hf_logging
hf_logging.set_verbosity_error() # no “> 512 tokens” warning
en_txt = orig.loc[orig["idioma"] == "en", "texto"].tolist()
n = np.array([len(x) for x in tok_bert(en_txt)["input_ids"]])
print(f"news > 256: {(n > 256).mean():.1%}",
f"> 512: {(n > 512).mean():.1%}")
docs = pd.read_parquet("filings6_documentos.parquet")
diez_k = docs.loc[docs["tipo"] == "10-K", "texto"].tolist()
n_k = np.array([len(x) for x in tok_bert(diez_k)["input_ids"]])
print(f"10-K > 512: {(n_k > 512).mean():.1%}",
f"median {np.median(n_k):.0f}")
```
```output
news > 256: 0.0% > 512: 0.0%
10-K > 512: 100.0% median 898
```
The news items all fit; the 10-Ks do not (the count includes the HTML tags, and these are short synthetic documents: a
real one has tens of thousands of words). When tokenizing without truncation, transformers warns in its log that the
sequence exceeds the model's maximum length; the cell lowers that log to errors so that the output is clean. With
`truncation=True` the model would only read the beginning. The right approach is to split the document into chunks that
fit (by sentences or by sections, lesson 6.3.2), score each chunk and aggregate (mean, or mean weighted by confidence),
or score only the section of interest, such as the risk factors.
3. A vendor offers a “news sentiment” computed with a model fine-tuned on news from 2015 to 2024 and a history going
back to 2010. What questions would you ask before using it in a backtest?
**Solution.**
With what data and as of what date the model that scored each news item was trained (if the 2016 score was computed
with a model trained up to 2024, the history carries information from the future: lesson 6.2.5); whether the history
was recalculated when the model changed or what was delivered at the time is preserved (point-in-time, lesson 6.7.4);
what languages and sources it covers and how it handles syndicated copies; what timestamp each score carries (the
publication of the news item or the vendor's computation); and what license the model and the texts have.
[^1]: Sennrich, Haddow and Birch (2016), “Neural Machine Translation of Rare Words with Subword Units”
(); Hugging Face, “Tokenization algorithms”: BPE, byte-level BPE (GPT-2),
WordPiece (BERT family) and Unigram/SentencePiece (),
accessed on October 23, 2026.
[^2]: Model card of `sentence-transformers/all-MiniLM-L6-v2`
(, accessed on October 23, 2026): Apache-2.0
license, 384-dimensional vectors, English, and “by default, input text longer than 256 word pieces is truncated”.
[^3]: Loughran and McDonald (2011), Journal of Finance 66(1), 35-65 ():
“almost three-fourths of the words identified as negative by the widely used Harvard Dictionary are words
typically not considered negative in financial contexts”. In the body of the article the figure is 73.8% of the
occurrences of negative words.
[^4]: University of Notre Dame, Software Repository for Accounting and Finance, “Loughran-McDonald Master Dictionary
w/ Sentiment Word Lists” (, accessed on October 23, 2026):
free for academic research, commercial license upon request to the authors.
[^5]: Reimers and Gurevych (2019), “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”
().
[^6]: Araci (2019), “FinBERT: Financial Sentiment Analysis with Pre-trained Language Models”
(); model card of `ProsusAI/finbert` (,
accessed on October 23, 2026): CC-BY-NC-SA-4.0 license, three labels and fine-tuned with Financial PhraseBank.
## Self-assessment
1. FinBERT's tokenizer (WordPiece, lowercase and without accents) turns “caída” (fall) into `cai`, `##da` and “demanda” (demand) into `demand`, `##a`. What is the practical consequence?
- A. None: the model understands Spanish just as well
- B. Spanish text is cut into many more pieces, some of them match English words with a different meaning, and the model, trained on English sentences, barely interprets it
- C. The text takes fewer tokens than in English
- D. Accents are kept in a separate token
*Answer B: Spanish text is cut into many more pieces, some of them match English words with a different meaning, and the model, trained on English sentences, barely interprets it.* BERT's vocabulary was built from English text: Spanish words are split into fragments and the accent disappears. In the lesson, FinBERT on Spanish news does not correlate with the true sentiment (−0.11), while in English it reaches 0.88.
2. A general dictionary marks “coste” (cost), “deuda” (debt), “impuesto” (tax) or “vicepresidente” (vice president) as negative. What problem did Loughran and McDonald document with this kind of list in 10-Ks?
- A. That they have few positive words
- B. That they do not distinguish upper case
- C. That almost three-fourths of the occurrences of “negative” words from the Harvard dictionary in 10-Ks are words that are not negative in finance
- D. That they do not work with long texts
*Answer C: That almost three-fourths of the occurrences of “negative” words from the Harvard dictionary in 10-Ks are words that are not negative in finance.* A financial report talks about costs, debts and taxes without that being bad news. That is why they built their own domain lists. In the course corpus, the general list correlates 0.35 with the true sentiment and the financial one, 0.71.
3. The sentence “la compañía descarta pérdidas” (the company rules out losses) contains a negative word from the lexicon. What happens to a dictionary and what to FinBERT?
- A. The dictionary counts “pérdidas” (losses) as negative and gets the sign wrong; a model that reads the whole sentence can capture the negation
- B. Both get it right
- C. Both always fail
- D. The dictionary gets it right because “descarta” (rules out) is positive
*Answer A: The dictionary counts “pérdidas” (losses) as negative and gets the sign wrong; a model that reads the whole sentence can capture the negation.* Counting words ignores order and context. In news with a negation, the dictionary's correlation with the true sentiment drops to 0.21; FinBERT, which reads the sentence as a sequence, keeps almost all the signal in English news with a negation (0.87).
4. A ridge regression is trained on MiniLM embeddings with the sentiment of 2019-2022 and evaluated on 2023-2025. Why that split and not a random one?
- A. Because it is faster
- B. Because it leaves more training data
- C. Because the templates change every year
- D. Because in a backtest the model can only have been trained on earlier texts; a random split mixes the future with the past and gives an optimistic estimate
*Answer D: Because in a backtest the model can only have been trained on earlier texts; a random split mixes the future with the past and gives an optimistic estimate.* It is the level 5 rule (lessons 5.4.1 and 5.4.3): validate through time. Besides, in practice the training labels would be an annotated set (such as Financial PhraseBank), not the true sentiment, which does not exist.
5. What does the license of the `ProsusAI/finbert` weights (CC-BY-NC-SA-4.0) restrict?
- A. Nothing: everything on Hugging Face is free to use
- B. It only prevents modifying them
- C. It allows using them with attribution, for non-commercial purposes and sharing derivative works under the same license: commercial use requires another model or a permission
- D. It forces you to publish the strategy's code
*Answer C: It allows using them with attribution, for non-commercial purposes and sharing derivative works under the same license: commercial use requires another model or a permission.* NC (non-commercial) and SA (share-alike) are the two clauses that matter in a fund or a company. The course uses it for educational purposes; the README of the `ia` environment names an alternative under the Apache-2.0 license.
## Key points
- Models work with subword tokens. The same text costs more tokens in Spanish than in English with vocabularies trained on English, and WordPiece without accents turns “demanda” into `demand` + `##a`.
- A dictionary is transparent, fast and free, but it does not understand context: in the course corpus the financial list correlates 0.71 with the true sentiment, the general one 0.35 and, with negations, the financial one drops to 0.21.
- The Loughran-McDonald dictionary was born because general lists mistake accounting vocabulary for bad news; it is free for academic research and its commercial use requires a license.
- Embeddings turn a sentence into a vector comparable by cosine similarity; with a linear model on top and temporal validation they learn the tone, but MiniLM is an English model and performs worse in Spanish.
- FinBERT, without seeing a single news item from the corpus, reaches 0.88 on the English ones (the dictionary, 0.70) and is useless in Spanish. Its license (CC-BY-NC-SA-4.0) prevents commercial use without permission.
- The course texts come from templates: the correlation figures are higher than with real news. What carries over to the real world is the ranking of the methods and the type of error of each one.
## Resources
### Documentation
- [Loughran and McDonald: Master Dictionary and sentiment word lists (University of Notre Dame)](https://sraf.nd.edu/loughranmcdonald-master-dictionary/) · sraf.nd.edu
- [Loughran and McDonald (2011): When Is a Liability Not a Liability?, Journal of Finance 66(1)](https://doi.org/10.1111/j.1540-6261.2010.01625.x) · doi.org
- [Araci (2019): FinBERT, Financial Sentiment Analysis with Pre-trained Language Models](https://arxiv.org/abs/1908.10063) · arxiv.org
- [ProsusAI/finbert model card on Hugging Face (CC-BY-NC-SA-4.0 license)](https://huggingface.co/ProsusAI/finbert) · huggingface.co
- [Hugging Face: tokenization algorithms (BPE, WordPiece, Unigram)](https://huggingface.co/docs/transformers/tokenizer_summary) · huggingface.co
- [Reimers and Gurevych (2019): Sentence-BERT](https://arxiv.org/abs/1908.10084) · arxiv.org
### Data
- [Level 6 data: invented financial lexicon (CSV)](https://pythonparatrading.com/curso/datos/nivel-6/texto6_lexico.csv) · Download · texto6_lexico.csv
Next lesson: [6.1.2 · Text sources: EDGAR, CNMV, transcripts and social media](https://pythonparatrading.com/en/python-trading-course/level-6/text-sources-edgar-cnmv-transcripts-social-media/)