Tokenization, embeddings, financial dictionaries and FinBERT.

From text to a number

A strategy that reads news needs, in the end, a column of numbers: one score per news item that can be aggregated, lagged and correlated with returns. Between the text and that column there are always the same steps: split the text into units, turn those units into numbers and summarize them in a score.

flowchart TB
  T[Text] --> K[Tokens]
  K --> D[Lexicon count]
  K --> E[Embedding]
  K --> F[Fine-tuned model]
  D --> S[Score]
  E --> R[Linear model]
  R --> S
  F --> S

The three branches are the three families of this lesson: dictionaries (counting positive and negative words), embeddings (one vector per sentence, with a model trained on top) and fine-tuned models such as FinBERT, which read the sentence and directly return the probability of each class. Generative LLMs, which also score sentiment, arrive in lesson 6.2.3.

The data are the synthetic news of level 6: 39,570 items from five fictitious agencies about 120 companies of the level 5 panel, with their publication timestamp. The corpus mixes news in Spanish and in English, and the Spanish texts are kept as they are, because the lesson measures precisely what each method does with each language. What does not exist in a real corpus is in noticias6_verdad.csv: the true sentiment of each news item, between −1 and 1, with which the generator wrote the text and moved the price. It is the known answer against which each method is measured.

import re
import textwrap

import numpy as np
import pandas as pd

noticias = pd.read_parquet("noticias6_corpus.parquet")
verdad = pd.read_csv("noticias6_verdad.csv")
df = noticias.merge(verdad, on="id_noticia")
orig = df[df["tipo_noticia"] == "original"].copy()
orig["texto"] = orig["titular"] + ". " + orig["cuerpo"]
print(len(orig), "original news items")
print(orig["idioma"].value_counts().to_dict())
fila = orig.iloc[2]
print(fila["texto"])
print("true sentiment:", fila["sentimiento_verdadero"])

22593 original news items
{'es': 16969, 'en': 5624}
Placeholder & Co. cuts Numvia Communications price target to $19.23. Analysts at Placeholder & Co. cut their price target on Numvia Communications to $19.23. Numvia Communications issued a warning about business weakness.
true sentiment: -0.546

Syndicated copies (the same story at another agency, sometimes translated) and market wrap-ups are left out: a story is scored once. How to detect the copies is part of lesson 6.1.2.

Tokenizing: words and subwords

The simplest way to split a text is a regular expression that extracts words. It is what dictionaries use: convert to lowercase and keep sequences of letters. NLTK adds rules for punctuation and contractions:

from nltk.tokenize import word_tokenize

frase = "Numvia didn't beat estimates; revenue fell 3.5%."

def palabras(texto):
    return re.findall(r"[a-záéíóúüñ]+", texto.lower())

print(palabras(frase))
print(word_tokenize(frase))

['numvia', 'didn', 't', 'beat', 'estimates', 'revenue', 'fell']
['Numvia', 'did', "n't", 'beat', 'estimates', ';', 'revenue', 'fell', '3.5', '%', '.']

The regular expression loses the negation (“didn't” ends up as didn and t) and the figures; word_tokenize separates did and n't and keeps 3.5 and %. For a dictionary the first one is enough; for a model neither works, because a vocabulary of whole words is never complete: company names, figures and rare words would be left out.

Current models use subwords: a fixed vocabulary (from 30,000 to 150,000 pieces) learned on a corpus, with which any text is broken down into known fragments. The two families that matter here:

BPE was proposed by Sennrich, Haddow and Birch (2016) for machine translation; the Hugging Face page on tokenization algorithms explains the variants11Note 1 · Sennrich, Haddow and Birch (2016), “Neural Machine Translation of Rare Words with Subword Units” (https://arxiv.org/abs/1508.07909); Hugging Face, “Tokenization algorithms”: BPE, byte-level BPE (GPT-2), WordPiece (BERT family) and Unigram/SentencePiece (https://huggingface.co/docs/transformers/tokenizer_summary), accessed on October 23, 2026.. What matters for a quant is what each vocabulary does with financial texts in two languages:

import tiktoken
from transformers import AutoTokenizer

tok_bert = AutoTokenizer.from_pretrained("ProsusAI/finbert")
tok_qwen = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tok_tik = tiktoken.get_encoding("cl100k_base")

en = "Revenue was $3,075.9 million, a weak quarter."
es = "Las ventas sumaron 3.075,9 millones: caída de la demanda."

def piezas(texto):
    q = [tok_qwen.convert_tokens_to_string([t])
         for t in tok_qwen.tokenize(texto)]
    t = [tok_tik.decode([i]) for i in tok_tik.encode(texto)]
    return {"wordpiece": tok_bert.tokenize(texto),
            "qwen": q, "cl100k": t}

for texto in (en, es):
    for nombre, p in piezas(texto).items():
        print(f"{nombre:9} {len(p):2}", "|".join(p))
    print()

wordpiece 15 revenue|was|$|3|,|07|##5|.|9|million|,|a|weak|quarter|.
qwen      16 Revenue| was| $|3|,|0|7|5|.|9| million|,| a| weak| quarter|.
cl100k    14 Revenue| was| $|3|,|075|.|9| million|,| a| weak| quarter|.

wordpiece 22 las|vent|##as|sum|##aro|##n|3|.|07|##5|,|9|mill|##ones|:|cai|##da|de|la|demand|##a|.
qwen      21 Las| ventas| sum|aron| |3|.|0|7|5|,|9| millones|:| ca|ída| de| la| demand|a|.
cl100k    19 Las| ventas| sum|aron| |3|.|075|,|9| millones|:| ca|ída| de| la| demand|a|.

The Spanish sentence means “sales totaled 3,075.9 million: a fall in demand”. Three details with consequences:

• Figures. WordPiece splits 3,075.9 into five pieces and Qwen writes each digit separately. No language model “sees” the number 3,075.9 as a magnitude; it sees a sequence of symbols. That is why extracting figures with an LLM needs validation (lesson 6.2.2).
• Accents and lowercase. The uncased BERT vocabulary strips accents: “caída” (fall) becomes cai + ##da.
• False friends. “demanda” (demand) becomes demand + ##a: an English piece with a different meaning. A model trained on English text will associate that piece with what demand means in English.

The cost also changes with the language. Measured on 400 news items of each language, in tokens per word:

muestra = (orig.groupby("idioma", group_keys=False)
           .sample(400, random_state=0))
n_palabras = muestra["texto"].str.split().str.len()

def n_tokens(textos, nombre):
    if nombre == "cl100k":
        return np.array([len(tok_tik.encode(t)) for t in textos])
    tok = tok_bert if nombre == "wordpiece" else tok_qwen
    return np.array([len(tok.tokenize(t)) for t in textos])

coste = {}
for nombre in ("wordpiece", "qwen", "cl100k"):
    ratio = n_tokens(muestra["texto"], nombre) / n_palabras
    coste[nombre] = ratio.groupby(muestra["idioma"]).mean()
pd.DataFrame(coste).round(2)

        wordpiece  qwen  cl100k
idioma
en           1.47  1.52    1.44
es           2.03  1.74    1.75

Spanish costs more tokens than English with all three vocabularies, and the gap is larger with BERT's, which was built only from English text. It matters for three reasons: commercial APIs charge per token (lesson 6.2.1); BERT-type models truncate the input to 512 tokens (FinBERT) or to 256 pieces (MiniLM, according to its model card22Note 2 · Model card of sentence-transformers/all-MiniLM-L6-v2 (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, accessed on October 23, 2026): Apache-2.0 license, 384-dimensional vectors, English, and “by default, input text longer than 256 word pieces is truncated”.), and a long document, such as the risk section of a 10-K, does not fit whole; and a vocabulary that chops a language into many pieces usually indicates that the model saw little text in it.

Dictionaries: from Harvard to Loughran-McDonald

A sentiment dictionary is a list of words with their category. The most common score counts the positive ($p$) and the negative ($n$) words of a text:

The 1 in the denominator avoids dividing by zero and pulls texts with few tone words toward zero. The result lies between −1 and 1.

For years general-purpose lists were used, such as the Harvard dictionary (General Inquirer). Loughran and McDonald (2011) showed that in annual reports (10-K) almost three-fourths of the occurrences of “negative” words from that list correspond to words that are not negative in a financial context: tax, cost, liability, capital, vice (from vice president)33Note 3 · Loughran and McDonald (2011), Journal of Finance 66(1), 35-65 (https://doi.org/10.1111/j.1540-6261.2010.01625.x): “almost three-fourths of the words identified as negative by the widely used Harvard Dictionary are words typically not considered negative in financial contexts”. In the body of the article the figure is 73.8% of the occurrences of negative words.. They built their own domain lists, which they maintain and update on their website with seven categories: negative, positive, uncertainty, litigious, strong modal, weak modal and constraining. The lists are free for academic research and their commercial use requires a license from the authors44Note 4 · University of Notre Dame, Software Repository for Accounting and Finance, “Loughran-McDonald Master Dictionary w/ Sentiment Word Lists” (https://sraf.nd.edu/loughranmcdonald-master-dictionary/, accessed on October 23, 2026): free for academic research, commercial license upon request to the authors.. They are not copied here.

Instead, the course ships texto6_lexico.csv, an invented lexicon with five of those categories in Spanish and in English (list financiera, financial) and a general list that imitates the flaw of general-purpose lists:

lex = pd.read_csv("texto6_lexico.csv")
print(lex.pivot_table(index=["lista", "categoria"], columns="idioma",
                      values="palabra", aggfunc="size",
                      fill_value=0))
gen_neg = lex.query("lista == 'general' and categoria == 'negativa'"
                    " and idioma == 'es'")
print(textwrap.fill(" ".join(gen_neg["palabra"]), 72))

idioma                    en  es
lista      categoria
financiera incertidumbre  15  17
           litigio        12  14
           negativa       26  31
           positiva       23  31
           restrictiva     9   9
general    negativa       17  17
           positiva        9  10
cargo cargos caída coste costes crudo cáncer demanda deuda débil
impuesto impuestos pasivo pasivos pérdida quiebra vicepresidente

The categories are incertidumbre (uncertainty), litigio (litigious), negativa (negative), positiva (positive) and restrictiva (constraining). The general list marks as negative coste (cost), deuda (debt), impuesto (tax), pasivo (liability), crudo (crude), vicepresidente (vice president) and demanda (demand, but also “lawsuit”): the same error that Loughran and McDonald found. Score all the news with the two lists and compare them with the true sentiment:

listas = {}
for (lista, idioma), g in lex.groupby(["lista", "idioma"]):
    pos = set(g.loc[g["categoria"] == "positiva", "palabra"])
    neg = set(g.loc[g["categoria"] == "negativa", "palabra"])
    listas[(lista, idioma)] = (pos, neg)

def tono(texto, idioma, lista="financiera"):
    pos, neg = listas[(lista, idioma)]
    w = palabras(texto)
    p = sum(x in pos for x in w)
    n = sum(x in neg for x in w)
    return (p - n) / (p + n + 1)

orig["dic_fin"] = [tono(t, i) for t, i in
                   zip(orig["texto"], orig["idioma"])]
orig["dic_gen"] = [tono(t, i, "general") for t, i in
                   zip(orig["texto"], orig["idioma"])]
s = orig["sentimiento_verdadero"]
print("financial:", round(orig["dic_fin"].corr(s), 3))
print("general:  ", round(orig["dic_gen"].corr(s), 3))
print("neutral news with general tone < 0: "
      f"{(orig.loc[s == 0, 'dic_gen'] < 0).mean():.1%}")

financial: 0.709
general:   0.351
neutral news with general tone < 0: 7.0%

The financial list correlates 0.71 with the true sentiment; the general one, half that (0.35). The last line points to a cause: 7% of the neutral news items (sentiment exactly zero, such as a shareholder meeting or a conference) get a negative tone from the general list. Which words trigger those false negatives?

_, neg_gen = listas[("general", "es")]
neutras = orig.loc[(s == 0) & (orig["idioma"] == "es"), "texto"]
cuenta = pd.Series([w for t in neutras for w in palabras(t)
                    if w in neg_gen]).value_counts()
cuenta.head(6)

demanda    426
caída       52
Name: count, dtype: int64

Almost all of them come from “demanda”, in sentences such as “el impacto final dependerá de la demanda” (the final impact will depend on demand): customer demand, not a lawsuit. The financial list classifies it as litigious (a category that does not enter the tone) and the general one, as negative. It is the same kind of error that Loughran and McDonald found with liability or tax: a word that is bad news in another context and descriptive in the financial one.

What a dictionary cannot see

Counting words ignores order. Two types of phrase break it: negations (“la compañía descarta pérdidas”, the company rules out losses, contains pérdidas, losses) and tone phrases without lexicon words (“es un jarro de agua fría para los accionistas”, it is a bucket of cold water for shareholders). The generator records in noticias6_verdad.csv how many phrases of each type each news item contains, so it is possible to measure where the dictionary fails:

grupo = np.select(
    [orig["frases_negacion"] > 0, orig["frases_fuera_lexico"] > 0],
    ["with negation", "off-lexicon"], "lexicon only")
orig["grupo"] = grupo
por_grupo = orig.groupby("grupo").apply(
    lambda g: pd.Series({
        "noticias": len(g),
        "corr_dic": g["dic_fin"].corr(g["sentimiento_verdadero"]),
    }), include_groups=False)
por_grupo.round(3)

               noticias  corr_dic
grupo
lexicon only    15921.0     0.784
off-lexicon      3718.0     0.833
with negation    2954.0     0.210

In news with a negation, the correlation drops to 0.21, a quarter of that of the rest: the dictionary reads the negative word and gets the sign wrong. Phrases outside the lexicon, on the other hand, do not lower the correlation: in the generator, the more intense news items carry more tone phrases (two on average, against fewer than one), and the other phrases do use the lexicon. An example of negation:

ej = orig[(orig["frases_negacion"] > 0)
          & (orig["idioma"] == "es")
          & (orig["sentimiento_verdadero"] > 0.4)
          & (orig["dic_fin"] < 0)].iloc[0]
print(textwrap.fill(ej["cuerpo"], 72))
print("true:", ej["sentimiento_verdadero"],
      "| dictionary:", round(ej["dic_fin"], 3))

Kogsa Bancorp ha firmado un contrato de 829,7 millones de dólares con la
administración de un estado de EE. UU. para los próximos 3 años. El
grupo deja atrás las dificultades del año pasado. Las acciones cerraron
la última sesión en 201,85 dólares.
true: 0.7784 | dictionary: -0.5

“Deja atrás las dificultades” (leaves the difficulties behind) is good news written with a negative word from the lexicon (dificultades, difficulties). The dictionary counts it as negative and the news item, clearly positive (a contract worth 829.7 million dollars), scores −0.5.

Dictionaries still have three virtues that keep them in use in research and in production: they are transparent (you know why a news item scores what it scores), cheap (millions of documents in seconds, without a GPU) and stable (the same list gives the same score today and ten years from now, without look-ahead bias; lesson 6.2.5). The usual improvements (negation rules within a window of a few words, tf-idf weighting, lists by sector) reduce their errors without changing their nature.

Embeddings: a sentence as a vector

A sentence embedding is a fixed-dimension vector that summarizes its meaning: similar sentences end up close together and closeness is measured with cosine similarity (the dot product of the normalized vectors). all-MiniLM-L6-v2 is a small sentence-transformers model (22 million parameters, 384 dimensions, trained on English text²) that follows the idea of Sentence-BERT55Note 5 · Reimers and Gurevych (2019), “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” (https://arxiv.org/abs/1908.10084).. Its weights are in the cache of the ia environment, pinned by commit:

import torch
from sentence_transformers import SentenceTransformer

torch.manual_seed(0)
st = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2",
                         device="cpu", local_files_only=True)
frases = [
    "The company reported record profits.",
    "Earnings hit an all-time high.",
    "The company warned of heavy losses.",
    "La empresa obtuvo beneficios récord.",
    "La empresa advirtió de fuertes pérdidas.",
]
E = st.encode(frases, normalize_embeddings=True,
              show_progress_bar=False)
print(E.shape)
pd.DataFrame(E @ E.T, index=range(5), columns=range(5)).round(2)

(5, 384)
      0     1     2     3     4
0  1.00  0.53  0.42  0.41  0.18
1  0.53  1.00  0.27  0.21  0.07
2  0.42  0.27  1.00  0.14  0.15
3  0.41  0.21  0.14  1.00  0.68
4  0.18  0.07  0.15  0.68  1.00

Sentences 0 and 1 say the same thing with different words and their similarity is the highest among the English ones (0.53); sentence 2 shares almost all its words with sentence 0 and says the opposite, but its similarity is not much lower (0.42). A general-purpose embedding mostly measures what a sentence is about, not in which direction. In Spanish it is worse: sentences 3 and 4 (the Spanish versions of 0 and 2), which say opposite things, are the most similar pair in the table (0.68), and each one is far from its English translation (0.41 and 0.15). MiniLM is not multilingual: with Spanish text it mostly compares the form.

Turning embeddings into sentiment requires a model on top trained with labeled examples. In practice, the labels would come from an annotated set (such as Financial PhraseBank, the one FinBERT used) or from the price reaction; here, to measure the ceiling of the method, the true sentiment from 2019 to 2022 is used and evaluation runs from 2023 onward, with the temporal split of level 5 (lesson 5.4.3). A sample of 1,200 news items per language is enough for the comparison and keeps the cell at a few seconds on CPU:

from sklearn.linear_model import Ridge

m = (orig.groupby("idioma", group_keys=False)
     .sample(1200, random_state=1).sort_values("ts"))
X = st.encode(m["texto"].tolist(), batch_size=64,
              normalize_embeddings=True, show_progress_bar=False)
y = m["sentimiento_verdadero"].to_numpy()
entreno = (m["ts"] < "2023-01-01").to_numpy()
m["ridge"] = np.nan
for idioma in ("en", "es"):
    i = (m["idioma"] == idioma).to_numpy()
    modelo = Ridge(alpha=1.0).fit(X[i & entreno], y[i & entreno])
    prueba = i & ~entreno
    m.loc[m.index[prueba], "ridge"] = modelo.predict(X[prueba])

test = m[m["ts"] >= "2023-01-01"]
test.groupby("idioma").apply(lambda g: pd.Series({
    "prueba": len(g),
    "corr_ridge": g["ridge"].corr(g["sentimiento_verdadero"]),
    "corr_dic": g["dic_fin"].corr(g["sentimiento_verdadero"]),
}), include_groups=False).round(3)

        prueba  corr_ridge  corr_dic
idioma
en       524.0       0.919     0.702
es       491.0       0.826     0.729

With a few hundred examples per language, the regression on embeddings beats the dictionary in both languages, and by more in English, MiniLM's language, than in Spanish. The figure must be read with care: the linear model has learned the generator's templates with perfect labels, and those of 2023 are the same as those of 2019. With real news, a few hundred noisy labels do not get that far, and language changes over the years. The lasting advantage of embeddings lies elsewhere: the same vector serves to search for similar documents, group news by topic (lesson 6.1.3) or feed a RAG (lesson 6.3.1).

FinBERT: a classifier fine-tuned for finance

FinBERT (Araci, 2019)66Note 6 · Araci (2019), “FinBERT: Financial Sentiment Analysis with Pre-trained Language Models” (https://arxiv.org/abs/1908.10063); model card of ProsusAI/finbert (https://huggingface.co/ProsusAI/finbert, accessed on October 23, 2026): CC-BY-NC-SA-4.0 license, three labels and fine-tuned with Financial PhraseBank. is a base-size BERT (110 million parameters) pre-trained on financial text and fine-tuned to classify sentences into three classes (positive, negative, neutral) with Financial PhraseBank, a set of English financial news sentences annotated by hand. The ProsusAI/finbert version is the most widely used.

flowchart LR
  F[Sentence] --> W[WordPiece]
  W --> B[BERT base]
  B --> C[Class layer]
  C --> P[Probabilities]

The output is three probabilities that add up to one. A common continuous score is $P(\text{positive}) - P(\text{negative})$, between −1 and 1, comparable with the dictionary tone:

from transformers import pipeline

finbert = pipeline("text-classification", model="ProsusAI/finbert",
                   device="cpu")

def puntua(textos, lote=32):
    salidas = finbert(list(textos), batch_size=lote, top_k=None,
                      truncation=True)
    filas = [{d["label"]: d["score"] for d in s} for s in salidas]
    probs = pd.DataFrame(filas)[["positive", "negative", "neutral"]]
    return probs.assign(fb=probs["positive"] - probs["negative"])

pruebas = ["The company does not expect any losses this year.",
           "Revenue declined sharply and the outlook was cut.",
           "The annual meeting will be held in Madrid."]
puntua(pruebas).round(3)

   positive  negative  neutral     fb
0     0.058     0.345    0.596 -0.287
1     0.008     0.974    0.019 -0.966
2     0.027     0.026    0.947  0.001

The second sentence is negative beyond doubt (0.97) and the third, neutral (0.95). The first is good news written with a negation, and FinBERT does not get it right either: it leaves it between neutral (0.60) and negative (0.35). A single sentence proves nothing; it has to be measured over many. The news items of the test period from the previous cell are used: the 524 English ones and 200 Spanish ones at random (FinBERT is the slowest step of the lesson: a few seconds per hundred news items on CPU), so that the four methods are compared on the same news:

sub_en = test[test["idioma"] == "en"].copy()
sub_es = test[test["idioma"] == "es"].sample(200, random_state=2)
sub_en["fb"] = puntua(sub_en["texto"])["fb"].to_numpy()
sub_es["fb"] = puntua(sub_es["texto"])["fb"].to_numpy()

def correlaciones(g):
    s = g["sentimiento_verdadero"]
    return pd.Series({"noticias": len(g),
                      "finbert": g["fb"].corr(s),
                      "ridge": g["ridge"].corr(s),
                      "dic_fin": g["dic_fin"].corr(s),
                      "dic_gen": g["dic_gen"].corr(s)})

pd.DataFrame({"en": correlaciones(sub_en),
              "es": correlaciones(sub_es)}).T.round(3)

    noticias  finbert  ridge  dic_fin  dic_gen
en     524.0    0.876  0.919    0.702    0.459
es     200.0   -0.114  0.857    0.760    0.473

In English, FinBERT (0.88) clearly beats the financial dictionary (0.70) and the general list (0.46) without having seen a single news item from the corpus. Only the ridge on embeddings (0.92) beats it, trained with the perfect labels of the same templates: an advantage that does not exist with real data. In Spanish, FinBERT gives −0.11: the model does not know the language and its tokenizer chops it into meaningless fragments, while the Spanish dictionary and the ridge keep their level. The header figure summarizes the four methods on the 524 English news items. Where does FinBERT win? Where the dictionary fails:

sub_en.groupby("grupo").apply(
    lambda g: pd.Series({
        "noticias": len(g),
        "finbert": g["fb"].corr(g["sentimiento_verdadero"]),
        "dic_fin": g["dic_fin"].corr(g["sentimiento_verdadero"]),
    }), include_groups=False).round(3)

               noticias  finbert  dic_fin
grupo
lexicon only      361.0    0.879    0.790
off-lexicon        91.0    0.886    0.841
with negation      72.0    0.870    0.100

In the news that only uses the lexicon, the dictionary (0.79) comes close to FinBERT (0.88): counting words is enough when the text contains them. In those with a negation, the dictionary collapses (0.10) and FinBERT keeps its level (0.87). That it failed with the single sentence of the previous cell and gets the group right is not a contradiction: in the news the negation comes with the rest of the text, and what is measured is the correlation over many items. With 524 news items, the small groups have a noticeable sampling error (with about 70 news items, a correlation of 0.1 has a standard error of a little more than 0.1), so it is better to read the ranking and not the decimals.

What to choose and what it costs

Three warnings to take into a backtest:

• The score is not the signal. A well-measured sentiment is only worth something if it moves the price after you can trade. Measuring that reaction requires aligning each news item with the right session: that is lesson 6.1.3, where the same dictionary recovers about 75 bp per unit of tone on the reaction day against 119 bp with the true sentiment (level 6 README).
• A fine-tuned model also has a date. FinBERT was trained on sentences from before 2019; a classifier trained on data after the backtest period introduces information from the future (lesson 6.2.5).
• Language is a design decision. With a European universe, half of the news can be in a language the model does not understand. In this corpus, applying FinBERT to everything would have spoiled the signal of 75% of the news.

Exercises

1. Add a negation rule to the dictionary: if one of the three words before a lexicon word is “no” (no), “descarta” (rules out), “sin” (without) or “niega” (denies), flip its sign. Measure the correlation with the true sentiment on the Spanish news with negation before and after.

NEG = {"no", "descarta", "sin", "niega"}

def tono_neg(texto, idioma="es"):
    pos, neg = listas[("financiera", idioma)]
    w = palabras(texto)
    p = n = 0
    for k, x in enumerate(w):
        signo = -1 if NEG & set(w[max(0, k - 3):k]) else 1
        if x in pos:
            p, n = (p + 1, n) if signo > 0 else (p, n + 1)
        elif x in neg:
            p, n = (p, n + 1) if signo > 0 else (p + 1, n)
    return (p - n) / (p + n + 1)

g = orig[(orig["idioma"] == "es") & (orig["grupo"] == "with negation")]
nuevo = g["texto"].map(tono_neg)
s_g = g["sentimiento_verdadero"]
print("before:", round(g["dic_fin"].corr(s_g), 3))
print("after: ", round(nuevo.corr(s_g), 3))

before: 0.21
after:  0.768

2. Compute how many original English news items exceed FinBERT's 512 tokens and how many exceed MiniLM's 256. Do the same with the 10-Ks in filings6_documentos.parquet. What would you do with the documents that do not fit?

from transformers.utils import logging as hf_logging

hf_logging.set_verbosity_error()  # no “> 512 tokens” warning
en_txt = orig.loc[orig["idioma"] == "en", "texto"].tolist()
n = np.array([len(x) for x in tok_bert(en_txt)["input_ids"]])
print(f"news > 256: {(n > 256).mean():.1%}",
      f"> 512: {(n > 512).mean():.1%}")
docs = pd.read_parquet("filings6_documentos.parquet")
diez_k = docs.loc[docs["tipo"] == "10-K", "texto"].tolist()
n_k = np.array([len(x) for x in tok_bert(diez_k)["input_ids"]])
print(f"10-K > 512: {(n_k > 512).mean():.1%}",
      f"median {np.median(n_k):.0f}")

news > 256: 0.0% > 512: 0.0%
10-K > 512: 100.0% median 898

3. A vendor offers a “news sentiment” computed with a model fine-tuned on news from 2015 to 2024 and a history going back to 2010. What questions would you ask before using it in a backtest?