Lesson 6.2.5 · 60 min

Look-ahead bias in LLMs: the model that already knows the future

An LLM trained on text up to 2024 “predicts” what happened in 2022 very well: not because it can predict, but because it has read it. This lesson measures the effect with four simulated models and known answers, shows how to estimate the effective cutoff when the declared one is not reliable, computes how many questions an honest evaluation needs and checks with a real LLM that removing the company's name is not enough.

Fig. — Accuracy by half-year of resolution for three simulated LLMs with a training cutoff and a point-in-time model (620 questions on stocks and sales)
05/05
Module
6.2 · LLMs for research and trading
Duration
60 min
Updated
Level 6 lessons
  1. 6.1 · Classical and modern financial NLP

    1. 6.1.1 Tokenization, embeddings, financial dictionaries and FinBERT
    2. 6.1.2 Text sources: EDGAR, CNMV, transcripts and social media
    3. 6.1.3 Event studies with text signals, timestamp alignment and topic modeling
  2. 6.2 · LLMs for research and trading

    1. 6.2.1 How an LLM works: from the next token to the API
    2. 6.2.2 Structured prompting: JSON you can validate
    3. 6.2.3 Guidance, KPIs and tone: what an LLM extracts well
    4. 6.2.4 Lightweight fine-tuning for financial tasks: LoRA and QLoRA
    5. 6.2.5 Look-ahead bias in LLMs: the model that already knows the future
  3. 6.3 · Financial RAG

    1. 6.3.1 Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant
    2. 6.3.2 Chunking financial documents: tables, footnotes and XBRL
    3. 6.3.3 RAG over reports, prospectuses and KIDs, and how to evaluate it
  4. 6.4 · AI agents for trading

    1. 6.4.1 Tool-calling agents: tool calling, ReAct and planning
    2. 6.4.2 Agent frameworks and MCP
    3. 6.4.3 TradingAgents-style multi-agent architectures
    4. 6.4.4 Evidence and limits of agents
    5. 6.4.5 Guardrails for agents: the agent proposes, the engine decides
    6. 6.4.6 Research agents: hypotheses, alpha mining and review
  5. 6.5 · AI as the quant's copilot

    1. 6.5.1 Coding assistants with a test-first workflow
    2. 6.5.2 Automated review of backtests and code migrations
    3. 6.5.3 Risks of AI-generated code
  6. 6.6 · Big data for trading

    1. 6.6.1 When data becomes “big”: ticks, order books, options and universes
    2. 6.6.2 Parquet, Arrow, Polars streaming and out-of-core DuckDB
    3. 6.6.3 Dask, Spark and Ray: from one machine to a cluster
    4. 6.6.4 Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+
    5. 6.6.5 Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF)
  7. 6.7 · Alternative data

    1. 6.7.1 Overview and size of the alternative data market
    2. 6.7.2 Types of alternative data: what each one measures
    3. 6.7.3 Responsible web scraping
    4. 6.7.4 Alternative data vendors and due diligence
    5. 6.7.5 From alternative data to a trading signal
    6. 6.7.6 Satellite imagery, computer vision and supply chains

A backtest that measures memory

Look-ahead bias is an old acquaintance of the course: using at a given date a piece of data that was not known then (lessons 2.5.8 and 5.2.2). With an LLM it shows up in a new form that is harder to see. The model was trained on trillions of tokens of text up to a cutoff date: news, reports, forums, commented quotes. If, in a backtest dated 2021, it is asked whether a stock will rise the following month, and the model was trained on text from 2023, the answer may be in its memory. The backtest looks spectacular and measures nothing that can be repeated.

The problem is real and documented. Glasserman and Lin (2023) separate two effects when scoring headlines with GPT: the look-ahead effect (the model knows how the price moved afterwards) and the “distraction” effect (what it knows about the company interferes with the tone of the text), and propose anonymizing the companies. Lopez-Lira, Tang and Zhu (2025) show that LLMs remember exact values of economic series from before their cutoff and none after it, and that neither asking them to respect the date nor masking names and dates prevents it. Sarkar and Vafa (2025) propose direct tests based on events that were unpredictable with the information available at the date and find the bias in earnings calls.

Since with a real model you do not know what it has read, this lesson uses the level 6 synthetic question bank, with known answers and four simulated models (there is no LLM behind them; their answers follow rules published in sesgo6_modelos_verdad.csv), and at the end checks the mechanism with the course's local LLM.

yesnoCorpus up to cutoffModelQuestion at date tBefore the cutoff?RemembersPredicts
Mermaid source
flowchart LR
  A[Corpus up to cutoff] --> B[Model]
  C[Question at date t] --> B
  B --> D{Before the cutoff?}
  D -->|yes| E[Remembers]
  D -->|no| F[Predicts]

The data: 620 questions and four models

Each question has a date (when it would be asked in the backtest), a resolution date (the first day the answer is known) and the correct answer, computed with the level's synthetic prices and sales. There are two types: whether a stock will close above its price 21 sessions later and whether the quarter's sales will beat the consensus. Three models declare a cutoff date; modelo-pit is point-in-time: for each question, its cutoff is the day before. The question texts are in Spanish, like the rest of the level 6 corpus, and stay as they are; the answers in the data are sí (yes) and no.

IN [01]python
import numpy as np
import pandas as pd

q = pd.read_csv("sesgo6_preguntas.csv",
                parse_dates=["fecha_pregunta", "fecha_resolucion"])
resp = pd.read_csv("sesgo6_respuestas.csv")
modelos = pd.read_csv("sesgo6_modelos.csv")
print(modelos.to_string(index=False))
print(q["tipo"].value_counts().to_string())
ej = q.iloc[0]
print(ej["fecha_pregunta"].date(), "·", ej["pregunta"])
print("resolves on", ej["fecha_resolucion"].date(), "· answer:",
      ej["respuesta_correcta"])
OUT [01]Questions with known answers and declared cutoffs
    modelo        fecha_corte_declarada
  modelo-a                   2021-09-30
  modelo-b                   2023-04-30
  modelo-c                   2024-10-31
modelo-pit día anterior a cada pregunta
tipo
precio_21_sesiones          420
ventas_frente_a_consenso    200
2019-03-01 · A cierre de 2019-03-01, Sofsa Software (SOFE) cotiza a 793,88 dólares. ¿Cerrará por encima de ese precio dentro de 21 sesiones?
resolves on 2019-04-01 · answer: no

The naive backtest

The simplest strategy: buy the stock if the model says “yes” and sell it if it says “no”, for 21 sessions. The questions use the ticker in force at their date and the prices, the current one: 11 questions are about companies that changed ticker later, the same entity-mapping problem as lesson 6.7.5.

IN [02]python
precios = pd.read_parquet("acciones6_precios.parquet")
cierres = precios.pivot(index="fecha", columns="ticker",
                        values="cierre")
# questions use the ticker at their date; prices, the current one
hist = pd.read_csv("acciones6_tickers.csv")
emp = pd.read_csv("acciones6_empresas.csv")
actual = (hist.merge(emp[["id_empresa", "ticker"]], on="id_empresa",
                     suffixes=("", "_hoy"))
          .set_index("ticker")["ticker_hoy"])
print("questions with an old ticker:",
      int((q["ticker"] != q["ticker"].map(actual)).sum()))
i = cierres.index.get_indexer(q["fecha_pregunta"])
cols = cierres.columns.get_indexer(q["ticker"].map(actual))
valores = cierres.to_numpy()
q["r21"] = valores[i + 21, cols] / valores[i, cols] - 1

d = resp.merge(q, on="id_pregunta")
d["acierto"] = d["respuesta"] == d["respuesta_correcta"]
d["pnl"] = np.where(d["respuesta"] == "sí", 1, -1) * d["r21"]
precio = d["tipo"] == "precio_21_sesiones"
resumen = pd.DataFrame({
    "accuracy (%)": 100 * d.groupby("modelo")["acierto"].mean(),
    "21-sess. ret. (%)":
        100 * d[precio].groupby("modelo")["pnl"].mean()})
print(resumen.round(1).to_string())
largo = q.loc[q["tipo"] == "precio_21_sesiones", "r21"].mean()
print(f"always long: {100 * largo:.1f} %")
OUT [02]The naive backtest: following the model over the whole sample
questions with an old ticker: 11
            accuracy (%)  21-sess. ret. (%)
modelo
modelo-a            65.6                2.5
modelo-b            79.8                4.6
modelo-c            87.4                5.9
modelo-pit          50.8                0.7
always long: 1.1 %

Over the whole sample, the most recent model looks the best: modelo-c gets 87.4% right and would earn an average 5.9% on each 21-session trade, against 1.1% for being always long. modelo-a, the oldest, stays at 65.6%, and the point-in-time one at 50.8%, a coin flip. A researcher comparing models this way would choose the most recent one and believe they had an exceptional strategy. What they have measured is how much of the sample falls before each model's cutoff.

Before and after the cutoff

IN [03]python
corte = pd.to_datetime(modelos.set_index("modelo")
                       ["fecha_corte_declarada"], errors="coerce")
d["corte"] = d["modelo"].map(corte)
d["zona"] = np.where(d["fecha_resolucion"] <= d["corte"],
                     "before", "after")

def t_media(x):
    return x.mean() / x.std(ddof=1) * np.sqrt(len(x))

filas = []
for (m, z), g in d.groupby(["modelo", "zona"]):
    p = g[g["tipo"] == "precio_21_sesiones"]["pnl"]
    filas.append({"modelo": m, "zona": z, "n": len(g),
                  "accuracy (%)": 100 * g["acierto"].mean(),
                  "ret. (%)": 100 * p.mean(), "t": t_media(p)})
print(pd.DataFrame(filas).round(2).to_string(index=False))
OUT [03]Before and after the declared cutoff
    modelo   zona   n  accuracy (%)  ret. (%)     t
  modelo-a  after 383         50.91      0.24  0.31
  modelo-a before 237         89.45      6.15 10.50
  modelo-b  after 231         57.58      1.66  1.56
  modelo-b before 389         93.06      6.31 12.34
  modelo-c  after 103         69.90      3.35  4.04
  modelo-c before 517         90.91      6.46 11.70
modelo-pit  after 620         50.81      0.69  1.22

The returns are the ones documented in the data README, and the table is the key to the lesson. Before their declared cutoff, the three models get 89-93% right and following them yields an average 6.2-6.5% over 21 sessions with t>10t > 10: the return of knowing the future. Afterwards, modelo-a drops to 50.9% and 0.24% per trade (t=0.31t = 0.31). modelo-b and modelo-c stay somewhat above (57.6% and 69.9%), and the explanation is in the next section. The point-in-time model gets 50.8% right and earns 0.69% (t=1.22t = 1.22), less than being always long.

Fig. — Mean return of buying or selling for 21 sessions as each model answers, before and after its stated cutoff (420 price questions)

The declared cutoff is not the effective one

The cutoff date a provider publishes is a statement about its corpus, and a corpus can include later texts: updated pages, fine-tuning or alignment data, documents with the wrong date. In the data, modelo-b and modelo-c were trained on some text from after the cutoff they declare. The figure at the top plots accuracy by half-year of resolution:

IN [04]python
d["semestre"] = (d["fecha_resolucion"].dt.year.astype(str)
                 + np.where(d["fecha_resolucion"].dt.month <= 6,
                            "-1", "-2"))
sem = d.pivot_table(index="semestre", columns="modelo",
                    values="acierto", aggfunc="mean")
print((100 * sem).round(0).astype(int).to_string())
OUT [04]Accuracy by half-year of resolution
modelo    modelo-a  modelo-b  modelo-c  modelo-pit
semestre
2019-1          81        90        90          43
2019-2          88        86        86          47
2020-1          89        93        89          50
2020-2          90        93        91          47
2021-1          91        98        93          60
2021-2          76        93        90          59
2022-1          43        96        94          51
2022-2          59        94        94          56
2023-1          50        89        84          48
2023-2          49        64        91          47
2024-1          56        61        94          58
2024-2          56        54        92          50
2025-1          47        51        62          51
2025-2          49        51        69          44

The accuracy of modelo-b stays at 89% in the first half of 2023, when its declared cutoff is April 30, and only drops to 64% in the second half. If accuracy is high while the model remembers and low when it predicts, the date that best separates the two rates is an estimate of the effective cutoff. The cell searches for it by maximum likelihood among all month-ends, with two constant accuracy rates (a break in a binomial model):

IN [05]python
def ruptura(g):
    """Month-end that best separates two accuracy rates."""
    mejor = None
    for c in pd.date_range("2019-06-30", "2025-09-30", freq="ME"):
        a = g.loc[g["fecha_resolucion"] <= c, "acierto"]
        b = g.loc[g["fecha_resolucion"] > c, "acierto"]
        if min(len(a), len(b)) < 20:
            continue
        ll = 0.0
        for s in (a, b):
            p = np.clip(s.mean(), 1e-9, 1 - 1e-9)
            ll += s.sum() * np.log(p) + (~s).sum() * np.log(1 - p)
        if mejor is None or ll > mejor[1]:
            mejor = (c.date(), ll, a.mean(), b.mean())
    return mejor

for m in ("modelo-a", "modelo-b", "modelo-c"):
    c, _, pa, pb = ruptura(d[d["modelo"] == m])
    print(f"{m}: declared {corte[m].date()} · estimated {c} "
          f"· accuracy {pa:.0%} before and {pb:.0%} after")
OUT [05]Estimate the effective cutoff with a break in accuracy
modelo-a: declared 2021-09-30 · estimated 2021-09-30 · accuracy 89% before and 51% after
modelo-b: declared 2023-04-30 · estimated 2023-07-31 · accuracy 93% before and 55% after
modelo-c: declared 2024-10-31 · estimated 2024-11-30 · accuracy 91% before and 66% after

For modelo-a the estimate matches the declared cutoff. For modelo-b it comes out at 2023-07-31, three months after the declared one, and for modelo-c at 2024-11-30, one month after. The generator's ground truth (sesgo6_modelos_verdad.csv) gives effective cutoffs of 2023-09-30 and 2024-12-31: the break gets the direction right (the real cutoff is later than the declared one) but not the exact date, because in the gray zone the model gets 75% right, an intermediate value, and the few data points per month do not allow more precision. Practical conclusion: the effective cutoff is estimated, taken with a margin and checked with several probes.

Confidence gives memory away (in this data)

The simulated models state a confidence with each answer. In the data, remembering shows:

IN [06]python
alta = d["confianza"] >= 0.85
tabla = (d[d["modelo"] != "modelo-pit"]
         .assign(alta=alta)
         .groupby(["modelo", "zona"])
         .agg(confianza=("confianza", "mean"),
              alta=("alta", "mean")))
print((tabla * [1, 100]).round(2).to_string())
gris = d[(d["zona"] == "after") & alta & (d["modelo"] == "modelo-b")]
print("modelo-b, high confidence after the declared cutoff:", len(gris),
      "answers, resolved between",
      gris["fecha_resolucion"].min().date(),
      "and", gris["fecha_resolucion"].max().date(),
      f"· accuracy {gris['acierto'].mean():.0%}")
OUT [06]Confidence gives memory away (in this data)
                 confianza    alta
modelo   zona
modelo-a after        0.63    0.00
         before       0.92  100.00
modelo-b after        0.66    5.19
         before       0.92  100.00
modelo-c after        0.66    6.80
         before       0.92  100.00
modelo-b, high confidence after the declared cutoff: 12 answers, resolved between 2023-06-06 and 2023-09-27 · accuracy 58%

Before the cutoff, every answer has a confidence of 0.85 or more (mean 0.92); afterwards, the mean drops to 0.63-0.66 and only 5-7% exceed 0.85. The 12 answers of modelo-b with high confidence after its declared cutoff resolve between June and September 2023: the gray zone, inside its effective cutoff. Here confidence is a clean fingerprint of memory because the generator builds it that way. With real LLMs, verbalized confidence or token probabilities do not separate what is remembered from what is inferred nearly as well, and they should not be trusted as the only probe.

Honest evaluation

An honest evaluation of a model with a cutoff date needs four things: questions resolved after the effective cutoff with a margin, a trivial baseline (here, always answering “yes”, which is right 53.7% of the time because stocks go up more often than they go down), a confidence interval and enough questions.

IN [07]python
from scipy.stats import binomtest

verdad = pd.read_csv("sesgo6_modelos_verdad.csv")
efectivo = pd.to_datetime(verdad.set_index("modelo")
                          ["fecha_corte_efectiva"], errors="coerce")
base_si = (q["respuesta_correcta"] == "sí").mean()

def honesta(g, corte_efectivo, margen_dias=30):
    g = g[g["fecha_resolucion"]
          > corte_efectivo + pd.Timedelta(days=margen_dias)]
    k, n = int(g["acierto"].sum()), len(g)
    ic = binomtest(k, n).proportion_ci(0.95)
    p = binomtest(k, n, base_si, alternative="greater").pvalue
    return n, k / n, ic.low, ic.high, p

print(f"baseline: always answering “yes” is right {base_si:.1%}")
for m in ("modelo-a", "modelo-b", "modelo-c"):
    n, acc, lo, hi, p = honesta(d[d["modelo"] == m], efectivo[m])
    print(f"{m}: n = {n:3d} · accuracy {acc:.1%} "
          f"(95% CI: {lo:.1%}-{hi:.1%}) · p against “yes” = {p:.2f}")
n, acc, lo, hi, p = honesta(d[d["modelo"] == "modelo-pit"],
                            pd.Timestamp("2018-12-31"))
print(f"modelo-pit: n = {n:3d} · accuracy {acc:.1%} "
      f"(95% CI: {lo:.1%}-{hi:.1%}) · p against “yes” = {p:.2f}")
OUT [07]Honest evaluation: only what resolved after the effective cutoff
baseline: always answering “yes” is right 53.7%
modelo-a: n = 380 · accuracy 50.8% (95% CI: 45.6%-55.9%) · p against “yes” = 0.88
modelo-b: n = 185 · accuracy 55.1% (95% CI: 47.7%-62.4%) · p against “yes” = 0.38
modelo-c: n =  78 · accuracy 66.7% (95% CI: 55.1%-76.9%) · p against “yes” = 0.01
modelo-pit: n = 620 · accuracy 50.8% (95% CI: 46.8%-54.8%) · p against “yes” = 0.93

modelo-a and modelo-b cannot be told apart from always answering “yes” (p = 0.88 and 0.38), and neither can the point-in-time model (p = 0.93). modelo-c gets 66.7% right with p = 0.01. Skill? The generator's ground truth says that outside their memory the three get 52-55% right. It is a lucky streak in a small sample: 78 questions give an interval from 55% to 77%, and when three models are evaluated, one coming out significant by chance is not rare (the multiple testing problem of lesson 2.5.4). With real data there would be no published ground truth to refute it: only waiting for more questions to resolve.

How many would be needed? With the normal approximation of a one-sided test at 5% and 80% power:

n=(z1−αp0(1−p0)+z1−βp1(1−p1)p1−p0)2n = \left( \frac{z_{1-\alpha} \sqrt{p_0 (1 - p_0)} + z_{1-\beta} \sqrt{p_1 (1 - p_1)}}{p_1 - p_0} \right)^2
IN [08]python
from scipy.stats import norm

def n_necesario(p1, p0=0.5, alfa=0.05, potencia=0.8):
    za, zb = norm.ppf(1 - alfa), norm.ppf(potencia)
    num = za * np.sqrt(p0 * (1 - p0)) + zb * np.sqrt(p1 * (1 - p1))
    return int(np.ceil((num / (p1 - p0)) ** 2))

for p1 in (0.52, 0.55, 0.60):
    print(f"true accuracy {p1:.0%} against 50%: "
          f"{n_necesario(p1):5d} questions")
OUT [08]How many questions would be needed? Power of the test
true accuracy 52% against 50%:  3863 questions
true accuracy 55% against 50%:   617 questions
true accuracy 60% against 50%:   153 questions

Telling 55% from 50% takes 617 questions resolved after the effective cutoff; 52%, almost 4,000. A model with a recent cutoff has only a few months of honest history, so most evaluations of LLMs in market prediction lack the power to detect a realistic edge. That is the practical reason to prefer point-in-time models: they make it possible to use the whole history.

Dated questionsEffective cutoffCutoff + marginMaskAnswersBaseline and CIPower?
Mermaid source
flowchart TB
  A[Dated questions] --> B[Effective cutoff]
  B --> C[Cutoff + margin]
  C --> D[Mask]
  D --> E[Answers]
  E --> F[Baseline and CI]
  F --> G{Power?}

Point-in-time: one model per date

The root solution is for the model not to have read anything later than the date of each question. That is what modelo-pit does in the data, and what several recent papers propose with real models:

  • Time Machine GPT (Drinkall et al., 2024): a series of models trained only on text prior to each date, designed to have no knowledge of the future, released with their training data.
  • ChronoBERT and ChronoGPT (He, Lv, Manela and Wu, 2025): models trained only on the text available at each point in time, competitive with BERT on language benchmarks; in a return-prediction test with news, their real-time Sharpe ratios are comparable to those of a much larger Llama, which suggests that in that application the bias is moderate.
  • DatedGPT (Yan et al., 2026): twelve 1.3B-parameter models with strict annual cutoffs from 2013 to 2024; in their test with headlines, the models whose training covers the results period show a significant “look-ahead premium” against those whose training does not.

They are small models compared with commercial ones. The trade-off is clear: a point-in-time model loses capability and gains a complete history of honest backtesting. Lopez-Lira and Tang (2023), who studied whether ChatGPT predicts returns from headlines, took the other route: headlines after the model's knowledge cutoff.

Masking entities: necessary, not sufficient

Glasserman and Lin (2023) replace the company's name with a placeholder so the model does not use what it knows about it. With the bank's questions (the masking replaces Spanish text with Spanish placeholders, such as “la empresa A” (company A) and “la fecha t” (date t)):

IN [09]python
import re

def enmascarar(pregunta, nombre, ticker):
    t = pregunta.replace(nombre, "la empresa A")
    t = t.replace(f" ({ticker})", "").replace(ticker, "A")
    t = re.sub(r"\b(19|20)\d{2}-\d{2}-\d{2}\b", "la fecha t", t)
    t = re.sub(r"\bde (19|20)\d{2}\b", "del año t", t)
    # the price level also identifies: it is rescaled to 100
    return re.sub(r"cotiza a [\d.,]+ dólares",
                  "cotiza a 100 (precio reescalado)", t)

nombres = emp.set_index("ticker")["nombre_corto"]
for _, f in q.iloc[[0, 1]].iterrows():
    nombre = nombres[actual[f["ticker"]]]
    print(enmascarar(f["pregunta"], nombre, f["ticker"]))
OUT [09]Mask entities and dates in the prompt
A cierre de la fecha t, la empresa A cotiza a 100 (precio reescalado). ¿Cerrará por encima de ese precio dentro de 21 sesiones?
El consenso espera unas ventas de 4.563,2 millones de dólares de la empresa A en el cuarto trimestre del año t. ¿Las superará?

The function removes the name, the ticker and the dates, and rescales the price, because a specific price level ($793.88) also identifies a company and a day. But the second question keeps the sales consensus, 4,563.2 million dollars: a figure like that, together with the sector and the quarter, may be enough to recognize a real company. Every piece of data in the prompt is a clue.

The same goes for a real LLM. The course's local model (Qwen2.5-0.5B-Instruct, released on September 19, 2024 and trained on up to 18 trillion tokens according to its blog1Note 1 · Qwen blog, “Qwen2.5: A Party of Foundation Models!”, September 19, 2024 (https://qwenlm.github.io/blog/qwen2.5/, accessed 2026-10-23). It does not publish a knowledge cutoff date.) is small, but it remembers famous facts. Lehman Brothers went bankrupt on September 15, 20082Note 2 · Statement by Thomas C. Baxter Jr. (Federal Reserve Bank of New York) before the Financial Crisis Inquiry Commission, September 1, 2010 (https://www.newyorkfed.org/newsevents/speeches/2010/bax100901): on September 15, Lehman's board took the parent company into Chapter 11.. The prompts are in Spanish, except the last one:

IN [10]python
import torch
import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

transformers.logging.set_verbosity_error()
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(NOMBRE)
llm = AutoModelForCausalLM.from_pretrained(NOMBRE, dtype=torch.float32)

def pregunta(texto, n=12):
    x = tok.apply_chat_template([{"role": "user", "content": texto}],
                                add_generation_prompt=True,
                                return_tensors="pt", return_dict=True)
    with torch.inference_mode():
        o = llm.generate(**x, do_sample=False, max_new_tokens=n,
                         pad_token_id=tok.eos_token_id)
    return tok.decode(o[0, x["input_ids"].shape[1]:],
                      skip_special_tokens=True)

PROMPTS = {
    "with name": "¿En qué año quebró Lehman Brothers? "
                 "Responde solo con el año.",
    "no name, with context": "Un gran banco de inversión de Nueva "
        "York se declaró en quiebra un 15 de septiembre, en plena "
        "crisis de las hipotecas subprime. ¿En qué año? "
        "Responde solo con el año.",
    "no name, no context": "Un gran banco de inversión de Nueva "
        "York se declaró en quiebra un 15 de septiembre. ¿En qué año?"
        " Responde solo con el año.",
    "cutoff, in Spanish": "¿Cuál es tu fecha de corte de conocimiento?"
                          " Responde solo con el mes y el año.",
    "cutoff, in English": "What is your knowledge cutoff date? "
                          "Answer with month and year only.",
}
for k, p in PROMPTS.items():
    print(f"{k:25s} -> {pregunta(p)!r}")
OUT [10]A real LLM remembers: Qwen2.5-0.5B and an event from 2008
with name                 -> '2008'
no name, with context     -> '2008'
no name, no context       -> '2023'
cutoff, in Spanish        -> 'Marzo 2019'
cutoff, in English        -> 'October 2023'

With the name, it answers “2008”. Without the name but with the context of the mortgage crisis (“Un gran banco de inversión de Nueva York se declaró en quiebra un 15 de septiembre, en plena crisis de las hipotecas subprime”, “A large New York investment bank filed for bankruptcy on a September 15, in the middle of the subprime mortgage crisis”), also “2008”: the context identifies the event. Without any context, “2023”, and it gets it wrong. And the cutoff date the model itself states is not a source: “Marzo 2019” (March 2019) if asked in Spanish and “October 2023” in English. A 0.5B-parameter model remembers little; commercial models, with hundreds of times more parameters, remember much more, and that is why Lopez-Lira, Tang and Zhu (2025) find that masking names and dates does not stop them from reconstructing the data.

Rules for a backtest with an LLM

  • Pin the exact version of the model (not an alias that changes) and record its declared cutoff and the source of that date.
  • Estimate the effective cutoff with probes and leave a margin; treat the declared one as a lower bound.
  • Evaluate only with questions resolved after the effective cutoff, or use a point-in-time model.
  • Mask names, tickers, dates and identifying figures, knowing that it is not enough.
  • Compare with a trivial baseline (always “yes”, always long, a model without text) with a confidence interval and a paired test, and compute the power before drawing conclusions.
  • Count the models and prompts tried: choosing the best of several on the same sample is selection (2.5.4).
  • Your own fine-tuning has a date too: a model fine-tuned on news from 2025 (lesson 6.2.4) cannot be backtested in 2022.

Exercises

  1. Repeat the honest evaluation of modelo-b using the declared cutoff instead of the effective one, with a margin of 0 days. How much is the accuracy inflated?
Solution
IN [11]python
b = d[d["modelo"] == "modelo-b"]
for nombre, c, m in [("declared, no margin", corte["modelo-b"], 0),
                     ("effective + 30 days", efectivo["modelo-b"], 30)]:
    n, acc, lo, hi, p = honesta(b, c, margen_dias=m)
    print(f"{nombre:22s} n = {n:3d} · accuracy {acc:.1%} "
          f"(CI {lo:.1%}-{hi:.1%}) · p = {p:.3f}")
OUT [11]Evaluation with the declared cutoff (no margin)
declared, no margin    n = 231 · accuracy 57.6% (CI 50.9%-64.0%) · p = 0.133
effective + 30 days    n = 185 · accuracy 55.1% (CI 47.7%-62.4%) · p = 0.377

With the declared cutoff, the gray-zone questions, which the model partly remembers, enter the evaluation. Accuracy rises and the p-value falls: the evaluation seems to show a skill that is memory of the months in which the corpus already covered the answer.

  1. Extend enmascarar so that the sales consensus is also rescaled (to 100) and apply it to the 200 sales questions. Is there any figure left that could identify the company?
Solution
IN [12]python
def enmascarar2(pregunta, nombre, ticker):
    t = enmascarar(pregunta, nombre, ticker)
    return re.sub(r"[\d.]+,\d millones de dólares",
                  "100 (consenso reescalado)", t)

ventas = q[q["tipo"] == "ventas_frente_a_consenso"]
textos = [enmascarar2(f["pregunta"], nombres[actual[f["ticker"]]],
                      f["ticker"]) for _, f in ventas.iterrows()]
print(textos[0])
resto = {n for t in textos for n in re.findall(r"\d[\d.,]*", t)}
print("figures left in the 200:", sorted(resto))
OUT [12]Mask the consensus too
El consenso espera unas ventas de 100 (consenso reescalado) de la empresa A en el cuarto trimestre del año t. ¿Las superará?
figures left in the 200: ['100']

The year has become “del año t” (of year t), the amount 100 and the company “la empresa A” (company A): the only figure left in the 200 questions is the 100 of the rescaling. Even so, the text keeps information: the quarter (“cuarto”, fourth) and the question itself. With real data you would also have to check that the sector, the size or a characteristic phrase do not identify the issuer, and measure the effect of the masking by comparing the answers with and without it, as Glasserman and Lin (2023) do.

  1. A provider offers a financial LLM with a declared cutoff of June 2024 and a backtest from 2015 to 2025 with a Sharpe of 3. What would you ask for before believing it?
Solution

The part of the backtest after the effective cutoff, separately: from 2015 to mid-2024 the model could have read the results. I would ask how that cutoff was estimated (and not only the declared one), the exact version of the model used at each date and whether it changed, the number of models, prompts and variants tried before choosing the one presented, and the result of a trivial baseline over the same period with its confidence interval. With a year and a bit of honest history, a high Sharpe has an enormous standard error (lesson 2.5.4): the reasonable thing is to wait to accumulate out-of-sample results or to ask for a point-in-time version of the model.

Footnotes

  1. Qwen blog, “Qwen2.5: A Party of Foundation Models!”, September 19, 2024 (https://qwenlm.github.io/blog/qwen2.5/, accessed 2026-10-23). It does not publish a knowledge cutoff date. ↩

  2. Statement by Thomas C. Baxter Jr. (Federal Reserve Bank of New York) before the Financial Crisis Inquiry Commission, September 1, 2010 (https://www.newyorkfed.org/newsevents/speeches/2010/bax100901): on September 15, Lehman's board took the parent company into Chapter 11. ↩

Each answer is folded under its question

Self-assessment

  1. An LLM with a declared cutoff of September 2021 gets 89% of 237 questions right on whether a stock will rise over the following month, all resolved before that date, and 51% of the 383 resolved after it. What does the 89% measure?

    • AWhat the model remembers from its training corpus, not its ability to predict
    • BThe model's skill in calm markets
    • CThat the model has degraded over time
    • DA good model that loses skill because of a regime change in 2022
    Show answer

    Answer A. What the model remembers from its training corpus, not its ability to predict

    The answers were already in the text it was trained on. After the cutoff, accuracy falls to that of a coin (51%), which is what the model can do without memory. A regime change does not explain a 38-point drop right at the cutoff date.

  2. modelo-b declares a cutoff of 2023-04-30, but its accuracy stays high until the summer and the estimated break falls on 2023-07-31. How should the declared date be treated?

    • AAs exact, because the provider publishes it
    • BAs a lower bound: the corpus may include some later text, so the honest evaluation starts after the estimated effective cutoff, with a margin
    • CAs an upper bound: the model knows less than it claims
    • DIt does not matter, because the difference is three months
    Show answer

    Answer B. As a lower bound: the corpus may include some later text, so the honest evaluation starts after the estimated effective cutoff, with a margin

    In the data, modelo-b was trained on text up to 2023-09-30. The questions in the gray zone get 76% right. With real data there is no published ground truth: the cutoff is estimated with the accuracy itself or with memory probes, and a margin is left.

  3. Outside its cutoff, modelo-c gets 66.7% of 78 questions right (95% CI: 55%-77%, p = 0.01 against always answering “yes”). The generator's ground truth says that outside their memory the models get 52-55% right. What happened?

    • AThe model has a real skill the generator does not capture
    • BThere is an error in the confidence interval
    • CThe model is still remembering
    • DA lucky streak in a small sample: with several models evaluated, one coming out significant by chance is not rare, and 78 questions give an interval more than 20 points wide
    Show answer

    Answer D. A lucky streak in a small sample: with several models evaluated, one coming out significant by chance is not rare, and 78 questions give an interval more than 20 points wide

    It is the multiple testing problem of level 2 in another guise. To tell 55% from 50% with 80% power takes about 617 questions resolved after the cutoff, far more than there are.

  4. To keep the model from recognizing the company, its name is removed from the prompt. Qwen2.5-0.5B still answers “2008” to “a large New York investment bank filed for bankruptcy on a September 15, in the middle of the subprime mortgage crisis” (asked in Spanish). What does it show?

    • AThat the context identifies the event even without the name: masking entities reduces the bias, it does not eliminate it
    • BThat the model is random
    • CThat the prompt has to be translated into English
    • DThat the model does not know Lehman Brothers
    Show answer

    Answer A. That the context identifies the event even without the name: masking entities reduces the bias, it does not eliminate it

    Without the crisis context, the model answers “2023” and gets it wrong. Lopez-Lira, Tang and Zhu (2025) document the same with large models: masking names and dates does not stop them from reconstructing the fact from a little context.

  5. What is the most reliable way to backtest a strategy that uses an LLM?

    • AAsk the model in the prompt not to use information after the question's date
    • BUse the most recent model, which is the best
    • CUse only questions resolved after the model's effective cutoff, or a point-in-time model trained only on text prior to each date, and compare with a trivial baseline with a confidence interval
    • DMask the ticker
    Show answer

    Answer C. Use only questions resolved after the model's effective cutoff, or a point-in-time model trained only on text prior to each date, and compare with a trivial baseline with a confidence interval

    Instructions do not erase memory (Lopez-Lira, Tang and Zhu, 2025; Sarkar and Vafa, 2025). Point-in-time models (TiMaGPT, ChronoBERT and ChronoGPT, DatedGPT) or post-cutoff evaluation are the routes the literature proposes.

Key points

  • An LLM remembers what happened before its cutoff: in the data, the three models get 89-93% of what was resolved before their declared cutoff right, and following their answers would earn an average 6.2-6.5% over 21 sessions (t > 10); afterwards, between 51% and 70%.
  • The declared cutoff is not enough: the effective one may be later. A break in accuracy estimates it (modelo-b: declared 2023-04-30, estimated 2023-07-31, ground truth 2023-09-30), and the honest evaluation starts after it, with a margin.
  • After the effective cutoff, no model clearly beats always answering “yes” (53.7%) except modelo-c with 78 questions, a streak the generator's ground truth refutes. Detecting 55% against 50% takes about 617 questions.
  • Masking the name and the date reduces recognition but does not eliminate it: the context, the figures and the style identify the event (Qwen2.5-0.5B answers “2008” to a description of the Lehman bankruptcy without the name).
  • The cutoff date the model itself states is not a source: Qwen2.5-0.5B answers “Marzo 2019” (March 2019) in Spanish and “October 2023” in English.
  • The defenses in the literature: evaluate only after the cutoff (Lopez-Lira and Tang), anonymize (Glasserman and Lin) and point-in-time models (TiMaGPT, ChronoBERT, DatedGPT).

Resources

Data