--- title: "6.2.5 Look-ahead bias in LLMs" description: "An LLM remembers what happened before its cutoff: how to detect it, estimate the effective cutoff and run an honest backtest with point-in-time models." url: "https://pythonparatrading.com/en/python-trading-course/level-6/look-ahead-bias-in-llms/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/look-ahead-bias-in-llms.png" --- # 6.2.5 · Look-ahead bias in LLMs: the model that already knows the future > An LLM trained on text up to 2024 “predicts” what happened in 2022 very well: not because it can predict, but because it has read it. This lesson measures the effect with four simulated models and known answers, shows how to estimate the effective cutoff when the declared one is not reliable, computes how many questions an honest evaluation needs and checks with a real LLM that removing the company's name is not enough. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.2 · LLMs for research and trading - Duration: 60 min - Prerequisites: [2.5.8 · “Honest backtest” checklist before moving to paper trading](https://pythonparatrading.com/en/python-trading-course/level-2/honest-backtest-checklist/), [5.2.2 · Point-in-time fundamentals, macro and alternative data](https://pythonparatrading.com/en/python-trading-course/level-5/point-in-time-fundamentals-macro-alternative-data/) ## By the end of this lesson - Explain why a backtest with an LLM over dates before its training cutoff measures memory and not the ability to predict. - Measure the accuracy and the return of following a model before and after its cutoff, and estimate the effective cutoff with a break in accuracy. - Design an honest evaluation: questions resolved after the effective cutoff with a margin, a confidence interval, a trivial baseline and enough power. - Know the defenses proposed in the literature (masking entities, point-in-time models, post-cutoff questions) and their limits. ## A backtest that measures memory Look-ahead bias is an old acquaintance of the course: using at a given date a piece of data that was not known then (lessons 2.5.8 and 5.2.2). With an LLM it shows up in a new form that is harder to see. The model was trained on trillions of tokens of text up to a **cutoff date**: news, reports, forums, commented quotes. If, in a backtest dated 2021, it is asked whether a stock will rise the following month, and the model was trained on text from 2023, the answer may be in its memory. The backtest looks spectacular and measures nothing that can be repeated. The problem is real and documented. Glasserman and Lin (2023) separate two effects when scoring headlines with GPT: the look-ahead effect (the model knows how the price moved afterwards) and the “distraction” effect (what it knows about the company interferes with the tone of the text), and propose anonymizing the companies. Lopez-Lira, Tang and Zhu (2025) show that LLMs remember exact values of economic series from before their cutoff and none after it, and that neither asking them to respect the date nor masking names and dates prevents it. Sarkar and Vafa (2025) propose direct tests based on events that were unpredictable with the information available at the date and find the bias in earnings calls. Since with a real model you do not know what it has read, this lesson uses the level 6 synthetic question bank, with known answers and four **simulated** models (there is no LLM behind them; their answers follow rules published in `sesgo6_modelos_verdad.csv`), and at the end checks the mechanism with the course's local LLM. ```mermaid flowchart LR A[Corpus up to cutoff] --> B[Model] C[Question at date t] --> B B --> D{Before the cutoff?} D -->|yes| E[Remembers] D -->|no| F[Predicts] ``` ## The data: 620 questions and four models Each question has a date (when it would be asked in the backtest), a resolution date (the first day the answer is known) and the correct answer, computed with the level's synthetic prices and sales. There are two types: whether a stock will close above its price 21 sessions later and whether the quarter's sales will beat the consensus. Three models declare a cutoff date; `modelo-pit` is **point-in-time**: for each question, its cutoff is the day before. The question texts are in Spanish, like the rest of the level 6 corpus, and stay as they are; the answers in the data are `sí` (yes) and `no`. ```python import numpy as np import pandas as pd q = pd.read_csv("sesgo6_preguntas.csv", parse_dates=["fecha_pregunta", "fecha_resolucion"]) resp = pd.read_csv("sesgo6_respuestas.csv") modelos = pd.read_csv("sesgo6_modelos.csv") print(modelos.to_string(index=False)) print(q["tipo"].value_counts().to_string()) ej = q.iloc[0] print(ej["fecha_pregunta"].date(), "·", ej["pregunta"]) print("resolves on", ej["fecha_resolucion"].date(), "· answer:", ej["respuesta_correcta"]) ``` ```output modelo fecha_corte_declarada modelo-a 2021-09-30 modelo-b 2023-04-30 modelo-c 2024-10-31 modelo-pit día anterior a cada pregunta tipo precio_21_sesiones 420 ventas_frente_a_consenso 200 2019-03-01 · A cierre de 2019-03-01, Sofsa Software (SOFE) cotiza a 793,88 dólares. ¿Cerrará por encima de ese precio dentro de 21 sesiones? resolves on 2019-04-01 · answer: no ``` ## The naive backtest The simplest strategy: buy the stock if the model says “yes” and sell it if it says “no”, for 21 sessions. The questions use the ticker in force at their date and the prices, the current one: 11 questions are about companies that changed ticker later, the same entity-mapping problem as lesson 6.7.5. ```python precios = pd.read_parquet("acciones6_precios.parquet") cierres = precios.pivot(index="fecha", columns="ticker", values="cierre") # questions use the ticker at their date; prices, the current one hist = pd.read_csv("acciones6_tickers.csv") emp = pd.read_csv("acciones6_empresas.csv") actual = (hist.merge(emp[["id_empresa", "ticker"]], on="id_empresa", suffixes=("", "_hoy")) .set_index("ticker")["ticker_hoy"]) print("questions with an old ticker:", int((q["ticker"] != q["ticker"].map(actual)).sum())) i = cierres.index.get_indexer(q["fecha_pregunta"]) cols = cierres.columns.get_indexer(q["ticker"].map(actual)) valores = cierres.to_numpy() q["r21"] = valores[i + 21, cols] / valores[i, cols] - 1 d = resp.merge(q, on="id_pregunta") d["acierto"] = d["respuesta"] == d["respuesta_correcta"] d["pnl"] = np.where(d["respuesta"] == "sí", 1, -1) * d["r21"] precio = d["tipo"] == "precio_21_sesiones" resumen = pd.DataFrame({ "accuracy (%)": 100 * d.groupby("modelo")["acierto"].mean(), "21-sess. ret. (%)": 100 * d[precio].groupby("modelo")["pnl"].mean()}) print(resumen.round(1).to_string()) largo = q.loc[q["tipo"] == "precio_21_sesiones", "r21"].mean() print(f"always long: {100 * largo:.1f} %") ``` ```output questions with an old ticker: 11 accuracy (%) 21-sess. ret. (%) modelo modelo-a 65.6 2.5 modelo-b 79.8 4.6 modelo-c 87.4 5.9 modelo-pit 50.8 0.7 always long: 1.1 % ``` Over the whole sample, the most recent model looks the best: `modelo-c` gets 87.4% right and would earn an average 5.9% on each 21-session trade, against 1.1% for being always long. `modelo-a`, the oldest, stays at 65.6%, and the point-in-time one at 50.8%, a coin flip. A researcher comparing models this way would choose the most recent one and believe they had an exceptional strategy. What they have measured is how much of the sample falls before each model's cutoff. ## Before and after the cutoff ```python corte = pd.to_datetime(modelos.set_index("modelo") ["fecha_corte_declarada"], errors="coerce") d["corte"] = d["modelo"].map(corte) d["zona"] = np.where(d["fecha_resolucion"] <= d["corte"], "before", "after") def t_media(x): return x.mean() / x.std(ddof=1) * np.sqrt(len(x)) filas = [] for (m, z), g in d.groupby(["modelo", "zona"]): p = g[g["tipo"] == "precio_21_sesiones"]["pnl"] filas.append({"modelo": m, "zona": z, "n": len(g), "accuracy (%)": 100 * g["acierto"].mean(), "ret. (%)": 100 * p.mean(), "t": t_media(p)}) print(pd.DataFrame(filas).round(2).to_string(index=False)) ``` ```output modelo zona n accuracy (%) ret. (%) t modelo-a after 383 50.91 0.24 0.31 modelo-a before 237 89.45 6.15 10.50 modelo-b after 231 57.58 1.66 1.56 modelo-b before 389 93.06 6.31 12.34 modelo-c after 103 69.90 3.35 4.04 modelo-c before 517 90.91 6.46 11.70 modelo-pit after 620 50.81 0.69 1.22 ``` The returns are the ones documented in the data README, and the table is the key to the lesson. Before their declared cutoff, the three models get 89-93% right and following them yields an average 6.2-6.5% over 21 sessions with $t > 10$: the return of knowing the future. Afterwards, `modelo-a` drops to 50.9% and 0.24% per trade ($t = 0.31$). `modelo-b` and `modelo-c` stay somewhat above (57.6% and 69.9%), and the explanation is in the next section. The point-in-time model gets 50.8% right and earns 0.69% ($t = 1.22$), less than being always long. *Fig. — Mean return of buying or selling for 21 sessions as each model answers, before and after its stated cutoff (420 price questions).* Horizontal bars. Before their cutoff, following models A, B and C yields 6.15%, 6.31% and 6.46% per trade (A highlighted). After it, 0.24%, 1.66% and 3.35%. The point-in-time model yields 0.69% and always being long 1.15%. ## The declared cutoff is not the effective one The cutoff date a provider publishes is a statement about its corpus, and a corpus can include later texts: updated pages, fine-tuning or alignment data, documents with the wrong date. In the data, `modelo-b` and `modelo-c` were trained on some text from after the cutoff they declare. The figure at the top plots accuracy by half-year of resolution: ```python d["semestre"] = (d["fecha_resolucion"].dt.year.astype(str) + np.where(d["fecha_resolucion"].dt.month <= 6, "-1", "-2")) sem = d.pivot_table(index="semestre", columns="modelo", values="acierto", aggfunc="mean") print((100 * sem).round(0).astype(int).to_string()) ``` ```output modelo modelo-a modelo-b modelo-c modelo-pit semestre 2019-1 81 90 90 43 2019-2 88 86 86 47 2020-1 89 93 89 50 2020-2 90 93 91 47 2021-1 91 98 93 60 2021-2 76 93 90 59 2022-1 43 96 94 51 2022-2 59 94 94 56 2023-1 50 89 84 48 2023-2 49 64 91 47 2024-1 56 61 94 58 2024-2 56 54 92 50 2025-1 47 51 62 51 2025-2 49 51 69 44 ``` The accuracy of `modelo-b` stays at 89% in the first half of 2023, when its declared cutoff is April 30, and only drops to 64% in the second half. If accuracy is high while the model remembers and low when it predicts, the date that best separates the two rates is an estimate of the effective cutoff. The cell searches for it by maximum likelihood among all month-ends, with two constant accuracy rates (a break in a binomial model): ```python def ruptura(g): """Month-end that best separates two accuracy rates.""" mejor = None for c in pd.date_range("2019-06-30", "2025-09-30", freq="ME"): a = g.loc[g["fecha_resolucion"] <= c, "acierto"] b = g.loc[g["fecha_resolucion"] > c, "acierto"] if min(len(a), len(b)) < 20: continue ll = 0.0 for s in (a, b): p = np.clip(s.mean(), 1e-9, 1 - 1e-9) ll += s.sum() * np.log(p) + (~s).sum() * np.log(1 - p) if mejor is None or ll > mejor[1]: mejor = (c.date(), ll, a.mean(), b.mean()) return mejor for m in ("modelo-a", "modelo-b", "modelo-c"): c, _, pa, pb = ruptura(d[d["modelo"] == m]) print(f"{m}: declared {corte[m].date()} · estimated {c} " f"· accuracy {pa:.0%} before and {pb:.0%} after") ``` ```output modelo-a: declared 2021-09-30 · estimated 2021-09-30 · accuracy 89% before and 51% after modelo-b: declared 2023-04-30 · estimated 2023-07-31 · accuracy 93% before and 55% after modelo-c: declared 2024-10-31 · estimated 2024-11-30 · accuracy 91% before and 66% after ``` For `modelo-a` the estimate matches the declared cutoff. For `modelo-b` it comes out at 2023-07-31, three months after the declared one, and for `modelo-c` at 2024-11-30, one month after. The generator's ground truth (`sesgo6_modelos_verdad.csv`) gives effective cutoffs of 2023-09-30 and 2024-12-31: the break gets the direction right (the real cutoff is later than the declared one) but not the exact date, because in the gray zone the model gets 75% right, an intermediate value, and the few data points per month do not allow more precision. Practical conclusion: the effective cutoff is estimated, taken with a margin and checked with several probes. ## Confidence gives memory away (in this data) The simulated models state a confidence with each answer. In the data, remembering shows: ```python alta = d["confianza"] >= 0.85 tabla = (d[d["modelo"] != "modelo-pit"] .assign(alta=alta) .groupby(["modelo", "zona"]) .agg(confianza=("confianza", "mean"), alta=("alta", "mean"))) print((tabla * [1, 100]).round(2).to_string()) gris = d[(d["zona"] == "after") & alta & (d["modelo"] == "modelo-b")] print("modelo-b, high confidence after the declared cutoff:", len(gris), "answers, resolved between", gris["fecha_resolucion"].min().date(), "and", gris["fecha_resolucion"].max().date(), f"· accuracy {gris['acierto'].mean():.0%}") ``` ```output confianza alta modelo zona modelo-a after 0.63 0.00 before 0.92 100.00 modelo-b after 0.66 5.19 before 0.92 100.00 modelo-c after 0.66 6.80 before 0.92 100.00 modelo-b, high confidence after the declared cutoff: 12 answers, resolved between 2023-06-06 and 2023-09-27 · accuracy 58% ``` Before the cutoff, every answer has a confidence of 0.85 or more (mean 0.92); afterwards, the mean drops to 0.63-0.66 and only 5-7% exceed 0.85. The 12 answers of `modelo-b` with high confidence after its declared cutoff resolve between June and September 2023: the gray zone, inside its effective cutoff. Here confidence is a clean fingerprint of memory because the generator builds it that way. With real LLMs, verbalized confidence or token probabilities do not separate what is remembered from what is inferred nearly as well, and they should not be trusted as the only probe. ## Honest evaluation An honest evaluation of a model with a cutoff date needs four things: questions resolved after the **effective** cutoff with a margin, a trivial baseline (here, always answering “yes”, which is right 53.7% of the time because stocks go up more often than they go down), a confidence interval and enough questions. ```python from scipy.stats import binomtest verdad = pd.read_csv("sesgo6_modelos_verdad.csv") efectivo = pd.to_datetime(verdad.set_index("modelo") ["fecha_corte_efectiva"], errors="coerce") base_si = (q["respuesta_correcta"] == "sí").mean() def honesta(g, corte_efectivo, margen_dias=30): g = g[g["fecha_resolucion"] > corte_efectivo + pd.Timedelta(days=margen_dias)] k, n = int(g["acierto"].sum()), len(g) ic = binomtest(k, n).proportion_ci(0.95) p = binomtest(k, n, base_si, alternative="greater").pvalue return n, k / n, ic.low, ic.high, p print(f"baseline: always answering “yes” is right {base_si:.1%}") for m in ("modelo-a", "modelo-b", "modelo-c"): n, acc, lo, hi, p = honesta(d[d["modelo"] == m], efectivo[m]) print(f"{m}: n = {n:3d} · accuracy {acc:.1%} " f"(95% CI: {lo:.1%}-{hi:.1%}) · p against “yes” = {p:.2f}") n, acc, lo, hi, p = honesta(d[d["modelo"] == "modelo-pit"], pd.Timestamp("2018-12-31")) print(f"modelo-pit: n = {n:3d} · accuracy {acc:.1%} " f"(95% CI: {lo:.1%}-{hi:.1%}) · p against “yes” = {p:.2f}") ``` ```output baseline: always answering “yes” is right 53.7% modelo-a: n = 380 · accuracy 50.8% (95% CI: 45.6%-55.9%) · p against “yes” = 0.88 modelo-b: n = 185 · accuracy 55.1% (95% CI: 47.7%-62.4%) · p against “yes” = 0.38 modelo-c: n = 78 · accuracy 66.7% (95% CI: 55.1%-76.9%) · p against “yes” = 0.01 modelo-pit: n = 620 · accuracy 50.8% (95% CI: 46.8%-54.8%) · p against “yes” = 0.93 ``` `modelo-a` and `modelo-b` cannot be told apart from always answering “yes” (p = 0.88 and 0.38), and neither can the point-in-time model (p = 0.93). `modelo-c` gets 66.7% right with p = 0.01. Skill? The generator's ground truth says that outside their memory the three get 52-55% right. It is a lucky streak in a small sample: 78 questions give an interval from 55% to 77%, and when three models are evaluated, one coming out significant by chance is not rare (the multiple testing problem of lesson 2.5.4). With real data there would be no published ground truth to refute it: only waiting for more questions to resolve. How many would be needed? With the normal approximation of a one-sided test at 5% and 80% power: $$ n = \left( \frac{z_{1-\alpha} \sqrt{p_0 (1 - p_0)} + z_{1-\beta} \sqrt{p_1 (1 - p_1)}}{p_1 - p_0} \right)^2 $$ ```python from scipy.stats import norm def n_necesario(p1, p0=0.5, alfa=0.05, potencia=0.8): za, zb = norm.ppf(1 - alfa), norm.ppf(potencia) num = za * np.sqrt(p0 * (1 - p0)) + zb * np.sqrt(p1 * (1 - p1)) return int(np.ceil((num / (p1 - p0)) ** 2)) for p1 in (0.52, 0.55, 0.60): print(f"true accuracy {p1:.0%} against 50%: " f"{n_necesario(p1):5d} questions") ``` ```output true accuracy 52% against 50%: 3863 questions true accuracy 55% against 50%: 617 questions true accuracy 60% against 50%: 153 questions ``` Telling 55% from 50% takes 617 questions resolved **after** the effective cutoff; 52%, almost 4,000. A model with a recent cutoff has only a few months of honest history, so most evaluations of LLMs in market prediction lack the power to detect a realistic edge. That is the practical reason to prefer point-in-time models: they make it possible to use the whole history. ```mermaid flowchart TB A[Dated questions] --> B[Effective cutoff] B --> C[Cutoff + margin] C --> D[Mask] D --> E[Answers] E --> F[Baseline and CI] F --> G{Power?} ``` ## Point-in-time: one model per date The root solution is for the model not to have read anything later than the date of each question. That is what `modelo-pit` does in the data, and what several recent papers propose with real models: - **Time Machine GPT** (Drinkall et al., 2024): a series of models trained only on text prior to each date, designed to have no knowledge of the future, released with their training data. - **ChronoBERT and ChronoGPT** (He, Lv, Manela and Wu, 2025): models trained only on the text available at each point in time, competitive with BERT on language benchmarks; in a return-prediction test with news, their real-time Sharpe ratios are comparable to those of a much larger Llama, which suggests that in that application the bias is moderate. - **DatedGPT** (Yan et al., 2026): twelve 1.3B-parameter models with strict annual cutoffs from 2013 to 2024; in their test with headlines, the models whose training covers the results period show a significant “look-ahead premium” against those whose training does not. They are small models compared with commercial ones. The trade-off is clear: a point-in-time model loses capability and gains a complete history of honest backtesting. Lopez-Lira and Tang (2023), who studied whether ChatGPT predicts returns from headlines, took the other route: headlines after the model's knowledge cutoff. ## Masking entities: necessary, not sufficient Glasserman and Lin (2023) replace the company's name with a placeholder so the model does not use what it knows about it. With the bank's questions (the masking replaces Spanish text with Spanish placeholders, such as “la empresa A” (company A) and “la fecha t” (date t)): ```python import re def enmascarar(pregunta, nombre, ticker): t = pregunta.replace(nombre, "la empresa A") t = t.replace(f" ({ticker})", "").replace(ticker, "A") t = re.sub(r"\b(19|20)\d{2}-\d{2}-\d{2}\b", "la fecha t", t) t = re.sub(r"\bde (19|20)\d{2}\b", "del año t", t) # the price level also identifies: it is rescaled to 100 return re.sub(r"cotiza a [\d.,]+ dólares", "cotiza a 100 (precio reescalado)", t) nombres = emp.set_index("ticker")["nombre_corto"] for _, f in q.iloc[[0, 1]].iterrows(): nombre = nombres[actual[f["ticker"]]] print(enmascarar(f["pregunta"], nombre, f["ticker"])) ``` ```output A cierre de la fecha t, la empresa A cotiza a 100 (precio reescalado). ¿Cerrará por encima de ese precio dentro de 21 sesiones? El consenso espera unas ventas de 4.563,2 millones de dólares de la empresa A en el cuarto trimestre del año t. ¿Las superará? ``` The function removes the name, the ticker and the dates, and rescales the price, because a specific price level ($793.88) also identifies a company and a day. But the second question keeps the sales consensus, 4,563.2 million dollars: a figure like that, together with the sector and the quarter, may be enough to recognize a real company. Every piece of data in the prompt is a clue. The same goes for a real LLM. The course's local model (Qwen2.5-0.5B-Instruct, released on September 19, 2024 and trained on up to 18 trillion tokens according to its blog[^1]) is small, but it remembers famous facts. Lehman Brothers went bankrupt on September 15, 2008[^2]. The prompts are in Spanish, except the last one: ```python import torch import transformers from transformers import AutoModelForCausalLM, AutoTokenizer transformers.logging.set_verbosity_error() torch.manual_seed(0) torch.use_deterministic_algorithms(True) NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct" tok = AutoTokenizer.from_pretrained(NOMBRE) llm = AutoModelForCausalLM.from_pretrained(NOMBRE, dtype=torch.float32) def pregunta(texto, n=12): x = tok.apply_chat_template([{"role": "user", "content": texto}], add_generation_prompt=True, return_tensors="pt", return_dict=True) with torch.inference_mode(): o = llm.generate(**x, do_sample=False, max_new_tokens=n, pad_token_id=tok.eos_token_id) return tok.decode(o[0, x["input_ids"].shape[1]:], skip_special_tokens=True) PROMPTS = { "with name": "¿En qué año quebró Lehman Brothers? " "Responde solo con el año.", "no name, with context": "Un gran banco de inversión de Nueva " "York se declaró en quiebra un 15 de septiembre, en plena " "crisis de las hipotecas subprime. ¿En qué año? " "Responde solo con el año.", "no name, no context": "Un gran banco de inversión de Nueva " "York se declaró en quiebra un 15 de septiembre. ¿En qué año?" " Responde solo con el año.", "cutoff, in Spanish": "¿Cuál es tu fecha de corte de conocimiento?" " Responde solo con el mes y el año.", "cutoff, in English": "What is your knowledge cutoff date? " "Answer with month and year only.", } for k, p in PROMPTS.items(): print(f"{k:25s} -> {pregunta(p)!r}") ``` ```output with name -> '2008' no name, with context -> '2008' no name, no context -> '2023' cutoff, in Spanish -> 'Marzo 2019' cutoff, in English -> 'October 2023' ``` With the name, it answers “2008”. Without the name but with the context of the mortgage crisis (“Un gran banco de inversión de Nueva York se declaró en quiebra un 15 de septiembre, en plena crisis de las hipotecas subprime”, “A large New York investment bank filed for bankruptcy on a September 15, in the middle of the subprime mortgage crisis”), also “2008”: the context identifies the event. Without any context, “2023”, and it gets it wrong. And the cutoff date the model itself states is not a source: “Marzo 2019” (March 2019) if asked in Spanish and “October 2023” in English. A 0.5B-parameter model remembers little; commercial models, with hundreds of times more parameters, remember much more, and that is why Lopez-Lira, Tang and Zhu (2025) find that masking names and dates does not stop them from reconstructing the data. ## Rules for a backtest with an LLM - Pin the **exact version** of the model (not an alias that changes) and record its declared cutoff and the source of that date. - Estimate the **effective cutoff** with probes and leave a margin; treat the declared one as a lower bound. - Evaluate only with questions **resolved after** the effective cutoff, or use a point-in-time model. - **Mask** names, tickers, dates and identifying figures, knowing that it is not enough. - Compare with a **trivial baseline** (always “yes”, always long, a model without text) with a confidence interval and a paired test, and compute the **power** before drawing conclusions. - Count the models and prompts tried: choosing the best of several on the same sample is selection (2.5.4). - Your own fine-tuning has a date too: a model fine-tuned on news from 2025 (lesson 6.2.4) cannot be backtested in 2022. ## Exercises 1. Repeat the honest evaluation of `modelo-b` using the **declared** cutoff instead of the effective one, with a margin of 0 days. How much is the accuracy inflated? **Solution.** ```python b = d[d["modelo"] == "modelo-b"] for nombre, c, m in [("declared, no margin", corte["modelo-b"], 0), ("effective + 30 days", efectivo["modelo-b"], 30)]: n, acc, lo, hi, p = honesta(b, c, margen_dias=m) print(f"{nombre:22s} n = {n:3d} · accuracy {acc:.1%} " f"(CI {lo:.1%}-{hi:.1%}) · p = {p:.3f}") ``` ```output declared, no margin n = 231 · accuracy 57.6% (CI 50.9%-64.0%) · p = 0.133 effective + 30 days n = 185 · accuracy 55.1% (CI 47.7%-62.4%) · p = 0.377 ``` With the declared cutoff, the gray-zone questions, which the model partly remembers, enter the evaluation. Accuracy rises and the p-value falls: the evaluation seems to show a skill that is memory of the months in which the corpus already covered the answer. 2. Extend `enmascarar` so that the sales consensus is also rescaled (to 100) and apply it to the 200 sales questions. Is there any figure left that could identify the company? **Solution.** ```python def enmascarar2(pregunta, nombre, ticker): t = enmascarar(pregunta, nombre, ticker) return re.sub(r"[\d.]+,\d millones de dólares", "100 (consenso reescalado)", t) ventas = q[q["tipo"] == "ventas_frente_a_consenso"] textos = [enmascarar2(f["pregunta"], nombres[actual[f["ticker"]]], f["ticker"]) for _, f in ventas.iterrows()] print(textos[0]) resto = {n for t in textos for n in re.findall(r"\d[\d.,]*", t)} print("figures left in the 200:", sorted(resto)) ``` ```output El consenso espera unas ventas de 100 (consenso reescalado) de la empresa A en el cuarto trimestre del año t. ¿Las superará? figures left in the 200: ['100'] ``` The year has become “del año t” (of year t), the amount 100 and the company “la empresa A” (company A): the only figure left in the 200 questions is the 100 of the rescaling. Even so, the text keeps information: the quarter (“cuarto”, fourth) and the question itself. With real data you would also have to check that the sector, the size or a characteristic phrase do not identify the issuer, and measure the effect of the masking by comparing the answers with and without it, as Glasserman and Lin (2023) do. 3. A provider offers a financial LLM with a declared cutoff of June 2024 and a backtest from 2015 to 2025 with a Sharpe of 3. What would you ask for before believing it? **Solution.** The part of the backtest after the effective cutoff, separately: from 2015 to mid-2024 the model could have read the results. I would ask how that cutoff was estimated (and not only the declared one), the exact version of the model used at each date and whether it changed, the number of models, prompts and variants tried before choosing the one presented, and the result of a trivial baseline over the same period with its confidence interval. With a year and a bit of honest history, a high Sharpe has an enormous standard error (lesson 2.5.4): the reasonable thing is to wait to accumulate out-of-sample results or to ask for a point-in-time version of the model. [^1]: Qwen blog, “Qwen2.5: A Party of Foundation Models!”, September 19, 2024 (https://qwenlm.github.io/blog/qwen2.5/, accessed 2026-10-23). It does not publish a knowledge cutoff date. [^2]: Statement by Thomas C. Baxter Jr. (Federal Reserve Bank of New York) before the Financial Crisis Inquiry Commission, September 1, 2010 (https://www.newyorkfed.org/newsevents/speeches/2010/bax100901): on September 15, Lehman's board took the parent company into Chapter 11. ## Self-assessment 1. An LLM with a declared cutoff of September 2021 gets 89% of 237 questions right on whether a stock will rise over the following month, all resolved before that date, and 51% of the 383 resolved after it. What does the 89% measure? - A. What the model remembers from its training corpus, not its ability to predict - B. The model's skill in calm markets - C. That the model has degraded over time - D. A good model that loses skill because of a regime change in 2022 *Answer A: What the model remembers from its training corpus, not its ability to predict.* The answers were already in the text it was trained on. After the cutoff, accuracy falls to that of a coin (51%), which is what the model can do without memory. A regime change does not explain a 38-point drop right at the cutoff date. 2. `modelo-b` declares a cutoff of 2023-04-30, but its accuracy stays high until the summer and the estimated break falls on 2023-07-31. How should the declared date be treated? - A. As exact, because the provider publishes it - B. As a lower bound: the corpus may include some later text, so the honest evaluation starts after the estimated effective cutoff, with a margin - C. As an upper bound: the model knows less than it claims - D. It does not matter, because the difference is three months *Answer B: As a lower bound: the corpus may include some later text, so the honest evaluation starts after the estimated effective cutoff, with a margin.* In the data, `modelo-b` was trained on text up to 2023-09-30. The questions in the gray zone get 76% right. With real data there is no published ground truth: the cutoff is estimated with the accuracy itself or with memory probes, and a margin is left. 3. Outside its cutoff, `modelo-c` gets 66.7% of 78 questions right (95% CI: 55%-77%, p = 0.01 against always answering “yes”). The generator's ground truth says that outside their memory the models get 52-55% right. What happened? - A. The model has a real skill the generator does not capture - B. There is an error in the confidence interval - C. The model is still remembering - D. A lucky streak in a small sample: with several models evaluated, one coming out significant by chance is not rare, and 78 questions give an interval more than 20 points wide *Answer D: A lucky streak in a small sample: with several models evaluated, one coming out significant by chance is not rare, and 78 questions give an interval more than 20 points wide.* It is the multiple testing problem of level 2 in another guise. To tell 55% from 50% with 80% power takes about 617 questions resolved after the cutoff, far more than there are. 4. To keep the model from recognizing the company, its name is removed from the prompt. Qwen2.5-0.5B still answers “2008” to “a large New York investment bank filed for bankruptcy on a September 15, in the middle of the subprime mortgage crisis” (asked in Spanish). What does it show? - A. That the context identifies the event even without the name: masking entities reduces the bias, it does not eliminate it - B. That the model is random - C. That the prompt has to be translated into English - D. That the model does not know Lehman Brothers *Answer A: That the context identifies the event even without the name: masking entities reduces the bias, it does not eliminate it.* Without the crisis context, the model answers “2023” and gets it wrong. Lopez-Lira, Tang and Zhu (2025) document the same with large models: masking names and dates does not stop them from reconstructing the fact from a little context. 5. What is the most reliable way to backtest a strategy that uses an LLM? - A. Ask the model in the prompt not to use information after the question's date - B. Use the most recent model, which is the best - C. Use only questions resolved after the model's effective cutoff, or a point-in-time model trained only on text prior to each date, and compare with a trivial baseline with a confidence interval - D. Mask the ticker *Answer C: Use only questions resolved after the model's effective cutoff, or a point-in-time model trained only on text prior to each date, and compare with a trivial baseline with a confidence interval.* Instructions do not erase memory (Lopez-Lira, Tang and Zhu, 2025; Sarkar and Vafa, 2025). Point-in-time models (TiMaGPT, ChronoBERT and ChronoGPT, DatedGPT) or post-cutoff evaluation are the routes the literature proposes. ## Key points - An LLM remembers what happened before its cutoff: in the data, the three models get 89-93% of what was resolved before their declared cutoff right, and following their answers would earn an average 6.2-6.5% over 21 sessions (t > 10); afterwards, between 51% and 70%. - The declared cutoff is not enough: the effective one may be later. A break in accuracy estimates it (`modelo-b`: declared 2023-04-30, estimated 2023-07-31, ground truth 2023-09-30), and the honest evaluation starts after it, with a margin. - After the effective cutoff, no model clearly beats always answering “yes” (53.7%) except `modelo-c` with 78 questions, a streak the generator's ground truth refutes. Detecting 55% against 50% takes about 617 questions. - Masking the name and the date reduces recognition but does not eliminate it: the context, the figures and the style identify the event (Qwen2.5-0.5B answers “2008” to a description of the Lehman bankruptcy without the name). - The cutoff date the model itself states is not a source: Qwen2.5-0.5B answers “Marzo 2019” (March 2019) in Spanish and “October 2023” in English. - The defenses in the literature: evaluate only after the cutoff (Lopez-Lira and Tang), anonymize (Glasserman and Lin) and point-in-time models (TiMaGPT, ChronoBERT, DatedGPT). ## Resources ### Books - [Glasserman, P. and Lin, C. (2023): Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis. arXiv:2309.17322](https://arxiv.org/abs/2309.17322) · arxiv.org - [Lopez-Lira, A., Tang, Y. and Zhu, M. (2025): The Memorization Problem: Can We Trust LLMs' Economic Forecasts? arXiv:2504.14765](https://arxiv.org/abs/2504.14765) · arxiv.org - [He, S., Lv, L., Manela, A. and Wu, J. (2025): Chronologically Consistent Large Language Models. arXiv:2502.21206](https://arxiv.org/abs/2502.21206) · arxiv.org - [Yan, Y. et al. (2026): DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining. arXiv:2603.11838](https://arxiv.org/abs/2603.11838) · arxiv.org - [Sarkar, S. and Vafa, K. (2025): Lookahead Bias in Pretrained Language Models (ICML 2025, workshop)](https://icml.cc/virtual/2025/50857) · icml.cc - [Drinkall, F. et al. (2024): Time Machine GPT. arXiv:2404.18543](https://arxiv.org/abs/2404.18543) · arxiv.org ### Data - [Level 6 data: look-ahead bias question bank (CSV)](https://pythonparatrading.com/curso/datos/nivel-6/sesgo6_preguntas.csv) · Download · sesgo6_preguntas.csv Previous lesson: [6.2.4 · Lightweight fine-tuning for financial tasks: LoRA and QLoRA](https://pythonparatrading.com/en/python-trading-course/level-6/lightweight-fine-tuning-lora-qlora/) Next lesson: [6.3.1 · Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant](https://pythonparatrading.com/en/python-trading-course/level-6/embeddings-and-vector-databases/)