Evidence and limits of agents.

Three questions an agent benchmark must answer

Lessons 6.4.1 to 6.4.3 built agents. This one asks what is known about their performance when they trade, and does it with three questions that any study of trading agents should answer before showing a figure:

1. Does general capability predict the ability to invest? If the model that wins the chat leaderboards also made money, choosing a model would be easy.
2. Does the result repeat if the run is repeated? A system whose decisions change from one run to another does not have a performance; it has a distribution of performances.
3. Does what shines in the backtest hold up live? Without a test after the model's training and the strategy's design, a backtest measures memory and selection (lessons 6.2.5 and 2.5.4).

Each section reads a study that answers one of them, with the figures as they appear in the PDF (read on 2026-10-23), and then reproduces the problem on the course data, where the answer is known.

flowchart TB
  A[Agent] --> B[After the cutoff]
  B --> C[Several runs]
  C --> D[Baselines and costs]
  D --> E[Many sessions]
  E --> F[Conclusion]

LiveTradeBench: the general ranking does not order returns

“LiveTradeBench: Seeking Real-World Alpha with Large Language Models”, by Haofei Yu, Fenghai Li and Jiaxuan You (University of Illinois Urbana-Champaign), arXiv:2511.03628, version 1 of November 5, 2025. The design avoids the backtest: the agents receive live prices, news (headlines and summaries) and their portfolio, and every day they return an allocation in percentages. There are two markets: 15 US stocks and ETFs, and Polymarket prediction markets. Each of the 21 LLMs (Claude, Grok, Qwen, Llama, GPT, Gemini, Kimi and DeepSeek: the text speaks of “six representative model families” and the table has eight) runs inside the same ReAct-style agent framework, with the same prompt, and the evaluation runs from August 18 to October 24, 2025, 50 trading days according to the paper (on the NYSE calendar of the course data there are 49 sessions between those two dates; the difference may come from how they count the first day or from the Polymarket calendar, and it changes nothing in what follows).

The figures the syllabus cites are in the paper:

• “In the stock market, the Spearman correlation between LMArena scores and cumulative returns is only 0.054—virtually no relationship. In Polymarket, the correlation drops to –0.38”. The next figure draws them.
• GPT-4.1 “achieves the highest cumulative return rate (> 6%) in the stock market but performs poorly in Polymarket (return < −30%)”. In table 2: +6.25% and −33.69%.

And the paper says more things worth reading next to those figures. The Sharpe correlation between the two markets “is close to zero”: good performance in one implies nothing in the other. The authors attribute GPT-4.1's loss on Polymarket to “overreactive allocation changes under higher volatility”. In their limitations section they acknowledge that the environment does not account for commissions, bid-ask spreads or liquidity, which can overstate the performance of strategies that rebalance a lot. The chat models are called with temperature 0.3 and each one is evaluated on a single trajectory: a live evaluation cannot be repeated. And the table does not include a passive benchmark portfolio with the same 15 assets. The header figure places the 21 models in both markets at once: in stocks they all earn, between 1.78% and 6.25%; on Polymarket, 17 of 21 lose.

import numpy as np
import pandas as pd
from scipy import stats

# Table 2 of arXiv:2511.03628 v1: cumulative return (%) and Sharpe
ltb = pd.DataFrame([
    ("Claude-Sonnet-3.7", 3.63, 1.45, 20.54, 2.38),
    ("Claude-Sonnet-4", 2.45, 0.72, -40.32, -2.40),
    ("Claude-Opus-4", 3.93, 1.72, -2.04, 0.09),
    ("Claude-Opus-4.1", 3.73, 1.51, -25.69, -3.02),
    ("Grok-3", 3.22, 1.26, -8.35, -0.55),
    ("Grok-4", 4.30, 1.75, 7.38, 1.01),
    ("Qwen2.5-72B", 5.15, 2.18, 1.63, 0.43),
    ("Qwen3-235B-Instruct", 3.52, 1.32, -54.24, -2.97),
    ("Qwen3-235B-Thinking", 1.78, 0.60, -57.62, -1.81),
    ("Llama3.3-70B", 2.72, 0.88, 1.58, 0.40),
    ("Llama4-Scout", 4.65, 1.99, -16.05, -1.18),
    ("Llama4-Maverick", 4.46, 1.65, -18.31, -1.92),
    ("GPT-4o", 3.55, 1.39, -30.96, -3.26),
    ("GPT-4.1", 6.25, 2.64, -33.69, -1.74),
    ("GPT-5", 5.31, 2.19, -23.96, -0.49),
    ("GPT-o3", 6.04, 2.57, -54.84, -3.68),
    ("Gemini-2.5-Flash", 2.10, 0.72, -22.40, -0.82),
    ("Gemini-2.5-Pro", 1.95, 0.61, -35.15, -1.65),
    ("Kimi-K2", 3.07, 1.15, -53.44, -5.26),
    ("DeepSeek-V3.1", 2.46, 0.86, -4.68, -0.07),
    ("DeepSeek-R1", 2.10, 0.78, -13.19, 0.14)],
    columns=["modelo", "cr_acc", "sr_acc", "cr_poly", "sr_poly"])
n = len(ltb)
print(n, "models | earn in stocks:", (ltb.cr_acc > 0).sum(),
      "| earn on Polymarket:", (ltb.cr_poly > 0).sum())
for a, b in [("cr_acc", "cr_poly"), ("sr_acc", "sr_poly")]:
    print(f"Spearman {a} vs {b}: "
          f"{stats.spearmanr(ltb[a], ltb[b])[0]:.2f}")

def ic_spearman(rho, n):
    """95% CI (Fisher with Fieller's error: 1.06/sqrt(n-3))."""
    z, se = np.arctanh(rho), 1.06 / np.sqrt(n - 3)
    return np.tanh(z - 1.96 * se), np.tanh(z + 1.96 * se)

for rho in (0.054, -0.38):
    t = rho * np.sqrt((n - 2) / (1 - rho**2))
    p = 2 * stats.t.sf(abs(t), n - 2)
    lo, hi = ic_spearman(rho, n)
    print(f"rho = {rho:+.3f}: 95% CI [{lo:+.2f}, {hi:+.2f}], "
          f"p = {p:.2f}")

21 models | earn in stocks: 21 | earn on Polymarket: 4
Spearman cr_acc vs cr_poly: 0.17
Spearman sr_acc vs sr_poly: 0.06
rho = +0.054: 95% CI [-0.41, +0.50], p = 0.82
rho = -0.380: 95% CI [-0.71, +0.09], p = 0.09

Three readings. First: the correlation between markets, computed from the table itself, is small both in return and in Sharpe, as the authors say. Second: with 21 models, the 95% interval of a rank correlation of 0.054 goes from −0.41 to 0.50: it does not rule out a moderate relationship in either direction, and the −0.38 on Polymarket, with p = 0.09, is not significant at 5% either. The conclusion the data support is the one in the paper's abstract, “high LMArena scores do not imply superior trading outcomes”, not that capability does not matter. Third: that all 21 earn money in stocks over 50 days says more about the market in those weeks than about the models; without a passive portfolio in the table there is no way to separate beta from skill.

The same question with a known answer

With real data nobody knows how much a model's capability really helps. With the course data it can be set. The cell builds a bench of 21 simulated agents in the style of LiveTradeBench: the 15 largest companies of level 6, a decision at the close of each session with the news published up to 16:00 New York time and the 5-session return, and a daily allocation between the 15 stocks and cash. The news corpus is the level's sample corpus, in Spanish, read as is. Each agent has:

• a simulated general score (like LMArena's), between 1,300 and 1,450;
• a news reading quality that grows with the score: capability really helps, because in the data the news move the price (the planted effect in the README);
• a risk appetite (fraction in cash), an aggressiveness and a taste for momentum that have nothing to do with the score;
• an inference noise: each decision reads the evidence with an error that changes in every run.

import re

emp = pd.read_csv("acciones6_empresas.csv")
hist = pd.read_csv("acciones6_tickers.csv")
ID = dict(zip(hist["ticker"], hist["id_empresa"]))
ACTUAL = dict(zip(emp["id_empresa"], emp["ticker"]))
px = pd.read_parquet("acciones6_precios.parquet")
cierres = px.pivot(index="fecha", columns="ticker", values="cierre")
acciones = emp.set_index("ticker")["acciones_millones"]
U = list((cierres.loc["2019-01-02"] * acciones).nlargest(15).index)
r = cierres[U].pct_change().loc["2019-01-02":]
F = r.index
R, R5 = r.to_numpy(), cierres[U].pct_change(5).loc[F].to_numpy()

lex = pd.read_csv("texto6_lexico.csv").query("lista == 'financiera'")
POS = set(lex.query("categoria == 'positiva'")["palabra"])
NEG = set(lex.query("categoria == 'negativa'")["palabra"])
def tono(t):
    pal = re.findall(r"\w+", t.lower())
    p, n_ = sum(w in POS for w in pal), sum(w in NEG for w in pal)
    return (p - n_) / (p + n_ + 1)
nt = pd.read_parquet("noticias6_corpus.parquet")
nt["tk"] = nt["ticker"].map(ID).map(ACTUAL)
nt = nt[nt["tk"].isin(U)]
nt["clave"] = nt["titular"].str.replace(
    r"^(?:ACTUALIZA|Última hora|UPDATE|Breaking):\s*", "",
    regex=True, case=False)
nt = nt.sort_values("ts").drop_duplicates(["tk", "clave"])
nt["tono"] = (nt["titular"] + " " + nt["cuerpo"]).map(tono)
ny = nt["ts"].dt.tz_convert("America/New_York").dt.tz_localize(None)
i = np.searchsorted(F + pd.Timedelta(hours=16), ny, side="left")
nt = nt[i < len(F)].assign(dia=F[i[i < len(F)]])
T = (nt.groupby(["dia", "tk"])["tono"].sum().unstack()
     .reindex(index=F, columns=U).fillna(0).to_numpy())

rng = np.random.default_rng(644)
NA = 21
puntuacion = np.round(1300 + 150 * rng.random(NA))
calidad = 0.2 + 0.6 * (puntuacion - 1300) / 150   # rises with the score
caja = rng.uniform(0.0, 0.6, NA)                   # independent of it
kappa = rng.uniform(0.5, 3.0, NA)
gusto_mom = rng.normal(0, 1, NA)

def ejecutar(a, t0, n=50, semilla=0, ruido=1.0):
    """Cumulative return of agent `a` over n sessions from t0."""
    g = np.random.default_rng([semilla, a, t0])
    v = 1.0
    for t in range(t0, t0 + n):
        lectura = (calidad[a] * T[t - 1]
                   + ruido * g.standard_normal(len(U)))
        s = kappa[a] * (lectura + 6 * gusto_mom[a] * R5[t - 1])
        w = np.exp(s - s.max())
        w = (1 - caja[a]) * w / w.sum()
        v *= 1 + w @ R[t]
    return v - 1

t0 = F.searchsorted(pd.Timestamp("2025-08-18"))
cr = np.array([ejecutar(a, t0) for a in range(NA)])
print(F[t0].date(), "to", F[t0 + 49].date(), "| news on",
      f"{(T != 0).mean():.1%} of stock-days")
print("Spearman(score, return):",
      round(stats.spearmanr(puntuacion, cr)[0], 3))
print("returns (%):", np.round(100 * np.sort(cr), 1))

2025-08-18 to 2025-10-27 | news on 9.0% of stock-days
Spearman(score, return): 0.316
returns (%): [-7.8 -5.3 -4.3 -4.1 -4.  -3.  -2.5 -2.4 -2.3 -1.1 -0.8 -0.7 -0.5 -0.3
  0.   0.2  0.5  1.3  1.6  2.   6.7]

On the same dates as LiveTradeBench, the simulated bench gives a correlation of 0.32 between the score and the return, and the returns of the 21 agents go from −7.8% to +6.7%. Is 0.32 the bench's “truth”? It cannot be known from one window: over 50 sessions, what separates the agents is how much money they have invested while the market rises or falls and the luck of their readings, and the edge of reading well only shows up when there is news about those 15 companies, which is rare (9% of the stock-days have some). To see the size of the noise, the next cell repeats the bench in every non-overlapping 50-session window from 2019 to 2025 and compares it with the long-run correlation, that of the average return over all the windows.

ventanas = list(range(6, len(F) - 50, 50))
CR = np.array([[ejecutar(a, t) for a in range(NA)] for t in ventanas])
rho = np.array([stats.spearmanr(puntuacion, c)[0] for c in CR])
print(len(ventanas), "windows | mean", round(rho.mean(), 3),
      "| 5th and 95th percentiles:",
      np.round(np.percentile(rho, [5, 95]), 2),
      "| negative:", f"{(rho < 0).mean():.0%}")
print("long run: Spearman(score, mean) =",
      round(stats.spearmanr(puntuacion, CR.mean(axis=0))[0], 3),
      "| Spearman(cash, mean) =",
      round(stats.spearmanr(caja, CR.mean(axis=0))[0], 3))

35 windows | mean 0.049 | 5th and 95th percentiles: [-0.29  0.49] | negative: 43%
long run: Spearman(score, mean) = 0.373 | Spearman(cash, mean) = -0.566

The bench's truth is positive: in the long run, with the 35 windows averaged, the correlation between the score and the average return is 0.37. But in a 50-session window it is a draw: the mean of the 35 is only 0.05, the central 90% goes from −0.29 to 0.49 and 43% of the windows give a negative correlation. And what best explains the average return is not capability but how much each agent invests: the correlation of the fraction in cash with the average return is −0.57, in a period in which the market rises. A 0.054 or a −0.38 in a single 50-day window is compatible with almost any true relationship.

AlphaForgeBench: the same question does not get the same answer

“AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models”, by Wentao Zhang and seven other authors (Nanyang Technological University, HKUST (Guangzhou) and Hong Kong Polytechnic University), arXiv:2602.18481, version 2 of May 27, 2026, with the KDD 2026 header. Its starting point is a critique of benchmarks like LiveTradeBench: when an LLM directly emits the trading actions, its decisions are unstable. In their words, LLMs “produce drastically different trading actions across runs even under deterministic decoding (Section C), rendering single-run live evaluations statistically unreliable”.

The experiment in appendix C is simple: BTC with daily data from 2025, a prompt with 30 days of prices and technical indicators, one action per day (buy everything, hold or sell everything), no costs, and five identical runs of each model with temperature 0 and another five with 0.7. The results for gemini-3-pro-preview:

Other models are more stable (gemini-3-flash-preview agrees 85.3% with temperature 0), but even so the return curves “gradually fan out”. The authors attribute it to floating-point non-determinism and to the routing of mixture-of-experts models, amplified because LLMs have no memory of their own previous actions: a different decision early on changes all the following ones. And they write that in direct trading benchmarks “run-to-run variance routinely exceeds intermodel variance”. Their proposal (lesson 6.4.6) is for the LLM to write factors and strategy code, which are then run in a deterministic backtest: that way the variability is confined to a single generation step.

Repeated runs in the simulated bench

The cell repeats the bench five times in the 2025 window with different seeds for the inference noise. It measures the standard deviation of each agent's return across its runs, compares it with the standard deviation between agents and computes how similar the rankings of two runs are. At the end it removes the inference noise (the equivalent of truly deterministic decoding).

runs = np.array([[ejecutar(a, t0, semilla=s) for a in range(NA)]
                 for s in range(5)])
dentro = runs.std(axis=0).mean()           # same agent, 5 draws
entre = runs.mean(axis=0).std()            # between agents
print(f"std. within each agent: {100 * dentro:.2f} pp | "
      f"between agents: {100 * entre:.2f} pp")
print("Spearman between the ranking of run 0 and the others:",
      [round(float(stats.spearmanr(runs[0], runs[k])[0]), 2)
       for k in range(1, 5)])
sin_ruido = np.array([[ejecutar(a, t0, semilla=s, ruido=0.0)
                       for a in range(NA)] for s in range(2)])
print("without inference noise, identical runs?",
      np.array_equal(sin_ruido[0], sin_ruido[1]))

std. within each agent: 3.58 pp | between agents: 2.00 pp
Spearman between the ranking of run 0 and the others: [-0.14, -0.17, 0.17, -0.34]
without inference noise, identical runs? True

The standard deviation of the same agent across its five runs (3.58 points) is almost double the standard deviation between agents (2.00), and the ranking of one run does not predict that of the others (correlations from −0.34 to 0.17): it is the pattern AlphaForgeBench describes in real models. Without inference noise the bench becomes reproducible, but the ranking is still that of a single 50-session window, with all the noise of the previous section.

A decision with no margin in the local model

Greedy decoding of the course's local model is deterministic: the same text always gives the same answer (lesson 6.2.1). The problem is a different one: the text changes for irrelevant reasons. The cell asks the Qwen2.5-0.5B-Instruct model whether it would buy, hold or sell a stock that has risen 1.5% in a week with news about a lawsuit, written in eight equivalent ways, and looks at the logits of the first token of the answer: the chosen option and its margin over the second. The prompt is in Spanish, like the rest of level 6's model inputs: “La acción ha variado un +1,5 % esta semana y hay una noticia sobre una demanda de un cliente” reads “The stock has changed by +1.5% this week and there is news about a lawsuit from a customer”, and the model must answer with one word, COMPRAR, MANTENER or VENDER (buy, hold or sell). Note that “1,5” uses the Spanish decimal comma.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(NOMBRE)
llm = AutoModelForCausalLM.from_pretrained(NOMBRE, dtype=torch.float32)

D = "hay una noticia sobre una demanda de un cliente"
variantes = [
    f"La acción ha variado un +1,5 % esta semana y {D}.",
    f"La acción ha variado un +1.5 % esta semana y {D}.",
    f"La acción ha variado un 1,5 % esta semana y {D}.",
    f"La acción ha subido un 1,5 % esta semana y {D}.",
    f"Esta semana la acción ha variado un +1,5 % y {D}.",
    f"La acción ha variado un +1,5 % esta semana; {D}.",
    f"La acción ha variado un +1,50 % esta semana y {D}.",
    f"La acción ha variado un +1,5% esta semana y {D}."]
PIDE = " Responde solo con una palabra: COMPRAR, MANTENER o VENDER."
ids = {o: tok(o, add_special_tokens=False)["input_ids"][0]
       for o in ("COMPRAR", "MANTENER", "VENDER")}
for v in variantes:
    x = tok.apply_chat_template([{"role": "user", "content": v + PIDE}],
                                add_generation_prompt=True,
                                return_tensors="pt", return_dict=True)
    with torch.inference_mode():
        lg = llm(**x).logits[0, -1]
    l = {o: float(lg[i]) for o, i in ids.items()}
    a, b = sorted(l, key=l.get, reverse=True)[:2]
    print(f"{a:8} margin {l[a] - l[b]:.2f} | {v[:46]}")

COMPRAR  margin 0.05 | La acción ha variado un +1,5 % esta semana y h
VENDER   margin 0.09 | La acción ha variado un +1.5 % esta semana y h
VENDER   margin 0.07 | La acción ha variado un 1,5 % esta semana y ha
VENDER   margin 0.04 | La acción ha subido un 1,5 % esta semana y hay
VENDER   margin 0.34 | Esta semana la acción ha variado un +1,5 % y h
COMPRAR  margin 0.35 | La acción ha variado un +1,5 % esta semana; ha
COMPRAR  margin 0.14 | La acción ha variado un +1,50 % esta semana y
VENDER   margin 0.02 | La acción ha variado un +1,5% esta semana y ha

With “+1,5 %” the model buys; with “+1.5 %” (decimal point), without the sign, with “subido” (risen) instead of “variado” (changed), with the sentence in another order or without a space before the “%”, it sells; with a semicolon or with “+1,50 %”, it buys again. The margins between the chosen option and the next one go from 0.02 to 0.35 logits: the “decision” is almost a tie that any detail of the text settles. A bigger model will have different margins, and the lesson does not extrapolate this 0.5B model to commercial models; what it does teach is what to measure before trusting an agent: how much its decision changes under irrelevant perturbations of the prompt, the data format and the order of the information. With a model served through an API, the sources of variation that AlphaForgeBench points out and that the user does not control are added on top.

From backtest to live: the degradation of deep learning models

The syllabus states that deep learning models that are strong in the backtest degrade live. The most recent primary source that documents it with a live phase is AlphaCrafter (Yishuo Yuan and four other authors, Nanjing University, arXiv:2605.05580, version 2 of July 28, 2026). Its protocol: daily portfolios on the CSI 300 and the S&P 500, training from 2016 to 2022, validation in 2023, a backtest from January 2024 to February 2026 (519 and 542 sessions) and a live phase with a real broker's paper trading API from March to June 2026 (69 and 73 sessions), after the training cutoff of all the LLMs they use. Their text: “Machine learning and deep learning methods, notably LSTM, demonstrate strong backtesting performance but exhibit pronounced degradation in live trading, exposing their vulnerability to overfitting and regime shifts. This pattern is consistent across both markets”.

# Table 2 of arXiv:2605.05580 v2: annualized Sharpe, methods without LLMs
ac = pd.DataFrame([
    ("MACD", 0.99, 0.87, 0.72, 1.22),
    ("Grid", 0.33, -0.26, 1.11, 0.39),
    ("LightGBM", 1.22, 0.51, 0.51, 1.19),
    ("XGBoost", 1.24, 0.36, -0.16, 1.01),
    ("LSTM", 1.43, 0.35, 1.38, 0.83),
    ("Transformer", 1.19, -0.32, 0.44, 0.28),
    ("TRA", 1.16, 0.71, 1.01, 1.02)],
    columns=["metodo", "csi_bt", "csi_vivo", "sp_bt", "sp_vivo"])
SES = {"csi": (519, 69), "sp": (542, 73)}       # sessions bt / live
se = lambda n: np.sqrt(252 / n)                 # standard error of the SR
for m, (nb, nv) in SES.items():
    d = ac[f"{m}_vivo"] - ac[f"{m}_bt"]
    s = np.hypot(se(nb), se(nv))
    print(f"{m}: {(d < 0).sum()} of 7 drop | standard error of the SR: "
          f"bt {se(nb):.2f}, live {se(nv):.2f}, difference {s:.2f}")
ac.set_index("metodo")

csi: 7 of 7 drop | standard error of the SR: bt 0.70, live 1.91, difference 2.03
sp: 3 of 7 drop | standard error of the SR: bt 0.68, live 1.86, difference 1.98
             csi_bt  csi_vivo  sp_bt  sp_vivo
metodo
MACD           0.99      0.87   0.72     1.22
Grid           0.33     -0.26   1.11     0.39
LightGBM       1.22      0.51   0.51     1.19
XGBoost        1.24      0.36  -0.16     1.01
LSTM           1.43      0.35   1.38     0.83
Transformer    1.19     -0.32   0.44     0.28
TRA            1.16      0.71   1.01     1.02

A careful reading has two parts. The one that supports the syllabus: on the CSI 300 the seven methods without LLMs drop from backtest to live, the LSTM goes from 1.43 to 0.35 and the Transformer from 1.19 to −0.32; on the S&P 500 the LSTM also drops (from 1.38 to 0.83). The one that calls for nuance: on the S&P 500 only three of seven drop; MACD, LightGBM and XGBoost clearly rise live and TRA stays the same, and the standard error of an annualized Sharpe measured over some 70 sessions is almost 1.9, so no individual drop can be distinguished from noise; the difference between backtest and live has a standard error of about 2. Moreover, the authors propose a competing method, which degrades less in their table. The pattern is consistent with what the course has already measured in other ways: validation overfitting (lesson 5.5.4), drift (lesson 5.10.2), the decay of published anomalies (lesson 5.1.1) and the collapse of foundation model Sharpes with costs (lesson 5.8.4). But a three-month live window does not prove it on its own.

Two more studies complete the picture of LLM agents. FINSABER (Li, Kim, Cucuringu and Ma, arXiv:2505.07078) repeats LLM timing strategies over two decades and with more than 100 stocks, and finds that the published advantages “deteriorate significantly under broader cross-section and over a longer-term evaluation”; the strategies are too conservative in bull markets and too aggressive in bear markets. DeepFund (Li et al., arXiv:2505.11065) evaluates nine models live, with data after each one's cutoff, because backtests let LLMs “time travel”, and finds that even DeepSeek-V3 and Claude-3.7-Sonnet have net losses.

How to evaluate an agent without fooling yourself

Common mistakes

• Choosing the model by a general ranking. In LiveTradeBench the correlation with return was 0.054 in stocks and −0.38 on Polymarket.
• Reading a 21-point correlation as if it were precise. Its 95% interval is almost 0.9 wide.
• Publishing a single run. An agent's variability between runs can exceed the variability between agents.
• Trusting temperature 0. It fixes the decoding, not the input text, nor the numerical precision, nor the server that responds.
• Taking three months live as proof. The standard error of a Sharpe over 70 sessions is almost 1.9.
• Forgetting beta. If all the agents earn money in a rising market, nobody has demonstrated skill.

Exercises

1. How many models would you need to compare for a true Spearman correlation of 0.18 (that of the simulated bench in the long run) to be detected with 80% power at 5% two-sided? Use the Fisher approximation with Fieller's error.

def modelos_necesarios(rho, alfa=0.05, potencia=0.8):
    z = stats.norm.ppf(1 - alfa / 2) + stats.norm.ppf(potencia)
    return int(np.ceil((1.06 * z / np.arctanh(rho)) ** 2 + 3))

for rho in (0.18, 0.3, 0.5):
    print(rho, "->", modelos_necesarios(rho), "models")

0.18 -> 270 models
0.3 -> 96 models
0.5 -> 33 models

2. Repeat the simulated bench with 250-session windows (one year) instead of 50. How do the mean and the 5th and 95th percentiles of the correlation change?

filas = []
for t in range(6, len(F) - 250, 250):
    c = [ejecutar(a, t, n=250) for a in range(NA)]
    filas.append({"inicio": F[t].date(),
                  "mercado_%": 100 * (np.prod(1 + R[t:t + 250]
                                              .mean(axis=1)) - 1),
                  "rho_puntuacion": stats.spearmanr(puntuacion, c)[0],
                  "rho_caja": stats.spearmanr(caja, c)[0]})
v = pd.DataFrame(filas).set_index("inicio")
print("mean of rho_puntuacion:", round(v["rho_puntuacion"].mean(), 3))
v.round(2)

mean of rho_puntuacion: 0.027
            mercado_%  rho_puntuacion  rho_caja
inicio
2019-01-10      17.87            0.07     -0.10
2020-01-08      12.65            0.25     -0.34
2021-01-05      17.37            0.03     -0.06
2021-12-31     -26.11            0.09      0.56
2022-12-29      10.51           -0.13     -0.15
2023-12-28      26.44            0.06     -0.33
2024-12-26       6.44           -0.20     -0.29

3. No code: design the evaluation of a research agent that proposes buys and sells on 50 stocks every week. You have a budget for one year of calls to a commercial model.