--- title: "6.4.4 Evidence and limits of trading agents" description: "LiveTradeBench, AlphaForgeBench and AlphaCrafter read at the source and a simulated agent bench: rankings that do not predict, runs that do not repeat." url: "https://pythonparatrading.com/en/python-trading-course/level-6/evidence-and-limits-of-trading-agents/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/evidence-and-limits-of-trading-agents.png" --- # 6.4.4 · Evidence and limits of agents > Does the model that wins the general leaderboards trade better? Does it repeat its decisions if asked twice? Does what shines in the backtest hold up live? This lesson reads three recent studies with their exact figures, measures how much can be concluded from 21 models and 50 sessions, and reproduces the three problems with simulated agents on the course data and with the local 0.5B model. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.4 · AI agents for trading - Duration: 70 min - Prerequisites: [6.4.3 · TradingAgents-style multi-agent architectures](https://pythonparatrading.com/en/python-trading-course/level-6/multi-agent-architectures-like-tradingagents/), [6.2.5 · Look-ahead bias in LLMs: the model that already knows the future](https://pythonparatrading.com/en/python-trading-course/level-6/look-ahead-bias-in-llms/), [6.2.1 · How an LLM works: from the next token to the API](https://pythonparatrading.com/en/python-trading-course/level-6/how-llms-work-apis-and-local-models/), [5.8.4 · Foundation models in finance: what the evidence says](https://pythonparatrading.com/en/python-trading-course/level-5/foundation-models-finance-evidence/), [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/) ## By the end of this lesson - Summarize with verified figures LiveTradeBench (21 LLMs, 50 live days, Spearman correlations of 0.054 and −0.38), AlphaForgeBench (instability between runs with deterministic decoding) and the backtest-to-live degradation documented in AlphaCrafter. - Compute the confidence interval of a rank correlation with 21 models and the standard error of a Sharpe measured over about 70 sessions. - Simulate an agent bench on the course data and check that the correlation between a general score and the 50-session return is almost noise, even though capability helps in the long run. - Measure the variability between runs of the same agent and compare it with the variability between agents. - Show with a local LLM that a format change in the prompt reverses a buy or sell decision, and design an honest evaluation of an agent. ## Three questions an agent benchmark must answer Lessons 6.4.1 to 6.4.3 built agents. This one asks what is known about their performance when they trade, and does it with three questions that any study of trading agents should answer before showing a figure: 1. **Does general capability predict the ability to invest?** If the model that wins the chat leaderboards also made money, choosing a model would be easy. 2. **Does the result repeat if the run is repeated?** A system whose decisions change from one run to another does not have a performance; it has a distribution of performances. 3. **Does what shines in the backtest hold up live?** Without a test after the model's training and the strategy's design, a backtest measures memory and selection (lessons 6.2.5 and 2.5.4). Each section reads a study that answers one of them, with the figures as they appear in the PDF (read on 2026-10-23), and then reproduces the problem on the course data, where the answer is known. ```mermaid flowchart TB A[Agent] --> B[After the cutoff] B --> C[Several runs] C --> D[Baselines and costs] D --> E[Many sessions] E --> F[Conclusion] ``` ## LiveTradeBench: the general ranking does not order returns “LiveTradeBench: Seeking Real-World Alpha with Large Language Models”, by Haofei Yu, Fenghai Li and Jiaxuan You (University of Illinois Urbana-Champaign), arXiv:2511.03628, version 1 of November 5, 2025. The design avoids the backtest: the agents receive **live** prices, news (headlines and summaries) and their portfolio, and every day they return an allocation in percentages. There are two markets: 15 US stocks and ETFs, and Polymarket prediction markets. Each of the **21 LLMs** (Claude, Grok, Qwen, Llama, GPT, Gemini, Kimi and DeepSeek: the text speaks of “six representative model families” and the table has eight) runs inside the same ReAct-style agent framework, with the same prompt, and the evaluation runs **from August 18 to October 24, 2025, 50 trading days** according to the paper (on the NYSE calendar of the course data there are 49 sessions between those two dates; the difference may come from how they count the first day or from the Polymarket calendar, and it changes nothing in what follows). The figures the syllabus cites are in the paper: - “In the stock market, the Spearman correlation between LMArena scores and cumulative returns is only 0.054—virtually no relationship. In Polymarket, the correlation drops to –0.38”. The next figure draws them. - GPT-4.1 “achieves the highest cumulative return rate (> 6%) in the stock market but performs poorly in Polymarket (return < −30%)”. In table 2: +6.25% and −33.69%. *Fig. — LiveTradeBench: correlation between LMArena score and the returns of 21 LLMs.* Two bars with the Spearman correlation between the general LMArena score of 21 language models and their cumulative return over 50 live sessions: 0.054 in stocks, close to zero, and −0.38 in Polymarket, negative. And the paper says more things worth reading next to those figures. The Sharpe correlation between the two markets “is close to zero”: good performance in one implies nothing in the other. The authors attribute GPT-4.1's loss on Polymarket to “overreactive allocation changes under higher volatility”. In their limitations section they acknowledge that the environment does not account for commissions, bid-ask spreads or liquidity, which can overstate the performance of strategies that rebalance a lot. The chat models are called with temperature 0.3 and each one is evaluated on a single trajectory: a live evaluation cannot be repeated. And the table does not include a passive benchmark portfolio with the same 15 assets. The header figure places the 21 models in both markets at once: in stocks they all earn, between 1.78% and 6.25%; on Polymarket, 17 of 21 lose. ```python import numpy as np import pandas as pd from scipy import stats # Table 2 of arXiv:2511.03628 v1: cumulative return (%) and Sharpe ltb = pd.DataFrame([ ("Claude-Sonnet-3.7", 3.63, 1.45, 20.54, 2.38), ("Claude-Sonnet-4", 2.45, 0.72, -40.32, -2.40), ("Claude-Opus-4", 3.93, 1.72, -2.04, 0.09), ("Claude-Opus-4.1", 3.73, 1.51, -25.69, -3.02), ("Grok-3", 3.22, 1.26, -8.35, -0.55), ("Grok-4", 4.30, 1.75, 7.38, 1.01), ("Qwen2.5-72B", 5.15, 2.18, 1.63, 0.43), ("Qwen3-235B-Instruct", 3.52, 1.32, -54.24, -2.97), ("Qwen3-235B-Thinking", 1.78, 0.60, -57.62, -1.81), ("Llama3.3-70B", 2.72, 0.88, 1.58, 0.40), ("Llama4-Scout", 4.65, 1.99, -16.05, -1.18), ("Llama4-Maverick", 4.46, 1.65, -18.31, -1.92), ("GPT-4o", 3.55, 1.39, -30.96, -3.26), ("GPT-4.1", 6.25, 2.64, -33.69, -1.74), ("GPT-5", 5.31, 2.19, -23.96, -0.49), ("GPT-o3", 6.04, 2.57, -54.84, -3.68), ("Gemini-2.5-Flash", 2.10, 0.72, -22.40, -0.82), ("Gemini-2.5-Pro", 1.95, 0.61, -35.15, -1.65), ("Kimi-K2", 3.07, 1.15, -53.44, -5.26), ("DeepSeek-V3.1", 2.46, 0.86, -4.68, -0.07), ("DeepSeek-R1", 2.10, 0.78, -13.19, 0.14)], columns=["modelo", "cr_acc", "sr_acc", "cr_poly", "sr_poly"]) n = len(ltb) print(n, "models | earn in stocks:", (ltb.cr_acc > 0).sum(), "| earn on Polymarket:", (ltb.cr_poly > 0).sum()) for a, b in [("cr_acc", "cr_poly"), ("sr_acc", "sr_poly")]: print(f"Spearman {a} vs {b}: " f"{stats.spearmanr(ltb[a], ltb[b])[0]:.2f}") def ic_spearman(rho, n): """95% CI (Fisher with Fieller's error: 1.06/sqrt(n-3)).""" z, se = np.arctanh(rho), 1.06 / np.sqrt(n - 3) return np.tanh(z - 1.96 * se), np.tanh(z + 1.96 * se) for rho in (0.054, -0.38): t = rho * np.sqrt((n - 2) / (1 - rho**2)) p = 2 * stats.t.sf(abs(t), n - 2) lo, hi = ic_spearman(rho, n) print(f"rho = {rho:+.3f}: 95% CI [{lo:+.2f}, {hi:+.2f}], " f"p = {p:.2f}") ``` ```output 21 models | earn in stocks: 21 | earn on Polymarket: 4 Spearman cr_acc vs cr_poly: 0.17 Spearman sr_acc vs sr_poly: 0.06 rho = +0.054: 95% CI [-0.41, +0.50], p = 0.82 rho = -0.380: 95% CI [-0.71, +0.09], p = 0.09 ``` Three readings. First: the correlation between markets, computed from the table itself, is small both in return and in Sharpe, as the authors say. Second: with 21 models, the 95% interval of a rank correlation of 0.054 goes from −0.41 to 0.50: it does **not** rule out a moderate relationship in either direction, and the −0.38 on Polymarket, with p = 0.09, is not significant at 5% either. The conclusion the data support is the one in the paper's abstract, “high LMArena scores do not imply superior trading outcomes”, not that capability does not matter. Third: that all 21 earn money in stocks over 50 days says more about the market in those weeks than about the models; without a passive portfolio in the table there is no way to separate beta from skill. ## The same question with a known answer With real data nobody knows how much a model's capability really helps. With the course data it can be set. The cell builds a bench of 21 simulated agents in the style of LiveTradeBench: the **15 largest companies** of level 6, a decision at the close of each session with the news published up to 16:00 New York time and the 5-session return, and a daily allocation between the 15 stocks and cash. The news corpus is the level's sample corpus, in Spanish, read as is. Each agent has: - a simulated **general score** (like LMArena's), between 1,300 and 1,450; - a news **reading quality** that **grows with the score**: capability really helps, because in the data the news move the price (the planted effect in the README); - a **risk appetite** (fraction in cash), an **aggressiveness** and a taste for momentum that have nothing to do with the score; - an **inference noise**: each decision reads the evidence with an error that changes in every run. ```python import re emp = pd.read_csv("acciones6_empresas.csv") hist = pd.read_csv("acciones6_tickers.csv") ID = dict(zip(hist["ticker"], hist["id_empresa"])) ACTUAL = dict(zip(emp["id_empresa"], emp["ticker"])) px = pd.read_parquet("acciones6_precios.parquet") cierres = px.pivot(index="fecha", columns="ticker", values="cierre") acciones = emp.set_index("ticker")["acciones_millones"] U = list((cierres.loc["2019-01-02"] * acciones).nlargest(15).index) r = cierres[U].pct_change().loc["2019-01-02":] F = r.index R, R5 = r.to_numpy(), cierres[U].pct_change(5).loc[F].to_numpy() lex = pd.read_csv("texto6_lexico.csv").query("lista == 'financiera'") POS = set(lex.query("categoria == 'positiva'")["palabra"]) NEG = set(lex.query("categoria == 'negativa'")["palabra"]) def tono(t): pal = re.findall(r"\w+", t.lower()) p, n_ = sum(w in POS for w in pal), sum(w in NEG for w in pal) return (p - n_) / (p + n_ + 1) nt = pd.read_parquet("noticias6_corpus.parquet") nt["tk"] = nt["ticker"].map(ID).map(ACTUAL) nt = nt[nt["tk"].isin(U)] nt["clave"] = nt["titular"].str.replace( r"^(?:ACTUALIZA|Última hora|UPDATE|Breaking):\s*", "", regex=True, case=False) nt = nt.sort_values("ts").drop_duplicates(["tk", "clave"]) nt["tono"] = (nt["titular"] + " " + nt["cuerpo"]).map(tono) ny = nt["ts"].dt.tz_convert("America/New_York").dt.tz_localize(None) i = np.searchsorted(F + pd.Timedelta(hours=16), ny, side="left") nt = nt[i < len(F)].assign(dia=F[i[i < len(F)]]) T = (nt.groupby(["dia", "tk"])["tono"].sum().unstack() .reindex(index=F, columns=U).fillna(0).to_numpy()) rng = np.random.default_rng(644) NA = 21 puntuacion = np.round(1300 + 150 * rng.random(NA)) calidad = 0.2 + 0.6 * (puntuacion - 1300) / 150 # rises with the score caja = rng.uniform(0.0, 0.6, NA) # independent of it kappa = rng.uniform(0.5, 3.0, NA) gusto_mom = rng.normal(0, 1, NA) def ejecutar(a, t0, n=50, semilla=0, ruido=1.0): """Cumulative return of agent `a` over n sessions from t0.""" g = np.random.default_rng([semilla, a, t0]) v = 1.0 for t in range(t0, t0 + n): lectura = (calidad[a] * T[t - 1] + ruido * g.standard_normal(len(U))) s = kappa[a] * (lectura + 6 * gusto_mom[a] * R5[t - 1]) w = np.exp(s - s.max()) w = (1 - caja[a]) * w / w.sum() v *= 1 + w @ R[t] return v - 1 t0 = F.searchsorted(pd.Timestamp("2025-08-18")) cr = np.array([ejecutar(a, t0) for a in range(NA)]) print(F[t0].date(), "to", F[t0 + 49].date(), "| news on", f"{(T != 0).mean():.1%} of stock-days") print("Spearman(score, return):", round(stats.spearmanr(puntuacion, cr)[0], 3)) print("returns (%):", np.round(100 * np.sort(cr), 1)) ``` ```output 2025-08-18 to 2025-10-27 | news on 9.0% of stock-days Spearman(score, return): 0.316 returns (%): [-7.8 -5.3 -4.3 -4.1 -4. -3. -2.5 -2.4 -2.3 -1.1 -0.8 -0.7 -0.5 -0.3 0. 0.2 0.5 1.3 1.6 2. 6.7] ``` On the same dates as LiveTradeBench, the simulated bench gives a correlation of 0.32 between the score and the return, and the returns of the 21 agents go from −7.8% to +6.7%. Is 0.32 the bench's “truth”? It cannot be known from one window: over 50 sessions, what separates the agents is how much money they have invested while the market rises or falls and the luck of their readings, and the edge of reading well only shows up when there is news about those 15 companies, which is rare (9% of the stock-days have some). To see the size of the noise, the next cell repeats the bench in every non-overlapping 50-session window from 2019 to 2025 and compares it with the long-run correlation, that of the average return over all the windows. ```python ventanas = list(range(6, len(F) - 50, 50)) CR = np.array([[ejecutar(a, t) for a in range(NA)] for t in ventanas]) rho = np.array([stats.spearmanr(puntuacion, c)[0] for c in CR]) print(len(ventanas), "windows | mean", round(rho.mean(), 3), "| 5th and 95th percentiles:", np.round(np.percentile(rho, [5, 95]), 2), "| negative:", f"{(rho < 0).mean():.0%}") print("long run: Spearman(score, mean) =", round(stats.spearmanr(puntuacion, CR.mean(axis=0))[0], 3), "| Spearman(cash, mean) =", round(stats.spearmanr(caja, CR.mean(axis=0))[0], 3)) ``` ```output 35 windows | mean 0.049 | 5th and 95th percentiles: [-0.29 0.49] | negative: 43% long run: Spearman(score, mean) = 0.373 | Spearman(cash, mean) = -0.566 ``` *Fig. — Simulated bench of 21 agents: rank correlation between score and return in each 50-session window.* Vertical bars with the Spearman correlation between the general score of 21 simulated agents and their return in 35 windows of 50 sessions from 2019 to 2025: from −0.35 to 0.56, mean 0.05 and 43% negative. A highlighted line marks the long-run relation, 0.37. The bench's truth is positive: in the long run, with the 35 windows averaged, the correlation between the score and the average return is 0.37. But in a 50-session window it is a draw: the mean of the 35 is only 0.05, the central 90% goes from −0.29 to 0.49 and 43% of the windows give a negative correlation. And what best explains the average return is not capability but how much each agent invests: the correlation of the fraction in cash with the average return is −0.57, in a period in which the market rises. A 0.054 or a −0.38 in a single 50-day window is compatible with almost any true relationship. ## AlphaForgeBench: the same question does not get the same answer “AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models”, by Wentao Zhang and seven other authors (Nanyang Technological University, HKUST (Guangzhou) and Hong Kong Polytechnic University), arXiv:2602.18481, version 2 of May 27, 2026, with the KDD 2026 header. Its starting point is a critique of benchmarks like LiveTradeBench: when an LLM directly emits the trading actions, its decisions are unstable. In their words, LLMs “produce drastically different trading actions across runs even under deterministic decoding (Section C), rendering single-run live evaluations statistically unreliable”. The experiment in appendix C is simple: BTC with daily data from 2025, a prompt with 30 days of prices and technical indicators, one action per day (buy everything, hold or sell everything), no costs, and **five identical runs** of each model with temperature 0 and another five with 0.7. The results for gemini-3-pro-preview: | Measure | Temperature 0 | Temperature 0.7 | |---|---|---| | Action agreement between pairs of runs | 0.36 to 0.48 | 0.32 to 0.41 | | Average step-by-step agreement | 61.4% | — | | Steps with some disagreement among the five | 95.7% | — | | Trajectories | one falls to around −50%, others near zero | — | Other models are more stable (gemini-3-flash-preview agrees 85.3% with temperature 0), but even so the return curves “gradually fan out”. The authors attribute it to floating-point non-determinism and to the routing of mixture-of-experts models, amplified because LLMs have no memory of their own previous actions: a different decision early on changes all the following ones. And they write that in direct trading benchmarks “run-to-run variance routinely exceeds intermodel variance”. Their proposal (lesson 6.4.6) is for the LLM to write **factors and strategy code**, which are then run in a deterministic backtest: that way the variability is confined to a single generation step. ### Repeated runs in the simulated bench The cell repeats the bench five times in the 2025 window with different seeds for the inference noise. It measures the standard deviation of each agent's return across its runs, compares it with the standard deviation between agents and computes how similar the rankings of two runs are. At the end it removes the inference noise (the equivalent of truly deterministic decoding). ```python runs = np.array([[ejecutar(a, t0, semilla=s) for a in range(NA)] for s in range(5)]) dentro = runs.std(axis=0).mean() # same agent, 5 draws entre = runs.mean(axis=0).std() # between agents print(f"std. within each agent: {100 * dentro:.2f} pp | " f"between agents: {100 * entre:.2f} pp") print("Spearman between the ranking of run 0 and the others:", [round(float(stats.spearmanr(runs[0], runs[k])[0]), 2) for k in range(1, 5)]) sin_ruido = np.array([[ejecutar(a, t0, semilla=s, ruido=0.0) for a in range(NA)] for s in range(2)]) print("without inference noise, identical runs?", np.array_equal(sin_ruido[0], sin_ruido[1])) ``` ```output std. within each agent: 3.58 pp | between agents: 2.00 pp Spearman between the ranking of run 0 and the others: [-0.14, -0.17, 0.17, -0.34] without inference noise, identical runs? True ``` The standard deviation of the same agent across its five runs (3.58 points) is almost double the standard deviation between agents (2.00), and the ranking of one run does not predict that of the others (correlations from −0.34 to 0.17): it is the pattern AlphaForgeBench describes in real models. Without inference noise the bench becomes reproducible, but the ranking is still that of a single 50-session window, with all the noise of the previous section. ### A decision with no margin in the local model Greedy decoding of the course's local model is deterministic: the same text always gives the same answer (lesson 6.2.1). The problem is a different one: **the text changes for irrelevant reasons**. The cell asks the Qwen2.5-0.5B-Instruct model whether it would buy, hold or sell a stock that has risen 1.5% in a week with news about a lawsuit, written in eight equivalent ways, and looks at the logits of the first token of the answer: the chosen option and its margin over the second. The prompt is in Spanish, like the rest of level 6's model inputs: “La acción ha variado un +1,5 % esta semana y hay una noticia sobre una demanda de un cliente” reads “The stock has changed by +1.5% this week and there is news about a lawsuit from a customer”, and the model must answer with one word, COMPRAR, MANTENER or VENDER (buy, hold or sell). Note that “1,5” uses the Spanish decimal comma. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer torch.manual_seed(0) torch.use_deterministic_algorithms(True) NOMBRE = "Qwen/Qwen2.5-0.5B-Instruct" tok = AutoTokenizer.from_pretrained(NOMBRE) llm = AutoModelForCausalLM.from_pretrained(NOMBRE, dtype=torch.float32) D = "hay una noticia sobre una demanda de un cliente" variantes = [ f"La acción ha variado un +1,5 % esta semana y {D}.", f"La acción ha variado un +1.5 % esta semana y {D}.", f"La acción ha variado un 1,5 % esta semana y {D}.", f"La acción ha subido un 1,5 % esta semana y {D}.", f"Esta semana la acción ha variado un +1,5 % y {D}.", f"La acción ha variado un +1,5 % esta semana; {D}.", f"La acción ha variado un +1,50 % esta semana y {D}.", f"La acción ha variado un +1,5% esta semana y {D}."] PIDE = " Responde solo con una palabra: COMPRAR, MANTENER o VENDER." ids = {o: tok(o, add_special_tokens=False)["input_ids"][0] for o in ("COMPRAR", "MANTENER", "VENDER")} for v in variantes: x = tok.apply_chat_template([{"role": "user", "content": v + PIDE}], add_generation_prompt=True, return_tensors="pt", return_dict=True) with torch.inference_mode(): lg = llm(**x).logits[0, -1] l = {o: float(lg[i]) for o, i in ids.items()} a, b = sorted(l, key=l.get, reverse=True)[:2] print(f"{a:8} margin {l[a] - l[b]:.2f} | {v[:46]}") ``` ```output COMPRAR margin 0.05 | La acción ha variado un +1,5 % esta semana y h VENDER margin 0.09 | La acción ha variado un +1.5 % esta semana y h VENDER margin 0.07 | La acción ha variado un 1,5 % esta semana y ha VENDER margin 0.04 | La acción ha subido un 1,5 % esta semana y hay VENDER margin 0.34 | Esta semana la acción ha variado un +1,5 % y h COMPRAR margin 0.35 | La acción ha variado un +1,5 % esta semana; ha COMPRAR margin 0.14 | La acción ha variado un +1,50 % esta semana y VENDER margin 0.02 | La acción ha variado un +1,5% esta semana y ha ``` With “+1,5 %” the model buys; with “+1.5 %” (decimal point), without the sign, with “subido” (risen) instead of “variado” (changed), with the sentence in another order or without a space before the “%”, it sells; with a semicolon or with “+1,50 %”, it buys again. The margins between the chosen option and the next one go from 0.02 to 0.35 logits: the “decision” is almost a tie that any detail of the text settles. A bigger model will have different margins, and the lesson does not extrapolate this 0.5B model to commercial models; what it does teach is what to measure before trusting an agent: **how much its decision changes under irrelevant perturbations** of the prompt, the data format and the order of the information. With a model served through an API, the sources of variation that AlphaForgeBench points out and that the user does not control are added on top. ## From backtest to live: the degradation of deep learning models The syllabus states that deep learning models that are strong in the backtest degrade live. The most recent primary source that documents it with a live phase is AlphaCrafter (Yishuo Yuan and four other authors, Nanjing University, arXiv:2605.05580, version 2 of July 28, 2026). Its protocol: daily portfolios on the CSI 300 and the S&P 500, training from 2016 to 2022, validation in 2023, a **backtest from January 2024 to February 2026** (519 and 542 sessions) and a **live phase with a real broker's paper trading API** from March to June 2026 (69 and 73 sessions), after the training cutoff of all the LLMs they use. Their text: “Machine learning and deep learning methods, notably LSTM, demonstrate strong backtesting performance but exhibit pronounced degradation in live trading, exposing their vulnerability to overfitting and regime shifts. This pattern is consistent across both markets”. ```python # Table 2 of arXiv:2605.05580 v2: annualized Sharpe, methods without LLMs ac = pd.DataFrame([ ("MACD", 0.99, 0.87, 0.72, 1.22), ("Grid", 0.33, -0.26, 1.11, 0.39), ("LightGBM", 1.22, 0.51, 0.51, 1.19), ("XGBoost", 1.24, 0.36, -0.16, 1.01), ("LSTM", 1.43, 0.35, 1.38, 0.83), ("Transformer", 1.19, -0.32, 0.44, 0.28), ("TRA", 1.16, 0.71, 1.01, 1.02)], columns=["metodo", "csi_bt", "csi_vivo", "sp_bt", "sp_vivo"]) SES = {"csi": (519, 69), "sp": (542, 73)} # sessions bt / live se = lambda n: np.sqrt(252 / n) # standard error of the SR for m, (nb, nv) in SES.items(): d = ac[f"{m}_vivo"] - ac[f"{m}_bt"] s = np.hypot(se(nb), se(nv)) print(f"{m}: {(d < 0).sum()} of 7 drop | standard error of the SR: " f"bt {se(nb):.2f}, live {se(nv):.2f}, difference {s:.2f}") ac.set_index("metodo") ``` ```output csi: 7 of 7 drop | standard error of the SR: bt 0.70, live 1.91, difference 2.03 sp: 3 of 7 drop | standard error of the SR: bt 0.68, live 1.86, difference 1.98 csi_bt csi_vivo sp_bt sp_vivo metodo MACD 0.99 0.87 0.72 1.22 Grid 0.33 -0.26 1.11 0.39 LightGBM 1.22 0.51 0.51 1.19 XGBoost 1.24 0.36 -0.16 1.01 LSTM 1.43 0.35 1.38 0.83 Transformer 1.19 -0.32 0.44 0.28 TRA 1.16 0.71 1.01 1.02 ``` *Fig. — AlphaCrafter: backtest (grey) and live (blue) Sharpe of seven non-LLM methods, CSI 300 and S&P 500.* Two panels of points joined by a line with the backtest Sharpe (January 2024 to February 2026) and the live Sharpe (March to June 2026) of MACD, Grid, LightGBM, XGBoost, LSTM, Transformer and TRA. In the CSI 300 all seven fall; LSTM, highlighted, goes from 1.43 to 0.35. In the S&P 500 three fall, LSTM among them (1.38 to 0.83), and MACD, LightGBM and XGBoost rise. A careful reading has two parts. The one that supports the syllabus: on the CSI 300 the seven methods without LLMs drop from backtest to live, the LSTM goes from 1.43 to 0.35 and the Transformer from 1.19 to −0.32; on the S&P 500 the LSTM also drops (from 1.38 to 0.83). The one that calls for nuance: on the S&P 500 only three of seven drop; MACD, LightGBM and XGBoost clearly **rise** live and TRA stays the same, and the standard error of an annualized Sharpe measured over some 70 sessions is almost 1.9, so no individual drop can be distinguished from noise; the difference between backtest and live has a standard error of about 2. Moreover, the authors propose a competing method, which degrades less in their table. The pattern is consistent with what the course has already measured in other ways: validation overfitting (lesson 5.5.4), drift (lesson 5.10.2), the decay of published anomalies (lesson 5.1.1) and the collapse of foundation model Sharpes with costs (lesson 5.8.4). But a three-month live window does not prove it on its own. Two more studies complete the picture of LLM agents. FINSABER (Li, Kim, Cucuringu and Ma, arXiv:2505.07078) repeats LLM *timing* strategies over two decades and with more than 100 stocks, and finds that the published advantages “deteriorate significantly under broader cross-section and over a longer-term evaluation”; the strategies are too conservative in bull markets and too aggressive in bear markets. DeepFund (Li et al., arXiv:2505.11065) evaluates nine models live, with data after each one's cutoff, because backtests let LLMs “time travel”, and finds that even DeepSeek-V3 and Claude-3.7-Sonnet have net losses. ## How to evaluate an agent without fooling yourself | Question | Red flag | What to ask for | |---|---|---| | When was the model trained? | Backtest before the cutoff | Period after the effective cutoff, live or in paper trading (lesson 6.2.5) | | How many runs? | One | Several seeds or repetitions and the distribution of the result | | What is it compared against? | Technical rules or nothing | Passive portfolio with the same assets and the same information combined without an LLM | | Costs? | Not mentioned | Commissions, bid-ask spread and slippage with the actual turnover (lesson 2.4.7) | | How many sessions and models? | 50 days, 21 models | Standard error smaller than the difference you want to detect | | How many configurations were tried? | Not stated | Deflated Sharpe (lesson 2.5.4) | | Is the decision stable? | Not measured | Sensitivity to irrelevant perturbations of the prompt and the data | ## Common mistakes - **Choosing the model by a general ranking.** In LiveTradeBench the correlation with return was 0.054 in stocks and −0.38 on Polymarket. - **Reading a 21-point correlation as if it were precise.** Its 95% interval is almost 0.9 wide. - **Publishing a single run.** An agent's variability between runs can exceed the variability between agents. - **Trusting temperature 0.** It fixes the decoding, not the input text, nor the numerical precision, nor the server that responds. - **Taking three months live as proof.** The standard error of a Sharpe over 70 sessions is almost 1.9. - **Forgetting beta.** If all the agents earn money in a rising market, nobody has demonstrated skill. ## Exercises 1. How many models would you need to compare for a true Spearman correlation of 0.18 (that of the simulated bench in the long run) to be detected with 80% power at 5% two-sided? Use the Fisher approximation with Fieller's error. **Solution.** ```python def modelos_necesarios(rho, alfa=0.05, potencia=0.8): z = stats.norm.ppf(1 - alfa / 2) + stats.norm.ppf(potencia) return int(np.ceil((1.06 * z / np.arctanh(rho)) ** 2 + 3)) for rho in (0.18, 0.3, 0.5): print(rho, "->", modelos_necesarios(rho), "models") ``` ```output 0.18 -> 270 models 0.3 -> 96 models 0.5 -> 33 models ``` For a true correlation of 0.18 you would need on the order of 270 models, more than ten times those of LiveTradeBench; for 0.5, about 33. And that with the noise of a fixed window: in the simulated bench, much of the dispersion comes from the 50-session window, so more windows would also be needed. An agent benchmark with a few dozen models can only detect strong relationships. 2. Repeat the simulated bench with 250-session windows (one year) instead of 50. How do the mean and the 5th and 95th percentiles of the correlation change? **Solution.** ```python filas = [] for t in range(6, len(F) - 250, 250): c = [ejecutar(a, t, n=250) for a in range(NA)] filas.append({"inicio": F[t].date(), "mercado_%": 100 * (np.prod(1 + R[t:t + 250] .mean(axis=1)) - 1), "rho_puntuacion": stats.spearmanr(puntuacion, c)[0], "rho_caja": stats.spearmanr(caja, c)[0]}) v = pd.DataFrame(filas).set_index("inicio") print("mean of rho_puntuacion:", round(v["rho_puntuacion"].mean(), 3)) v.round(2) ``` ```output mean of rho_puntuacion: 0.027 mercado_% rho_puntuacion rho_caja inicio 2019-01-10 17.87 0.07 -0.10 2020-01-08 12.65 0.25 -0.34 2021-01-05 17.37 0.03 -0.06 2021-12-31 -26.11 0.09 0.56 2022-12-29 10.51 -0.13 -0.15 2023-12-28 26.44 0.06 -0.33 2024-12-26 6.44 -0.20 -0.29 ``` With one-year windows seven fit in the data, and the correlation does not stabilize either: it goes from −0.20 to 0.25, with a mean of 0.03, far from the long-run 0.37. The cash column explains it: in the window that starts at the end of 2021, with the market of the 15 companies falling 26%, the correlation of cash with the return is 0.56. Lengthening the window is not enough when skill is small compared with the inference noise, which accumulates throughout the year, and with the market exposure: in a bear year the agents with more cash win, whatever their capability. The true relationship only appears by averaging many different periods, and even then, with few agents (exercise 1), the error is large. 3. No code: design the evaluation of a research agent that proposes buys and sells on 50 stocks every week. You have a budget for one year of calls to a commercial model. **Solution.** First, dates: only weeks after the model's effective cutoff, in paper trading or in shadow mode (lesson 4.5.3), and no backtest before the cutoff except for debugging. Second, repetition: every week, several independent runs of the same decision (for example, five) to measure stability and use the majority decision; the budget is split between runs and sessions, not spent on a single long draw. Third, benchmarks computed with the same information without an LLM: the passive portfolio of the 50 stocks and a simple rule with the same signals (lesson 6.4.3). Fourth, costs with the actual turnover. Fifth, perturbations: the same prompt with irrelevant changes (number format, order of the news) on a sample of decisions, to estimate how much the decision changes. And at the end, the honest count: with 52 weeks, the standard error of an annualized Sharpe is 1; the year will serve to discard bad or unstable agents, not to prove that one is good. ## Self-assessment 1. In LiveTradeBench, the Spearman correlation between the LMArena score and the cumulative stock return of 21 models is 0.054. What can be concluded? - A. That in that 50-day window the general ranking did not order returns; with 21 models, the 95% interval of that correlation goes roughly from −0.41 to 0.50, so it does not rule out a moderate relationship either - B. That a model's general capability has no relationship at all with its ability to invest - C. That the models with the highest scores lose money - D. That you should pick the model with the lowest LMArena score *Answer A: That in that 50-day window the general ranking did not order returns; with 21 models, the 95% interval of that correlation goes roughly from −0.41 to 0.50, so it does not rule out a moderate relationship either.* With n = 21 the standard error of a rank correlation is on the order of 0.25. The authors' conclusion, that a high score does not imply better results, holds; the conclusion that there is no relationship at all does not. The lesson's simulation shows how a true positive relationship produces, over 50 sessions, correlations from −0.29 to 0.49 most of the time. 2. GPT-4.1 had the highest stock return (6.25%) and lost 33.69% on Polymarket. Which reading does the paper itself support? - A. That GPT-4.1 is the best model for investing in stocks - B. That Polymarket is an inefficient market for any model - C. That the result in one market does not generalize to the other: the Sharpe correlation between the two markets is almost zero, and the authors attribute the loss to exaggerated portfolio changes under higher volatility - D. That the paper deducts commissions and that is why the figures are reliable *Answer C: That the result in one market does not generalize to the other: the Sharpe correlation between the two markets is almost zero, and the authors attribute the loss to exaggerated portfolio changes under higher volatility.* The authors write that performance in one market does not generalize to the other and that GPT-4.1 performed poorly on Polymarket probably because of exaggerated allocation changes. Moreover, the paper acknowledges that it does not account for commissions, bid-ask spreads or liquidity, and in 50 days the 21 stock portfolios earned between 1.78% and 6.25%. 3. AlphaForgeBench repeats the same BTC trading task five times with temperature 0. What does it find for gemini-3-pro-preview? - A. That with temperature 0 the five runs are identical - B. An agreement between pairs of runs of only 0.36 to 0.48 in the actions, and trajectories so different that one run falls to around −50% while others stay near zero - C. That temperature has no influence at all - D. That the model always gets the direction of BTC right *Answer B: An agreement between pairs of runs of only 0.36 to 0.48 in the actions, and trajectories so different that one run falls to around −50% while others stay near zero.* The authors attribute the variability under nominally deterministic decoding to floating-point non-determinism and to the routing of mixture-of-experts models, amplified because a different action early on changes all the following ones. That is why they propose evaluating the strategy code the LLM generates, which is then run deterministically. 4. In AlphaCrafter, the LSTM goes on the S&P 500 from a Sharpe of 1.38 in the backtest (542 sessions) to 0.83 live (73 sessions). What has to be added before concluding? - A. Nothing: the drop proves that the LSTM is overfitted - B. That the LSTM works better live than in the backtest - C. That the result shows that deep learning is useless in finance - D. That an annualized Sharpe measured over 73 sessions has a standard error of almost 1.9, so that particular drop fits within the noise; the pattern is more informative (on the CSI 300 all seven methods without LLMs drop) and is consistent with other evidence of overfitting *Answer D: That an annualized Sharpe measured over 73 sessions has a standard error of almost 1.9, so that particular drop fits within the noise; the pattern is more informative (on the CSI 300 all seven methods without LLMs drop) and is consistent with other evidence of overfitting.* The standard error of the annualized Sharpe is approximately the square root of 252 divided by the number of sessions: 1.86 with 73. A single three-month live window is not enough to measure the degradation of a method; several signals in the same direction, with a known mechanism (overfitting and regime shifts), do point to it. 5. In the lesson, the local 0.5B model decides COMPRAR (buy) or VENDER (sell) given the same information depending on whether it reads “+1,5 %” or “+1.5 %”. What does this indicate? - A. That the model does not understand the decimal comma - B. That numbers should always be written with a point - C. That its decision hinges on differences of hundredths between the logits of the two options: any irrelevant change in the text reverses it, and a decision like that cannot be the basis of an order - D. That greedy decoding is random *Answer C: That its decision hinges on differences of hundredths between the logits of the two options: any irrelevant change in the text reverses it, and a decision like that cannot be the basis of an order.* In the lesson's eight formats, the difference between the logit of the first option and that of the second goes from 0.02 to 0.35, and the model chooses COMPRAR in three and VENDER in five. Greedy decoding is deterministic for a given text, but the text itself changes for irrelevant reasons. A bigger model will have different margins, but the underlying problem, decisions with no margin, is measured the same way. ## Key points - LiveTradeBench (Yu, Li and You, arXiv:2511.03628): 21 LLMs live from August 18 to October 24, 2025; Spearman between LMArena and return, 0.054 in stocks and −0.38 on Polymarket; GPT-4.1, +6.25% in stocks and −33.69% on Polymarket; no transaction costs. - With 21 models a rank correlation has a standard error of about 0.25: 0.054 does not distinguish “nothing” from “something moderate”. What is clear is that the general ranking does not order the results. - AlphaForgeBench (arXiv:2602.18481) documents “drastically different” actions between runs of the same model with deterministic decoding: a single-run backtest is not reproducible. - AlphaCrafter (arXiv:2605.05580) observes that deep learning methods, especially the LSTM, perform well in the backtest and degrade live; with 69-73 live sessions each drop fits within the noise, but the pattern repeats. - In a simulated bench where capability really helps, 50 sessions give correlations between −0.29 and 0.49, and the variability of the same agent between runs is almost double that between agents. - Evaluating an agent requires periods after the model's cutoff, several runs, passive baselines, costs and enough sessions for the standard error to be smaller than the difference you want to measure. ## Resources ### Books - [Yu, H., Li, F. and You, J. (2025): LiveTradeBench: Seeking Real-World Alpha with Large Language Models (arXiv:2511.03628, v1)](https://arxiv.org/abs/2511.03628) · arxiv.org - [Zhang, W. et al. (2026): AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models (arXiv:2602.18481, v2)](https://arxiv.org/abs/2602.18481) · arxiv.org - [Yuan, Y. et al. (2026): AlphaCrafter: Harnessing Multi-Agent Workflows for Cross-Sectional Quantitative Trading (arXiv:2605.05580, v2)](https://arxiv.org/abs/2605.05580) · arxiv.org - [Li, W. W., Kim, H., Cucuringu, M. and Ma, T. (2025): Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? (arXiv:2505.07078)](https://arxiv.org/abs/2505.07078) · arxiv.org - [Li, C. et al. (2025): Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking (arXiv:2505.11065)](https://arxiv.org/abs/2505.11065) · arxiv.org ### Data - [acciones6_precios.parquet: prices of the 120 level 6 companies (synthetic data)](https://pythonparatrading.com/curso/datos/nivel-6/acciones6_precios.parquet) · Download · acciones6_precios.parquet Previous lesson: [6.4.3 · TradingAgents-style multi-agent architectures](https://pythonparatrading.com/en/python-trading-course/level-6/multi-agent-architectures-like-tradingagents/) Next lesson: [6.4.5 · Guardrails for agents: the agent proposes, the engine decides](https://pythonparatrading.com/en/python-trading-course/level-6/guardrails-for-ai-agents/)