Lesson 6.4.3 · 75 min
TradingAgents-style multi-agent architectures
TradingAgents splits the investment decision among analysts, researchers who debate, a trader and a risk team, like a real desk. This lesson reads the paper with its figures and its limits, builds the same architecture on the course data with simulated agents and measures it honestly: against the market, against each signal alone and against the deterministic rule the debate converges to, with costs.
- Module
- 6.4 · AI agents for trading
- Duration
- 75 min
- Prerequisites
- 6.4.2 · Agent frameworks and MCP, 6.4.1 · Tool-calling agents: tool calling, ReAct and planning, 6.1.3 · Event studies with text signals, timestamp alignment and topic modeling, 2.4.7 · Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing, 2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)
- Author
- Jesús Cuesta
- Updated
Level 6 lessons
6.1 · Classical and modern financial NLP
6.2 · LLMs for research and trading
6.3 · Financial RAG
6.4 · AI agents for trading
- 6.4.1 Tool-calling agents: tool calling, ReAct and planning
- 6.4.2 Agent frameworks and MCP
- 6.4.3 TradingAgents-style multi-agent architectures
- 6.4.4 Evidence and limits of agents
- 6.4.5 Guardrails for agents: the agent proposes, the engine decides
- 6.4.6 Research agents: hypotheses, alpha mining and review
6.5 · AI as the quant's copilot
6.6 · Big data for trading
- 6.6.1 When data becomes “big”: ticks, order books, options and universes
- 6.6.2 Parquet, Arrow, Polars streaming and out-of-core DuckDB
- 6.6.3 Dask, Spark and Ray: from one machine to a cluster
- 6.6.4 Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+
- 6.6.5 Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF)
6.7 · Alternative data
An investment desk made of agents
Lesson 6.4.2 built a workflow with a single writer. The idea of multi-agent architectures is to split the work among agents with different roles, as on an investment desk: some gather data and write reports, others argue, someone decides and someone controls the risk. The most cited reference is TradingAgents, by Yijia Xiao, Edward Sun, Di Luo and Wei Wang (UCLA, MIT and Tauric Research), arXiv:2412.20138; what follows comes from the PDF of version 7 (June 3, 2025), read on 2026-10-23. Its organization has five pieces:
- Analyst team: four agents work at the same time, each with its own tools: fundamental (financial statements, earnings reports, insider transactions), sentiment (social media posts and sentiment scores), news (news, government announcements and macroeconomic indicators) and technical (indicators such as MACD and RSI, prices and volumes). Each one writes a structured report.
- Research team: a bullish researcher and a bearish one read the reports and debate in natural language for n rounds; a facilitator reviews the debate and picks the prevailing perspective.
- Trader: turns the analysis into a decision (buy, sell or hold) with its justification.
- Risk management team: three agents, one risk-seeking, one neutral and one conservative, discuss for n rounds and adjust the trader's plan to the risk limits.
- Fund manager: reviews that discussion, decides the adjustments and approves the trade.
Mermaid source
flowchart TB A[Fundamental] --> R[Bull vs bear] B[Sentiment] --> R C[News] --> R D[Technical] --> R R --> F[Facilitator] F --> T[Trader] T --> G[Risk team] G --> M[Fund manager]
Two design decisions in the paper deserve attention. The agents communicate mainly through structured documents
kept in a global state, not through long conversations, so as not to lose information in a game of “telephone”; only
the debates use dialogue. And they split the models: quick models (gpt-4o-mini, gpt-4o) to summarize and retrieve
data, and reasoning models (o1-preview) to analyze and decide. The public code (repository
TauricResearch/TradingAgents, Apache-2.0 license) is built with LangGraph (lesson 6.4.2), lets you configure the
number of debate and risk rounds and warns in its README that it is designed for research and that performance may
vary with the model, the temperature, the period, the data “and other non-deterministic factors”.
What the paper measures, and what it does not
The original evaluation is a backtest from January 1 to March 29, 2024 on large tech stocks, with data on prices, news, social media, insider transactions, financial statements and 60 technical indicators. The comparisons are with five rules: buy and hold, MACD, KDJ with RSI, ZMR and SMA. Table 1 shows three stocks:
| Stock | TradingAgents: cum. return | Sharpe | Buy and hold: cum. return |
|---|---|---|---|
| AAPL | 26.62% | 8.21 | −5.23% |
| GOOGL | 24.36% | 6.39 | 7.78% |
| AMZN | 23.21% | 5.60 | 17.1% |
Before reading those figures as an edge, you have to set next to them what the paper itself says and what it does not say:
- Three months. A footnote explains that the backtest lasts three months “due to intensive LLM and tool use (11 LLM calls & 20+ tool calls/prediction)”, and that the highest Sharpe “exceeds our expected empirical range”; the authors attribute it to there having been few pullbacks in that period. An annualized Sharpe measured over some 60 sessions has a standard error of more than 2 points.
- No declared costs. The text does not mention commissions, bid-ask spreads or slippage.
- Possible look-ahead bias. The models used were trained on Internet text; the paper states that the agents only see data up to each day, but does not discuss whether the models knew from their training what happened to those stocks (lesson 6.2.5).
- Weak baselines. Single-stock technical rules, not portfolios or quantitative models.
There is a later independent evaluation, with every caveat because its authors propose a competing method: AlphaCrafter (Yuan et al., arXiv:2605.05580, version 2 of July 2026) reimplements TradingAgents on CSI 300 and S&P 500 portfolios with a backtest from January 2024 to February 2026 and a live phase from March to June 2026 with a paper trading API. With Claude Opus 4.6 as the model, TradingAgents goes on the CSI 300 from a Sharpe of 1.37 in the backtest to 0.67 live, and on the S&P 500 from 1.08 to 1.25; the authors describe the “role-playing” agents as unstable depending on the model used. Far, in any case, from a Sharpe of 8.
The four analysts on the course data
To measure an architecture like this, you have to know what signal there is in the data. The level 6 synthetic data make that possible: the 120 companies have planted effects with parameters published in the README. The lesson builds the four analysts with point-in-time data and one decision a week, at the close of the last day of each week, from January 2019 to December 2025. The sample news and social media corpus is in Spanish, like the rest of level 6, and the code reads it as is:
| Analyst | Data | Signal | Planted effect (README) |
|---|---|---|---|
| Fundamental | panel5_features: book value / price and SUE | g(valor_bp) + g(sue)·e^(−days/45) | Value and earnings surprise (level 5) |
| Technical | panel5_features: 12-1 momentum and the week's return | g(mom_12_1) − g(ret_1s) | Momentum (with crashes in rebounds) and weekly reversal |
| News | News without copies up to 16:00 NY | Sum of the week's dictionary tone | Reaction and 5-session drift |
| Sentiment | Cashtag posts from accounts older than 60 days | Count of bullish and bearish words | 10 bp the next day per unit of opinion, daily AR(1) 0.7 |
g(·) is the cross-sectional normal rank of each week. Each post is attributed to the company of its cashtag (with
the ticker in force on its date, which the ticker table translates to the current one). Posts from accounts 60 days
old or younger are excluded because the generator's automated accounts are recent and have few followers: filtering by
age is a heuristic, not the truth of the README (41% of the excluded accounts are bots; of those that remain, 16%).
import re
import numpy as np
import pandas as pd
from scipy import stats
emp = pd.read_csv("acciones6_empresas.csv")
hist = pd.read_csv("acciones6_tickers.csv")
ID = dict(zip(hist["ticker"], hist["id_empresa"]))
ACTUAL = dict(zip(emp["id_empresa"], emp["ticker"]))
TK = sorted(emp["ticker"])
px = pd.read_parquet("acciones6_precios.parquet")
cierres = px.pivot(index="fecha", columns="ticker",
values="cierre")[TK]
f5 = pd.read_parquet("panel5_features.parquet", columns=[
"fecha", "ticker", "mom_12_1", "ret_1s", "valor_bp", "sue",
"dias_desde_informe"])
f5 = f5[f5["ticker"].isin(TK) & (f5["fecha"] >= "2019-01-01")]
FECHAS = pd.DatetimeIndex(sorted(f5["fecha"].unique()))
sem = cierres.reindex(FECHAS)
r_sig = sem.shift(-1) / sem - 1 # following week
CORTES = FECHAS + pd.Timedelta(hours=16) # New York close
def g(df):
"""Cross-sectional normal rank of each date (rows)."""
rk = df.rank(axis=1).sub(0.5)
return pd.DataFrame(stats.norm.ppf(rk.div(df.count(axis=1),
axis=0)),
index=df.index, columns=df.columns)
def semana_de(ts):
"""First decision date whose close is later than `ts`."""
ny = ts.dt.tz_convert("America/New_York").dt.tz_localize(None)
i = np.searchsorted(CORTES, ny, side="left")
return pd.Series(FECHAS[i.clip(0, len(FECHAS) - 1)],
index=ts.index).where(i < len(FECHAS))
print(len(FECHAS), "weeks,", FECHAS[0].date(), "to",
FECHAS[-1].date(), "|", len(TK), "companies")366 weeks, 2019-01-04 to 2025-12-31 | 120 companiespiv = lambda c: f5.pivot(index="fecha", columns="ticker",
values=c)[TK]
fun = g(piv("valor_bp")) + g(piv("sue")) * np.exp(
-piv("dias_desde_informe") / 45)
tec = g(piv("mom_12_1")) - g(piv("ret_1s"))
lex = pd.read_csv("texto6_lexico.csv").query("lista == 'financiera'")
POS = set(lex.query("categoria == 'positiva'")["palabra"])
NEG = set(lex.query("categoria == 'negativa'")["palabra"])
def tono(t):
pal = re.findall(r"\w+", t.lower())
p, n = sum(w in POS for w in pal), sum(w in NEG for w in pal)
return (p - n) / (p + n + 1)
nt = pd.read_parquet("noticias6_corpus.parquet")
PREF = r"^(?:ACTUALIZA|Última hora|UPDATE|Breaking):\s*"
nt["clave"] = nt["titular"].str.replace(PREF, "", regex=True,
case=False)
nt = nt.sort_values("ts").drop_duplicates(["ticker", "clave"])
nt["tono"] = (nt["titular"] + " " + nt["cuerpo"]).map(tono)
nt["semana"] = semana_de(nt["ts"])
nt["tk"] = nt["ticker"].map(ID).map(ACTUAL)
noti = g(nt.groupby(["semana", "tk"])["tono"].sum().unstack()
.reindex(index=FECHAS, columns=TK)).fillna(0.0)
so = pd.read_parquet("social6_posts.parquet")
so["tk"] = (so["texto"].str.extract(r"\$([A-Z]{4})")[0]
.map(ID).map(ACTUAL))
so = so[so["antiguedad_cuenta_dias"] > 60] # drop new accounts
ALC = ["bien", "cargando", "rompe", "gusta", "compra", "aguantando",
"sorpresa", "buenos", "strong", "loading", "breaking out",
"holding", "buy", "higher"]
BAJ = ["pique", "vendido", "cuidado", "pierde", "castigo", "levanta",
"weak", "shorting", "sold", "dead cat", "breaking down",
"avoid", "bleeding"]
tl = so["texto"].str.lower()
cuenta = lambda ws: sum(tl.str.contains(w, regex=False).astype(int)
for w in ws)
so["s"] = (cuenta(ALC) - cuenta(BAJ)).clip(-1, 1)
so["semana"] = semana_de(so["ts"])
soc = g(so.groupby(["semana", "tk"])["s"].mean().unstack()
.reindex(index=FECHAS, columns=TK)).fillna(0.0)
SEN = {"fundamental": fun, "tecnico": tec, "noticias": noti,
"sentimiento": soc}
def ic_semanal(sig):
a = sig.iloc[:-1].rank(axis=1)
b = r_sig.iloc[:-1].rank(axis=1)
a, b = a.sub(a.mean(axis=1), axis=0), b.sub(b.mean(axis=1), axis=0)
with np.errstate(invalid="ignore", divide="ignore"):
v = (a * b).sum(axis=1) / np.sqrt((a**2).sum(axis=1)
* (b**2).sum(axis=1))
v = v.dropna()
return v.mean(), v.mean() / v.std() * np.sqrt(len(v))
tabla = pd.DataFrame({k: {"IC": ic_semanal(v)[0],
"t": ic_semanal(v)[1],
"cobertura": (v != 0).mean().mean()}
for k, v in SEN.items()}).T
tabla.round(3) IC t cobertura
fundamental 0.016 3.352 1.000
tecnico 0.010 1.552 0.992
noticias 0.013 2.771 0.355
sentimiento 0.018 3.772 0.520The bullish and bearish word lists (ALC, BAJ) mix Spanish and English, like the generator's posts: “cargando”,
“rompe”, “aguantando” are “loading”, “breaking out”, “holding”, and “vendido”, “cuidado”, “castigo” are “sold”,
“careful”, “punishment”. The four signals have the sign of their planted effect and a small weekly IC, as befits data
with a realistic signal-to-noise ratio: the best, social media sentiment, is around 0.018, and the technical one,
0.010 with a t that does not reach 2 because the generator's momentum suffers crashes in rebounds. The news and social
media signals only cover the weeks with posts (35% and 52% of the company-weeks); the rest stays at zero, “no data”. No
agent, however well it writes, can pass on more information than its signal carries: these numbers are each analyst's
ceiling.
Simulated agents: read with noise, debate, decide
Now the architecture. To run it thousands of times without an LLM, each agent is simulated with a rule that imitates what an LLM does with a report:
- Analyst: reads its signal with a reading error (normal, standard deviation 0.6) and turns it into an integer rating from −2 (strong sell) to +2 (strong buy), like the “rating” a prompt asks for. With no data, rating 0.
- Debate: in each round, the bullish researcher adds up the positive ratings and the bearish one the negative ones; from the second round on they reread the reports, with a new error (an LLM does not read a long text the same way twice). The facilitator averages the balance of the rounds: the verdict.
- Trader: buys if the verdict reaches +0.75, sells if it falls below −0.75, and otherwise stays out.
- Risk manager (the conservative stance, which is the one the fund manager approves here): does not trade when the standard deviation of the four ratings reaches 1.25 (high disagreement). The portfolio puts 50% of the capital in the buys and 50% in the sells, in equal parts.
Mermaid source
flowchart TB S[PIT signals] --> N["Ratings -2..+2"] N --> D[Debate rounds] D --> V[Verdict] V --> T[Trader: threshold] T --> G[Risk: disagreement] G --> C[Weekly portfolio]
Y = r_sig.iloc[:-1].to_numpy() # (weeks, companies)
S = {k: v.iloc[:-1].fillna(0).to_numpy() for k, v in SEN.items()}
W, N = Y.shape
def pipeline(semilla, rondas=1, ruido=0.6, umbral=0.75, riesgo=True):
rng = np.random.default_rng(semilla)
def leer(): # each analyst gives its rating
return np.stack([np.where(s != 0, np.clip(np.rint(
s + ruido * rng.standard_normal(s.shape)), -2, 2), 0)
for s in S.values()])
notas = leer()
desacuerdo = notas.std(axis=0)
veredicto = np.zeros((W, N))
for r in range(rondas): # bull / bear debate
nr = notas if r == 0 else leer()
veredicto += (np.clip(nr, 0, None).sum(0)
- np.clip(-nr, 0, None).sum(0)) / 4
veredicto /= rondas
compra, venta = veredicto >= umbral, veredicto <= -umbral
if riesgo: # conservative risk manager
alto = desacuerdo >= 1.25
compra, venta = compra & ~alto, venta & ~alto
nc = np.maximum(compra.sum(1, keepdims=True), 1)
nv = np.maximum(venta.sum(1, keepdims=True), 1)
w = np.where(compra, 0.5 / nc, 0) - np.where(venta, 0.5 / nv, 0)
return w, notas, veredicto, desacuerdo
w, notas, ver, des = pipeline(1)
i, j = FECHAS.get_loc(pd.Timestamp("2024-02-09")), TK.index("SERI")
print("SERI 2024-02-09 | ratings:",
dict(zip(S, notas[:, i, j].astype(int).tolist())),
"| verdict:", ver[i, j], "| disagreement:", round(des[i, j], 2),
"| weight:", round(w[i, j], 4))
print("positions per week:", round((w != 0).sum(1).mean(), 1))SERI 2024-02-09 | ratings: {'fundamental': 0, 'tecnico': 2, 'noticias': -2, 'sentimiento': 0} | verdict: 0.0 | disagreement: 1.41 | weight: 0.0
positions per week: 24.5The record of a decision is the equivalent of the paper's structured reports: four ratings, a verdict, the disagreement and the final weight. For Sertis Water, the week of lesson 6.4.2, the technical analyst gives +2 (the week's fall triggers the reversal), the news analyst −2 (the product recall) and the other two 0: the balance is zero, the disagreement high and the system does not trade. That is what should be saved for each decision of a real system (lesson 6.4.5), with the text of each report next to it.
The honest comparison: baselines and costs
Any system of this kind is compared, at a minimum, with four benchmarks: the market (the 120 stocks in equal parts), each signal alone in a long-short quintile portfolio (24 stocks on each side), the linear combination of the four signals with no agent at all and the same rule without noise: the verdict computed with the signals neither rounded nor read with error, which is what the agents would do if they read perfectly. Costs are 10 basis points per unit of turnover (lesson 2.4.7), a moderate figure for liquid stocks.
COSTE = 0.0010 # 10 bp per unit of turnover
def metricas(w):
bruto = np.nansum(w * Y, axis=1)
rot = np.abs(np.diff(np.vstack([np.zeros(N), w]), axis=0)).sum(1)
neto = bruto - COSTE * rot
sr = lambda x: x.mean() / x.std() * np.sqrt(52)
return {"bruto_%": 5200 * bruto.mean(),
"neto_%": 5200 * neto.mean(),
"sharpe_bruto": sr(bruto), "sharpe_neto": sr(neto),
"rotacion": rot.mean(),
"t_neto": neto.mean() / neto.std() * np.sqrt(len(neto))}
def quintiles(s, k=24):
rk = pd.DataFrame(s).rank(axis=1, method="first").to_numpy()
return np.where(rk > N - k, 0.5 / k, 0) - np.where(rk <= k,
0.5 / k, 0)
def regla_sin_ruido(umbral=0.75):
st = np.stack([np.clip(s, -2, 2) for s in S.values()])
v = (np.clip(st, 0, None).sum(0) - np.clip(-st, 0, None).sum(0)) / 4
des = np.stack([np.where(s != 0, np.clip(np.rint(s), -2, 2), 0)
for s in S.values()]).std(axis=0)
c = (v >= umbral) & (des < 1.25)
d = (v <= -umbral) & (des < 1.25)
nc = np.maximum(c.sum(1, keepdims=True), 1)
nd = np.maximum(d.sum(1, keepdims=True), 1)
return np.where(c, 0.5 / nc, 0) - np.where(d, 0.5 / nd, 0)
filas = {"market (long)": metricas(np.full((W, N), 1 / N))}
for k, s in S.items():
filas[k] = metricas(quintiles(s))
filas["linear (4 signals)"] = metricas(quintiles(sum(S.values())))
filas["noise-free rule"] = metricas(regla_sin_ruido())
filas["multi-agent, 1 round"] = metricas(pipeline(1)[0])
filas["multi-agent, no risk"] = metricas(
pipeline(1, riesgo=False)[0])
pd.DataFrame(filas).T.round(2) bruto_% neto_% sharpe_bruto sharpe_neto rotacion t_neto
market (long) 11.86 11.85 0.58 0.58 0.00 1.55
fundamental 4.56 3.76 1.18 0.97 0.16 2.57
tecnico 3.01 -2.74 0.56 -0.51 1.11 -1.35
noticias 3.70 -3.79 0.93 -0.95 1.44 -2.52
sentimiento 5.81 -1.85 1.57 -0.50 1.47 -1.32
linear (4 signals) 7.51 2.04 1.52 0.41 1.05 1.10
noise-free rule 18.94 10.67 1.90 1.07 1.59 2.84
multi-agent, 1 round 7.89 -0.04 1.24 -0.01 1.52 -0.01
multi-agent, no risk 7.56 -0.17 1.24 -0.03 1.49 -0.07The table has several readings, all of them uncomfortable for enthusiasm:
- Fast signals do not survive costs. Technical, news and sentiment have a positive gross return (3% to 6%) and a negative net one: they turn over between 110% and 147% a week. Only the fundamental signal, which changes slowly (turnover of 16%), keeps a net return: 3.8% a year with a t of 2.57.
- The multi-agent pipeline with one round falls far below the noise-free rule: 7.9% gross and a zero net (−0.04%), against the rule's 18.9% gross and 10.7% net, with a net Sharpe of 1.07. Both turn over the same (≈1.5 a week): the difference is information that the reading noise and the rounding to ratings destroy.
- The risk manager barely changes anything in this data: removing the positions with high disagreement does not improve the result (the section on disagreement explains why).
- Little is conclusive. Only the fundamental signal (t = 2.57) and the noise-free rule (t = 2.84) exceed a net t of 2, and more than ten configurations have been tried, with the threshold and the risk filter chosen by hand: before believing the rule you would have to deflate its Sharpe (lesson 2.5.4) and test it on another period. Here, moreover, we know that the four signals have planted effects: a rule that combines them reasonably has to earn something. What the table shows is something else: the agent pipeline, with the same information, earns less than the rule. The market, doing nothing, had a Sharpe of 0.58.
Rounds, runs and calls
The paper makes the number of debate rounds configurable. The cell measures what they contribute: for 1, 2, 3, 5 and 10 rounds, ten runs with different seeds (ten “draws” of the same system) and the noise-free rule as the limit. It also counts the LLM calls the real architecture would make with one call per agent and turn: four analysts, two per debate round, the facilitator, the trader, three risk agents with one round and the fund manager.
decisiones = W * N
res = []
for rondas in (1, 2, 3, 5, 10):
sr = [metricas(pipeline(s, rondas)[0])["sharpe_neto"]
for s in range(1, 11)]
llamadas = 4 + 2 * rondas + 1 + 1 + 3 + 1
res.append({"rondas": rondas, "sharpe_neto_medio": np.mean(sr),
"min": min(sr), "max": max(sr),
"llamadas_decision": llamadas,
"llamadas_total_miles": decisiones * llamadas / 1e3})
print("noise-free rule: net Sharpe",
round(metricas(regla_sin_ruido())["sharpe_neto"], 2))
signos = [np.sign(pipeline(s, 1)[0]) for s in range(1, 6)]
print("decisions that change between runs (1 round):",
round(np.mean([np.mean(signos[0] != x) for x in signos[1:]]), 3))
pd.DataFrame(res).set_index("rondas").round(2)noise-free rule: net Sharpe 1.07
decisions that change between runs (1 round): 0.199
sharpe_neto_medio min max llamadas_decision llamadas_total_miles
rondas
1 0.16 -0.22 0.38 12 525.6
2 0.49 0.25 0.77 14 613.2
3 0.53 0.10 0.83 16 700.8
5 0.68 0.44 1.03 20 876.0
10 0.65 0.45 0.88 30 1314.0The header figure draws it. With one round, the average net Sharpe of ten runs is 0.16 and goes from −0.22 to 0.38: the same system, with the same data, can come out a loser or modest depending on the draw, and 20% of its decisions change from one run to another. Each additional round rereads the evidence and the average rises toward the noise-free rule (1.07): 0.49 with two rounds, 0.68 with five and 0.65 with ten (the difference between five and ten is seed noise). Not even with ten rounds does it get there: rounding to integer ratings keeps losing information even when averaged. The debate discovers nothing: it averages noisy readings of the same information, like an ensemble (lesson 5.10.3), and its limit is a rule that needs no LLM at all.
The price grows in a straight line: 12 calls per decision with one round, 30 with ten, and between 525,600 and 1.3 million calls for a weekly backtest of 120 stocks over seven years. It is the reason the paper gives for limiting its evaluation to three months, and the reason a long, repeated backtest of an architecture like this is almost never published.
Does disagreement warn of mistakes?
An attractive idea: if the analysts do not agree, the decision is riskier and the risk manager should step on the brakes. The cell checks it with the pipeline without the risk manager: for each open position, whether the sign matches that of the following week's return minus the average of the 120 (a hit), grouped by the standard deviation of the four ratings.
w, notas, ver, des = pipeline(1, riesgo=False)
exceso = Y - np.nanmean(Y, axis=1, keepdims=True)
abierta = w != 0
acierto = (np.sign(w) == np.sign(exceso))[abierta]
nivel = pd.cut(des[abierta], [-0.01, 0.6, 1.1, 9],
labels=["low", "medium", "high"])
pd.Series(acierto).groupby(nivel, observed=True).agg(
acierto="mean", posiciones="size").round(3) acierto posiciones
low 0.505 1001
medium 0.517 7787
high 0.520 865The hit rate is between 50.5% and 52% at the three levels, and is even slightly higher with high disagreement: disagreement warns of nothing. It makes sense with this data: the four signals come from independent planted effects, and the news signal saying buy while the technical one says sell is normal, not a symptom of error. In real data the signals are more correlated and disagreement could mean something else, but the principle holds: a risk rule based on disagreement is measured like any other, with its cost and its effect, before putting it into production (lesson 4.2).
What to take from these architectures
None of the above makes the idea of splitting the work useless. What the lesson separates is where an LLM adds value and where it does not:
| Piece | An LLM adds value | Better without an LLM |
|---|---|---|
| Extracting data from text (guidance, KPIs, tone) | Yes, validated against a ground truth (lesson 6.2.3) | — |
| Summarizing the evidence for a person | Yes, with checked citations (lesson 6.4.2) | — |
| Combining numerical signals | — | A rule or a statistical model, measured out of sample |
| Sizing positions and risk limits | — | A deterministic risk engine (lesson 6.4.5) |
| Deciding whether to trade | — | The rule, with human approval for changes |
The written reports, the debate and the justification in natural language have value for explaining a decision to a person, which is what the paper highlights as an advantage over deep learning models. The decision itself is better taken by something reproducible.
Common mistakes
- Believing a three-month Sharpe. With 60 sessions, the standard error of the annualized Sharpe exceeds 2 points.
- Forgetting costs. The news and social media signals turn over more than 100% a week; without costs they look good.
- A single run. The same pipeline gives net Sharpes from −0.22 to 0.38 depending on the seed.
- Comparing only with single-stock technical rules. The fair benchmark is the same information combined without an LLM.
- Ignoring the model's memory. A backtest on dates before the model's cutoff measures what it remembers (lesson 6.2.5).
- Paying for debate rounds to get an average. If the debate converges to a rule, the rule is cheaper, faster and does not change between runs.
Exercises
- Repeat the comparison with 0, 10 and 20 basis points per unit of turnover for the linear combination, the noise-free rule and the one-round pipeline. Above what cost does each one stop making sense?
Solution
filas = {}
for c in (0.0, 0.0010, 0.0020):
COSTE = c
filas[f"{c * 1e4:.0f} bp"] = {
"linear": metricas(quintiles(sum(S.values())))["neto_%"],
"rule": metricas(regla_sin_ruido())["neto_%"],
"multi-agent": metricas(pipeline(1)[0])["neto_%"]}
COSTE = 0.0010
pd.DataFrame(filas).round(2) 0 bp 10 bp 20 bp
linear 7.51 2.04 -3.42
rule 18.94 10.67 2.40
multi-agent 7.89 -0.04 -7.96All three turn over between 100% and 160% a week, so every 10 basis points of cost takes away between 5% and 8% a year. Without costs they all earn; with 10 basis points the one-round pipeline ends up at zero, the linear combination keeps 2% and the noise-free rule 10.7%; with 20, only the rule stays positive (2.4%), and the pipeline loses almost 8%. An architecture that decides every week on 120 stocks needs, before agents, a turnover rule (entry and exit thresholds, less frequent rebalancing).
- Remove the sentiment analyst (the one with the highest IC) from the pipeline and from the noise-free rule. What happens to the gross and the net return? Why?
Solution
S_todo = S
S = {k: v for k, v in S_todo.items() if k != "sentimiento"}
r = {"noise-free rule": metricas(regla_sin_ruido()),
"multi-agent, 1 round": metricas(pipeline(1)[0])}
S = S_todo
pd.DataFrame(r).T[["bruto_%", "neto_%", "rotacion"]].round(2) bruto_% neto_% rotacion
noise-free rule 13.74 5.90 1.51
multi-agent, 1 round 7.39 -0.34 1.49Without sentiment, both versions lose gross return (the rule goes from 18.9% to 13.7% and the pipeline from 7.9% to 7.4%): it was the signal with the most IC. Turnover barely falls (from 1.59 to 1.51 and from 1.52 to 1.49), because the other fast signals are still there, so the net also drops: from 10.7% to 5.9% for the rule and from 0.0% to −0.3% for the pipeline. Here the sentiment signal paid for its cost within the combination, even though on its own it lost money: the useful question with each analyst is how much it contributes to the whole net of what it costs, not how much IC it has or how much it earns separately.
- No code: a vendor offers you a multi-agent system “in the style of TradingAgents” with “a Sharpe of 3.1 in a backtest from 2023 to 2025 on the S&P 500” and shows you an example of its debate among agents, very convincing. What would you ask for before continuing the conversation?
Solution
The number of backtest runs and the dispersion of the result among them (with a single one, the Sharpe is a draw); the assumed costs and the turnover; which models it uses and their cutoff date, because a backtest from 2023 to 2025 with models trained on texts from those years measures memory (lesson 6.2.5); how many configurations were tried before choosing this one (Deflated Sharpe, lesson 2.5.4); the comparison with the same information combined without an LLM; the number of calls per decision and their cost; and a period after the cutoff of all the models, live or in paper trading, as AlphaCrafter does. A convincing debate is not proof: it is text generated by the same system being evaluated.
Each answer is folded under its question
Self-assessment
The TradingAgents paper reports, with data from January to March 2024, a cumulative return of 26.62% and a Sharpe of 8.21 on AAPL. Which reading is correct?
- AThat the multi-agent architecture reliably beats the market
- BThat with a Sharpe of 8 the strategy is risk-free
- CThat it is a three-month result, on three stocks shown, with no declared transaction costs, using models that may have read texts from that period; the authors themselves say the Sharpe exceeds the range they expected
- DThat the chosen baselines (MACD, SMA…) were professional benchmark strategies
Show answer
Answer C. That it is a three-month result, on three stocks shown, with no declared transaction costs, using models that may have read texts from that period; the authors themselves say the Sharpe exceeds the range they expected
The paper's footnote explains that they evaluated only three months because of the cost of 11 LLM calls and more than 20 tool calls per prediction, and that the highest Sharpe “exceeds our expected empirical range”. A Sharpe over 60 sessions has a huge standard error, and without costs or control of look-ahead bias there is no basis for generalizing.
In the lesson, the multi-agent pipeline with a single debate round gives an average net Sharpe of 0.16 over ten runs, with 5 or 10 rounds it rises to 0.65-0.68 and the equivalent deterministic rule, with no LLM at all, gives 1.07. What explains the pattern?
- AThat the debate generates new information in each round
- BThat each round rereads the same evidence with new noise: averaging readings reduces the noise and the verdict converges to a fixed function of the signals, which can be written without an LLM
- CThat with more rounds the model learns from its mistakes
- DThat costs go down with the number of rounds
Show answer
Answer B. That each round rereads the same evidence with new noise: averaging readings reduces the noise and the verdict converges to a fixed function of the signals, which can be written without an LLM
The simulated agents do not invent signal: they read the four signals with an error and round them to a rating. Repeating the reading and averaging is an ensemble that approaches the expected value of the ratings. The limit is a deterministic rule, which costs zero calls and does not vary between runs.
Between two runs of the pipeline with one round, 20% of the decisions (buy, sell or stay out) change. What does that imply for evaluating an agent of this kind?
- AThat a single run of the backtest is not enough: it has to be repeated with several seeds or samples and the distribution of the result reported
- BThat setting the temperature to zero is enough to make it stop happening
- CThat 20% of the decisions are mistakes
- DNothing: the average Sharpe is the only thing that matters
Show answer
Answer A. That a single run of the backtest is not enough: it has to be repeated with several seeds or samples and the distribution of the result reported
With one round, the net Sharpe of ten runs goes from −0.22 to 0.38: the result of one run is a sample from a wide distribution. Lesson 6.4.4 shows that with real LLMs, even with deterministic decoding, the actions also change between runs.
The social media signal has the highest weekly IC of the four (0.018, t = 3.8), but its long-short portfolio loses 1.9% a year net. Why?
- ABecause the IC is computed wrong
- BBecause social media have no effect in this data
- CBecause the portfolio is long only
- DBecause the signal changes almost completely every week: it turns over 147% a week and 10 basis points per unit of turnover eat up the whole gross return of 5.8%
Show answer
Answer D. Because the signal changes almost completely every week: it turns over 147% a week and 10 basis points per unit of turnover eat up the whole gross return of 5.8%
The generator's retail sentiment is persistent for only a few days (daily AR(1) with ρ = 0.7) and the quintile portfolios are rebuilt almost entirely every week. A signal with a high IC and a short life is worth little if it is traded with costs (lesson 2.4.7).
Does disagreement among the four analysts predict that the decision will fail?
- AYes: with high disagreement the hit rate drops to 30%
- BNot in this data: the hit rate is between 50.5% and 52% at the three levels of disagreement, because the four sources measure independent effects and disagreeing is normal
- CYes, and that is why the risk manager should trade only when there is disagreement
- DIt cannot be measured without a real LLM
Show answer
Answer B. Not in this data: the hit rate is between 50.5% and 52% at the three levels of disagreement, because the four sources measure independent effects and disagreeing is normal
The four signals come from different planted effects (value and surprise, momentum and reversal, news drift, retail sentiment). One saying buy and another saying sell does not mean either is wrong. Filtering by disagreement, as the lesson's risk manager does, reduces positions but does not improve the hit rate.
Key points
- TradingAgents organizes a simulated desk: four analysts (fundamental, sentiment, news, technical), bull and bear researchers who debate for n rounds, a trader, a risk team with three profiles and a manager who approves. Its original evaluation covers three months of 2024 with no declared costs.
- An analyst agent can only pass on the signal that is in its data: in the course, the four signals have weekly ICs of 0.010 to 0.018 against known planted effects.
- Debate among agents that read with noise works as an average: more rounds bring the verdict closer to a deterministic rule that can be written without an LLM, while calls grow by 2 per round and company.
- Decisions change between runs (20% with one round) and the net Sharpe of ten runs goes from −0.22 to 0.38: a single-run backtest does not describe the system.
- With 10 basis points per unit of turnover, fast signals lose money; the fundamental signal alone and the rule with no LLM are the only ones with a net t above 2, and after trying more than ten configurations, Deflated Sharpe before believing anything (lesson 2.5.4).
Resources
Books
- Xiao, Y., Sun, E., Luo, D. and Wang, W.: TradingAgents: Multi-Agents LLM Financial Trading Framework (arXiv:2412.20138, v7)
- Yuan, Y. et al. (2026): AlphaCrafter: Harnessing Multi-Agent Workflows for Cross-Sectional Quantitative Trading (arXiv:2605.05580, v2)
- Du, Y. et al. (2023): Improving Factuality and Reasoning in Language Models through Multiagent Debate (arXiv:2305.14325)