Lesson 6.4.3 · 75 min

TradingAgents-style multi-agent architectures

TradingAgents splits the investment decision among analysts, researchers who debate, a trader and a risk team, like a real desk. This lesson reads the paper with its figures and its limits, builds the same architecture on the course data with simulated agents and measures it honestly: against the market, against each signal alone and against the deterministic rule the debate converges to, with costs.

Fig. — Net Sharpe of the simulated multi-agent pipeline by debate rounds, against the rule without an LLM
Level 6 lessons
  1. 6.1 · Classical and modern financial NLP

    1. 6.1.1 Tokenization, embeddings, financial dictionaries and FinBERT
    2. 6.1.2 Text sources: EDGAR, CNMV, transcripts and social media
    3. 6.1.3 Event studies with text signals, timestamp alignment and topic modeling
  2. 6.2 · LLMs for research and trading

    1. 6.2.1 How an LLM works: from the next token to the API
    2. 6.2.2 Structured prompting: JSON you can validate
    3. 6.2.3 Guidance, KPIs and tone: what an LLM extracts well
    4. 6.2.4 Lightweight fine-tuning for financial tasks: LoRA and QLoRA
    5. 6.2.5 Look-ahead bias in LLMs: the model that already knows the future
  3. 6.3 · Financial RAG

    1. 6.3.1 Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant
    2. 6.3.2 Chunking financial documents: tables, footnotes and XBRL
    3. 6.3.3 RAG over reports, prospectuses and KIDs, and how to evaluate it
  4. 6.4 · AI agents for trading

    1. 6.4.1 Tool-calling agents: tool calling, ReAct and planning
    2. 6.4.2 Agent frameworks and MCP
    3. 6.4.3 TradingAgents-style multi-agent architectures
    4. 6.4.4 Evidence and limits of agents
    5. 6.4.5 Guardrails for agents: the agent proposes, the engine decides
    6. 6.4.6 Research agents: hypotheses, alpha mining and review
  5. 6.5 · AI as the quant's copilot

    1. 6.5.1 Coding assistants with a test-first workflow
    2. 6.5.2 Automated review of backtests and code migrations
    3. 6.5.3 Risks of AI-generated code
  6. 6.6 · Big data for trading

    1. 6.6.1 When data becomes “big”: ticks, order books, options and universes
    2. 6.6.2 Parquet, Arrow, Polars streaming and out-of-core DuckDB
    3. 6.6.3 Dask, Spark and Ray: from one machine to a cluster
    4. 6.6.4 Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+
    5. 6.6.5 Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF)
  7. 6.7 · Alternative data

    1. 6.7.1 Overview and size of the alternative data market
    2. 6.7.2 Types of alternative data: what each one measures
    3. 6.7.3 Responsible web scraping
    4. 6.7.4 Alternative data vendors and due diligence
    5. 6.7.5 From alternative data to a trading signal
    6. 6.7.6 Satellite imagery, computer vision and supply chains

An investment desk made of agents

Lesson 6.4.2 built a workflow with a single writer. The idea of multi-agent architectures is to split the work among agents with different roles, as on an investment desk: some gather data and write reports, others argue, someone decides and someone controls the risk. The most cited reference is TradingAgents, by Yijia Xiao, Edward Sun, Di Luo and Wei Wang (UCLA, MIT and Tauric Research), arXiv:2412.20138; what follows comes from the PDF of version 7 (June 3, 2025), read on 2026-10-23. Its organization has five pieces:

  1. Analyst team: four agents work at the same time, each with its own tools: fundamental (financial statements, earnings reports, insider transactions), sentiment (social media posts and sentiment scores), news (news, government announcements and macroeconomic indicators) and technical (indicators such as MACD and RSI, prices and volumes). Each one writes a structured report.
  2. Research team: a bullish researcher and a bearish one read the reports and debate in natural language for n rounds; a facilitator reviews the debate and picks the prevailing perspective.
  3. Trader: turns the analysis into a decision (buy, sell or hold) with its justification.
  4. Risk management team: three agents, one risk-seeking, one neutral and one conservative, discuss for n rounds and adjust the trader's plan to the risk limits.
  5. Fund manager: reviews that discussion, decides the adjustments and approves the trade.
FundamentalBull vs bearSentimentNewsTechnicalFacilitatorTraderRisk teamFund manager
Mermaid source
flowchart TB
  A[Fundamental] --> R[Bull vs bear]
  B[Sentiment] --> R
  C[News] --> R
  D[Technical] --> R
  R --> F[Facilitator]
  F --> T[Trader]
  T --> G[Risk team]
  G --> M[Fund manager]

Two design decisions in the paper deserve attention. The agents communicate mainly through structured documents kept in a global state, not through long conversations, so as not to lose information in a game of “telephone”; only the debates use dialogue. And they split the models: quick models (gpt-4o-mini, gpt-4o) to summarize and retrieve data, and reasoning models (o1-preview) to analyze and decide. The public code (repository TauricResearch/TradingAgents, Apache-2.0 license) is built with LangGraph (lesson 6.4.2), lets you configure the number of debate and risk rounds and warns in its README that it is designed for research and that performance may vary with the model, the temperature, the period, the data “and other non-deterministic factors”.

What the paper measures, and what it does not

The original evaluation is a backtest from January 1 to March 29, 2024 on large tech stocks, with data on prices, news, social media, insider transactions, financial statements and 60 technical indicators. The comparisons are with five rules: buy and hold, MACD, KDJ with RSI, ZMR and SMA. Table 1 shows three stocks:

StockTradingAgents: cum. returnSharpeBuy and hold: cum. return
AAPL26.62%8.21−5.23%
GOOGL24.36%6.397.78%
AMZN23.21%5.6017.1%

Before reading those figures as an edge, you have to set next to them what the paper itself says and what it does not say:

  • Three months. A footnote explains that the backtest lasts three months “due to intensive LLM and tool use (11 LLM calls & 20+ tool calls/prediction)”, and that the highest Sharpe “exceeds our expected empirical range”; the authors attribute it to there having been few pullbacks in that period. An annualized Sharpe measured over some 60 sessions has a standard error of more than 2 points.
  • No declared costs. The text does not mention commissions, bid-ask spreads or slippage.
  • Possible look-ahead bias. The models used were trained on Internet text; the paper states that the agents only see data up to each day, but does not discuss whether the models knew from their training what happened to those stocks (lesson 6.2.5).
  • Weak baselines. Single-stock technical rules, not portfolios or quantitative models.

There is a later independent evaluation, with every caveat because its authors propose a competing method: AlphaCrafter (Yuan et al., arXiv:2605.05580, version 2 of July 2026) reimplements TradingAgents on CSI 300 and S&P 500 portfolios with a backtest from January 2024 to February 2026 and a live phase from March to June 2026 with a paper trading API. With Claude Opus 4.6 as the model, TradingAgents goes on the CSI 300 from a Sharpe of 1.37 in the backtest to 0.67 live, and on the S&P 500 from 1.08 to 1.25; the authors describe the “role-playing” agents as unstable depending on the model used. Far, in any case, from a Sharpe of 8.

The four analysts on the course data

To measure an architecture like this, you have to know what signal there is in the data. The level 6 synthetic data make that possible: the 120 companies have planted effects with parameters published in the README. The lesson builds the four analysts with point-in-time data and one decision a week, at the close of the last day of each week, from January 2019 to December 2025. The sample news and social media corpus is in Spanish, like the rest of level 6, and the code reads it as is:

AnalystDataSignalPlanted effect (README)
Fundamentalpanel5_features: book value / price and SUEg(valor_bp) + g(sue)·e^(−days/45)Value and earnings surprise (level 5)
Technicalpanel5_features: 12-1 momentum and the week's returng(mom_12_1) − g(ret_1s)Momentum (with crashes in rebounds) and weekly reversal
NewsNews without copies up to 16:00 NYSum of the week's dictionary toneReaction and 5-session drift
SentimentCashtag posts from accounts older than 60 daysCount of bullish and bearish words10 bp the next day per unit of opinion, daily AR(1) 0.7

g(·) is the cross-sectional normal rank of each week. Each post is attributed to the company of its cashtag (with the ticker in force on its date, which the ticker table translates to the current one). Posts from accounts 60 days old or younger are excluded because the generator's automated accounts are recent and have few followers: filtering by age is a heuristic, not the truth of the README (41% of the excluded accounts are bots; of those that remain, 16%).

IN [01]python
import re
import numpy as np
import pandas as pd
from scipy import stats

emp = pd.read_csv("acciones6_empresas.csv")
hist = pd.read_csv("acciones6_tickers.csv")
ID = dict(zip(hist["ticker"], hist["id_empresa"]))
ACTUAL = dict(zip(emp["id_empresa"], emp["ticker"]))
TK = sorted(emp["ticker"])
px = pd.read_parquet("acciones6_precios.parquet")
cierres = px.pivot(index="fecha", columns="ticker",
                   values="cierre")[TK]
f5 = pd.read_parquet("panel5_features.parquet", columns=[
    "fecha", "ticker", "mom_12_1", "ret_1s", "valor_bp", "sue",
    "dias_desde_informe"])
f5 = f5[f5["ticker"].isin(TK) & (f5["fecha"] >= "2019-01-01")]
FECHAS = pd.DatetimeIndex(sorted(f5["fecha"].unique()))
sem = cierres.reindex(FECHAS)
r_sig = sem.shift(-1) / sem - 1          # following week
CORTES = FECHAS + pd.Timedelta(hours=16)   # New York close

def g(df):
    """Cross-sectional normal rank of each date (rows)."""
    rk = df.rank(axis=1).sub(0.5)
    return pd.DataFrame(stats.norm.ppf(rk.div(df.count(axis=1),
                                              axis=0)),
                        index=df.index, columns=df.columns)

def semana_de(ts):
    """First decision date whose close is later than `ts`."""
    ny = ts.dt.tz_convert("America/New_York").dt.tz_localize(None)
    i = np.searchsorted(CORTES, ny, side="left")
    return pd.Series(FECHAS[i.clip(0, len(FECHAS) - 1)],
                     index=ts.index).where(i < len(FECHAS))

print(len(FECHAS), "weeks,", FECHAS[0].date(), "to",
      FECHAS[-1].date(), "|", len(TK), "companies")
OUT [01]Weekly data: decision at the close and the following week
366 weeks, 2019-01-04 to 2025-12-31 | 120 companies
IN [02]python
piv = lambda c: f5.pivot(index="fecha", columns="ticker",
                         values=c)[TK]
fun = g(piv("valor_bp")) + g(piv("sue")) * np.exp(
    -piv("dias_desde_informe") / 45)
tec = g(piv("mom_12_1")) - g(piv("ret_1s"))

lex = pd.read_csv("texto6_lexico.csv").query("lista == 'financiera'")
POS = set(lex.query("categoria == 'positiva'")["palabra"])
NEG = set(lex.query("categoria == 'negativa'")["palabra"])
def tono(t):
    pal = re.findall(r"\w+", t.lower())
    p, n = sum(w in POS for w in pal), sum(w in NEG for w in pal)
    return (p - n) / (p + n + 1)
nt = pd.read_parquet("noticias6_corpus.parquet")
PREF = r"^(?:ACTUALIZA|Última hora|UPDATE|Breaking):\s*"
nt["clave"] = nt["titular"].str.replace(PREF, "", regex=True,
                                        case=False)
nt = nt.sort_values("ts").drop_duplicates(["ticker", "clave"])
nt["tono"] = (nt["titular"] + " " + nt["cuerpo"]).map(tono)
nt["semana"] = semana_de(nt["ts"])
nt["tk"] = nt["ticker"].map(ID).map(ACTUAL)
noti = g(nt.groupby(["semana", "tk"])["tono"].sum().unstack()
         .reindex(index=FECHAS, columns=TK)).fillna(0.0)

so = pd.read_parquet("social6_posts.parquet")
so["tk"] = (so["texto"].str.extract(r"\$([A-Z]{4})")[0]
            .map(ID).map(ACTUAL))
so = so[so["antiguedad_cuenta_dias"] > 60]   # drop new accounts
ALC = ["bien", "cargando", "rompe", "gusta", "compra", "aguantando",
       "sorpresa", "buenos", "strong", "loading", "breaking out",
       "holding", "buy", "higher"]
BAJ = ["pique", "vendido", "cuidado", "pierde", "castigo", "levanta",
       "weak", "shorting", "sold", "dead cat", "breaking down",
       "avoid", "bleeding"]
tl = so["texto"].str.lower()
cuenta = lambda ws: sum(tl.str.contains(w, regex=False).astype(int)
                        for w in ws)
so["s"] = (cuenta(ALC) - cuenta(BAJ)).clip(-1, 1)
so["semana"] = semana_de(so["ts"])
soc = g(so.groupby(["semana", "tk"])["s"].mean().unstack()
        .reindex(index=FECHAS, columns=TK)).fillna(0.0)

SEN = {"fundamental": fun, "tecnico": tec, "noticias": noti,
       "sentimiento": soc}
def ic_semanal(sig):
    a = sig.iloc[:-1].rank(axis=1)
    b = r_sig.iloc[:-1].rank(axis=1)
    a, b = a.sub(a.mean(axis=1), axis=0), b.sub(b.mean(axis=1), axis=0)
    with np.errstate(invalid="ignore", divide="ignore"):
        v = (a * b).sum(axis=1) / np.sqrt((a**2).sum(axis=1)
                                          * (b**2).sum(axis=1))
    v = v.dropna()
    return v.mean(), v.mean() / v.std() * np.sqrt(len(v))
tabla = pd.DataFrame({k: {"IC": ic_semanal(v)[0],
                          "t": ic_semanal(v)[1],
                          "cobertura": (v != 0).mean().mean()}
                      for k, v in SEN.items()}).T
tabla.round(3)
OUT [02]The four analysts and their IC against the following week
                IC      t  cobertura
fundamental  0.016  3.352      1.000
tecnico      0.010  1.552      0.992
noticias     0.013  2.771      0.355
sentimiento  0.018  3.772      0.520

The bullish and bearish word lists (ALC, BAJ) mix Spanish and English, like the generator's posts: “cargando”, “rompe”, “aguantando” are “loading”, “breaking out”, “holding”, and “vendido”, “cuidado”, “castigo” are “sold”, “careful”, “punishment”. The four signals have the sign of their planted effect and a small weekly IC, as befits data with a realistic signal-to-noise ratio: the best, social media sentiment, is around 0.018, and the technical one, 0.010 with a t that does not reach 2 because the generator's momentum suffers crashes in rebounds. The news and social media signals only cover the weeks with posts (35% and 52% of the company-weeks); the rest stays at zero, “no data”. No agent, however well it writes, can pass on more information than its signal carries: these numbers are each analyst's ceiling.

Simulated agents: read with noise, debate, decide

Now the architecture. To run it thousands of times without an LLM, each agent is simulated with a rule that imitates what an LLM does with a report:

  • Analyst: reads its signal with a reading error (normal, standard deviation 0.6) and turns it into an integer rating from −2 (strong sell) to +2 (strong buy), like the “rating” a prompt asks for. With no data, rating 0.
  • Debate: in each round, the bullish researcher adds up the positive ratings and the bearish one the negative ones; from the second round on they reread the reports, with a new error (an LLM does not read a long text the same way twice). The facilitator averages the balance of the rounds: the verdict.
  • Trader: buys if the verdict reaches +0.75, sells if it falls below −0.75, and otherwise stays out.
  • Risk manager (the conservative stance, which is the one the fund manager approves here): does not trade when the standard deviation of the four ratings reaches 1.25 (high disagreement). The portfolio puts 50% of the capital in the buys and 50% in the sells, in equal parts.
PIT signalsRatings -2..+2Debate roundsVerdictTrader: thresholdRisk: disagreementWeekly portfolio
Mermaid source
flowchart TB
  S[PIT signals] --> N["Ratings -2..+2"]
  N --> D[Debate rounds]
  D --> V[Verdict]
  V --> T[Trader: threshold]
  T --> G[Risk: disagreement]
  G --> C[Weekly portfolio]
IN [03]python
Y = r_sig.iloc[:-1].to_numpy()                # (weeks, companies)
S = {k: v.iloc[:-1].fillna(0).to_numpy() for k, v in SEN.items()}
W, N = Y.shape

def pipeline(semilla, rondas=1, ruido=0.6, umbral=0.75, riesgo=True):
    rng = np.random.default_rng(semilla)
    def leer():                       # each analyst gives its rating
        return np.stack([np.where(s != 0, np.clip(np.rint(
            s + ruido * rng.standard_normal(s.shape)), -2, 2), 0)
            for s in S.values()])
    notas = leer()
    desacuerdo = notas.std(axis=0)
    veredicto = np.zeros((W, N))
    for r in range(rondas):           # bull / bear debate
        nr = notas if r == 0 else leer()
        veredicto += (np.clip(nr, 0, None).sum(0)
                      - np.clip(-nr, 0, None).sum(0)) / 4
    veredicto /= rondas
    compra, venta = veredicto >= umbral, veredicto <= -umbral
    if riesgo:                        # conservative risk manager
        alto = desacuerdo >= 1.25
        compra, venta = compra & ~alto, venta & ~alto
    nc = np.maximum(compra.sum(1, keepdims=True), 1)
    nv = np.maximum(venta.sum(1, keepdims=True), 1)
    w = np.where(compra, 0.5 / nc, 0) - np.where(venta, 0.5 / nv, 0)
    return w, notas, veredicto, desacuerdo

w, notas, ver, des = pipeline(1)
i, j = FECHAS.get_loc(pd.Timestamp("2024-02-09")), TK.index("SERI")
print("SERI 2024-02-09 | ratings:",
      dict(zip(S, notas[:, i, j].astype(int).tolist())),
      "| verdict:", ver[i, j], "| disagreement:", round(des[i, j], 2),
      "| weight:", round(w[i, j], 4))
print("positions per week:", round((w != 0).sum(1).mean(), 1))
OUT [03]The simulated multi-agent pipeline
SERI 2024-02-09 | ratings: {'fundamental': 0, 'tecnico': 2, 'noticias': -2, 'sentimiento': 0} | verdict: 0.0 | disagreement: 1.41 | weight: 0.0
positions per week: 24.5

The record of a decision is the equivalent of the paper's structured reports: four ratings, a verdict, the disagreement and the final weight. For Sertis Water, the week of lesson 6.4.2, the technical analyst gives +2 (the week's fall triggers the reversal), the news analyst −2 (the product recall) and the other two 0: the balance is zero, the disagreement high and the system does not trade. That is what should be saved for each decision of a real system (lesson 6.4.5), with the text of each report next to it.

The honest comparison: baselines and costs

Any system of this kind is compared, at a minimum, with four benchmarks: the market (the 120 stocks in equal parts), each signal alone in a long-short quintile portfolio (24 stocks on each side), the linear combination of the four signals with no agent at all and the same rule without noise: the verdict computed with the signals neither rounded nor read with error, which is what the agents would do if they read perfectly. Costs are 10 basis points per unit of turnover (lesson 2.4.7), a moderate figure for liquid stocks.

IN [04]python
COSTE = 0.0010                          # 10 bp per unit of turnover

def metricas(w):
    bruto = np.nansum(w * Y, axis=1)
    rot = np.abs(np.diff(np.vstack([np.zeros(N), w]), axis=0)).sum(1)
    neto = bruto - COSTE * rot
    sr = lambda x: x.mean() / x.std() * np.sqrt(52)
    return {"bruto_%": 5200 * bruto.mean(),
            "neto_%": 5200 * neto.mean(),
            "sharpe_bruto": sr(bruto), "sharpe_neto": sr(neto),
            "rotacion": rot.mean(),
            "t_neto": neto.mean() / neto.std() * np.sqrt(len(neto))}

def quintiles(s, k=24):
    rk = pd.DataFrame(s).rank(axis=1, method="first").to_numpy()
    return np.where(rk > N - k, 0.5 / k, 0) - np.where(rk <= k,
                                                        0.5 / k, 0)

def regla_sin_ruido(umbral=0.75):
    st = np.stack([np.clip(s, -2, 2) for s in S.values()])
    v = (np.clip(st, 0, None).sum(0) - np.clip(-st, 0, None).sum(0)) / 4
    des = np.stack([np.where(s != 0, np.clip(np.rint(s), -2, 2), 0)
                    for s in S.values()]).std(axis=0)
    c = (v >= umbral) & (des < 1.25)
    d = (v <= -umbral) & (des < 1.25)
    nc = np.maximum(c.sum(1, keepdims=True), 1)
    nd = np.maximum(d.sum(1, keepdims=True), 1)
    return np.where(c, 0.5 / nc, 0) - np.where(d, 0.5 / nd, 0)

filas = {"market (long)": metricas(np.full((W, N), 1 / N))}
for k, s in S.items():
    filas[k] = metricas(quintiles(s))
filas["linear (4 signals)"] = metricas(quintiles(sum(S.values())))
filas["noise-free rule"] = metricas(regla_sin_ruido())
filas["multi-agent, 1 round"] = metricas(pipeline(1)[0])
filas["multi-agent, no risk"] = metricas(
    pipeline(1, riesgo=False)[0])
pd.DataFrame(filas).T.round(2)
OUT [04]Metrics net of costs against the benchmarks
                      bruto_%  neto_%  sharpe_bruto  sharpe_neto  rotacion  t_neto
market (long)           11.86   11.85          0.58         0.58      0.00    1.55
fundamental              4.56    3.76          1.18         0.97      0.16    2.57
tecnico                  3.01   -2.74          0.56        -0.51      1.11   -1.35
noticias                 3.70   -3.79          0.93        -0.95      1.44   -2.52
sentimiento              5.81   -1.85          1.57        -0.50      1.47   -1.32
linear (4 signals)       7.51    2.04          1.52         0.41      1.05    1.10
noise-free rule         18.94   10.67          1.90         1.07      1.59    2.84
multi-agent, 1 round     7.89   -0.04          1.24        -0.01      1.52   -0.01
multi-agent, no risk     7.56   -0.17          1.24        -0.03      1.49   -0.07
Fig. — Annual return gross and net of costs: signals, combination, rule without LLM and multi-agent pipeline (2019-2025)

The table has several readings, all of them uncomfortable for enthusiasm:

  • Fast signals do not survive costs. Technical, news and sentiment have a positive gross return (3% to 6%) and a negative net one: they turn over between 110% and 147% a week. Only the fundamental signal, which changes slowly (turnover of 16%), keeps a net return: 3.8% a year with a t of 2.57.
  • The multi-agent pipeline with one round falls far below the noise-free rule: 7.9% gross and a zero net (−0.04%), against the rule's 18.9% gross and 10.7% net, with a net Sharpe of 1.07. Both turn over the same (≈1.5 a week): the difference is information that the reading noise and the rounding to ratings destroy.
  • The risk manager barely changes anything in this data: removing the positions with high disagreement does not improve the result (the section on disagreement explains why).
  • Little is conclusive. Only the fundamental signal (t = 2.57) and the noise-free rule (t = 2.84) exceed a net t of 2, and more than ten configurations have been tried, with the threshold and the risk filter chosen by hand: before believing the rule you would have to deflate its Sharpe (lesson 2.5.4) and test it on another period. Here, moreover, we know that the four signals have planted effects: a rule that combines them reasonably has to earn something. What the table shows is something else: the agent pipeline, with the same information, earns less than the rule. The market, doing nothing, had a Sharpe of 0.58.

Rounds, runs and calls

The paper makes the number of debate rounds configurable. The cell measures what they contribute: for 1, 2, 3, 5 and 10 rounds, ten runs with different seeds (ten “draws” of the same system) and the noise-free rule as the limit. It also counts the LLM calls the real architecture would make with one call per agent and turn: four analysts, two per debate round, the facilitator, the trader, three risk agents with one round and the fund manager.

IN [05]python
decisiones = W * N
res = []
for rondas in (1, 2, 3, 5, 10):
    sr = [metricas(pipeline(s, rondas)[0])["sharpe_neto"]
          for s in range(1, 11)]
    llamadas = 4 + 2 * rondas + 1 + 1 + 3 + 1
    res.append({"rondas": rondas, "sharpe_neto_medio": np.mean(sr),
                "min": min(sr), "max": max(sr),
                "llamadas_decision": llamadas,
                "llamadas_total_miles": decisiones * llamadas / 1e3})
print("noise-free rule: net Sharpe",
      round(metricas(regla_sin_ruido())["sharpe_neto"], 2))
signos = [np.sign(pipeline(s, 1)[0]) for s in range(1, 6)]
print("decisions that change between runs (1 round):",
      round(np.mean([np.mean(signos[0] != x) for x in signos[1:]]), 3))
pd.DataFrame(res).set_index("rondas").round(2)
OUT [05]Debate rounds, variability between runs and cost
noise-free rule: net Sharpe 1.07
decisions that change between runs (1 round): 0.199
        sharpe_neto_medio   min   max  llamadas_decision  llamadas_total_miles
rondas
1                    0.16 -0.22  0.38                 12                 525.6
2                    0.49  0.25  0.77                 14                 613.2
3                    0.53  0.10  0.83                 16                 700.8
5                    0.68  0.44  1.03                 20                 876.0
10                   0.65  0.45  0.88                 30                1314.0

The header figure draws it. With one round, the average net Sharpe of ten runs is 0.16 and goes from −0.22 to 0.38: the same system, with the same data, can come out a loser or modest depending on the draw, and 20% of its decisions change from one run to another. Each additional round rereads the evidence and the average rises toward the noise-free rule (1.07): 0.49 with two rounds, 0.68 with five and 0.65 with ten (the difference between five and ten is seed noise). Not even with ten rounds does it get there: rounding to integer ratings keeps losing information even when averaged. The debate discovers nothing: it averages noisy readings of the same information, like an ensemble (lesson 5.10.3), and its limit is a rule that needs no LLM at all.

The price grows in a straight line: 12 calls per decision with one round, 30 with ten, and between 525,600 and 1.3 million calls for a weekly backtest of 120 stocks over seven years. It is the reason the paper gives for limiting its evaluation to three months, and the reason a long, repeated backtest of an architecture like this is almost never published.

Does disagreement warn of mistakes?

An attractive idea: if the analysts do not agree, the decision is riskier and the risk manager should step on the brakes. The cell checks it with the pipeline without the risk manager: for each open position, whether the sign matches that of the following week's return minus the average of the 120 (a hit), grouped by the standard deviation of the four ratings.

IN [06]python
w, notas, ver, des = pipeline(1, riesgo=False)
exceso = Y - np.nanmean(Y, axis=1, keepdims=True)
abierta = w != 0
acierto = (np.sign(w) == np.sign(exceso))[abierta]
nivel = pd.cut(des[abierta], [-0.01, 0.6, 1.1, 9],
               labels=["low", "medium", "high"])
pd.Series(acierto).groupby(nivel, observed=True).agg(
    acierto="mean", posiciones="size").round(3)
OUT [06]Hit rate by disagreement among analysts
        acierto  posiciones
low       0.505        1001
medium    0.517        7787
high      0.520         865

The hit rate is between 50.5% and 52% at the three levels, and is even slightly higher with high disagreement: disagreement warns of nothing. It makes sense with this data: the four signals come from independent planted effects, and the news signal saying buy while the technical one says sell is normal, not a symptom of error. In real data the signals are more correlated and disagreement could mean something else, but the principle holds: a risk rule based on disagreement is measured like any other, with its cost and its effect, before putting it into production (lesson 4.2).

What to take from these architectures

None of the above makes the idea of splitting the work useless. What the lesson separates is where an LLM adds value and where it does not:

PieceAn LLM adds valueBetter without an LLM
Extracting data from text (guidance, KPIs, tone)Yes, validated against a ground truth (lesson 6.2.3)—
Summarizing the evidence for a personYes, with checked citations (lesson 6.4.2)—
Combining numerical signals—A rule or a statistical model, measured out of sample
Sizing positions and risk limits—A deterministic risk engine (lesson 6.4.5)
Deciding whether to trade—The rule, with human approval for changes

The written reports, the debate and the justification in natural language have value for explaining a decision to a person, which is what the paper highlights as an advantage over deep learning models. The decision itself is better taken by something reproducible.

Common mistakes

  • Believing a three-month Sharpe. With 60 sessions, the standard error of the annualized Sharpe exceeds 2 points.
  • Forgetting costs. The news and social media signals turn over more than 100% a week; without costs they look good.
  • A single run. The same pipeline gives net Sharpes from −0.22 to 0.38 depending on the seed.
  • Comparing only with single-stock technical rules. The fair benchmark is the same information combined without an LLM.
  • Ignoring the model's memory. A backtest on dates before the model's cutoff measures what it remembers (lesson 6.2.5).
  • Paying for debate rounds to get an average. If the debate converges to a rule, the rule is cheaper, faster and does not change between runs.

Exercises

  1. Repeat the comparison with 0, 10 and 20 basis points per unit of turnover for the linear combination, the noise-free rule and the one-round pipeline. Above what cost does each one stop making sense?
Solution
IN [07]python
filas = {}
for c in (0.0, 0.0010, 0.0020):
    COSTE = c
    filas[f"{c * 1e4:.0f} bp"] = {
        "linear": metricas(quintiles(sum(S.values())))["neto_%"],
        "rule": metricas(regla_sin_ruido())["neto_%"],
        "multi-agent": metricas(pipeline(1)[0])["neto_%"]}
COSTE = 0.0010
pd.DataFrame(filas).round(2)
OUT [07]Sensitivity to costs
              0 bp  10 bp  20 bp
linear        7.51   2.04  -3.42
rule         18.94  10.67   2.40
multi-agent   7.89  -0.04  -7.96

All three turn over between 100% and 160% a week, so every 10 basis points of cost takes away between 5% and 8% a year. Without costs they all earn; with 10 basis points the one-round pipeline ends up at zero, the linear combination keeps 2% and the noise-free rule 10.7%; with 20, only the rule stays positive (2.4%), and the pipeline loses almost 8%. An architecture that decides every week on 120 stocks needs, before agents, a turnover rule (entry and exit thresholds, less frequent rebalancing).

  1. Remove the sentiment analyst (the one with the highest IC) from the pipeline and from the noise-free rule. What happens to the gross and the net return? Why?
Solution
IN [08]python
S_todo = S
S = {k: v for k, v in S_todo.items() if k != "sentimiento"}
r = {"noise-free rule": metricas(regla_sin_ruido()),
     "multi-agent, 1 round": metricas(pipeline(1)[0])}
S = S_todo
pd.DataFrame(r).T[["bruto_%", "neto_%", "rotacion"]].round(2)
OUT [08]Without the sentiment analyst
                      bruto_%  neto_%  rotacion
noise-free rule         13.74    5.90      1.51
multi-agent, 1 round     7.39   -0.34      1.49

Without sentiment, both versions lose gross return (the rule goes from 18.9% to 13.7% and the pipeline from 7.9% to 7.4%): it was the signal with the most IC. Turnover barely falls (from 1.59 to 1.51 and from 1.52 to 1.49), because the other fast signals are still there, so the net also drops: from 10.7% to 5.9% for the rule and from 0.0% to −0.3% for the pipeline. Here the sentiment signal paid for its cost within the combination, even though on its own it lost money: the useful question with each analyst is how much it contributes to the whole net of what it costs, not how much IC it has or how much it earns separately.

  1. No code: a vendor offers you a multi-agent system “in the style of TradingAgents” with “a Sharpe of 3.1 in a backtest from 2023 to 2025 on the S&P 500” and shows you an example of its debate among agents, very convincing. What would you ask for before continuing the conversation?
Solution

The number of backtest runs and the dispersion of the result among them (with a single one, the Sharpe is a draw); the assumed costs and the turnover; which models it uses and their cutoff date, because a backtest from 2023 to 2025 with models trained on texts from those years measures memory (lesson 6.2.5); how many configurations were tried before choosing this one (Deflated Sharpe, lesson 2.5.4); the comparison with the same information combined without an LLM; the number of calls per decision and their cost; and a period after the cutoff of all the models, live or in paper trading, as AlphaCrafter does. A convincing debate is not proof: it is text generated by the same system being evaluated.

Each answer is folded under its question

Self-assessment

  1. The TradingAgents paper reports, with data from January to March 2024, a cumulative return of 26.62% and a Sharpe of 8.21 on AAPL. Which reading is correct?

    • AThat the multi-agent architecture reliably beats the market
    • BThat with a Sharpe of 8 the strategy is risk-free
    • CThat it is a three-month result, on three stocks shown, with no declared transaction costs, using models that may have read texts from that period; the authors themselves say the Sharpe exceeds the range they expected
    • DThat the chosen baselines (MACD, SMA…) were professional benchmark strategies
    Show answer

    Answer C. That it is a three-month result, on three stocks shown, with no declared transaction costs, using models that may have read texts from that period; the authors themselves say the Sharpe exceeds the range they expected

    The paper's footnote explains that they evaluated only three months because of the cost of 11 LLM calls and more than 20 tool calls per prediction, and that the highest Sharpe “exceeds our expected empirical range”. A Sharpe over 60 sessions has a huge standard error, and without costs or control of look-ahead bias there is no basis for generalizing.

  2. In the lesson, the multi-agent pipeline with a single debate round gives an average net Sharpe of 0.16 over ten runs, with 5 or 10 rounds it rises to 0.65-0.68 and the equivalent deterministic rule, with no LLM at all, gives 1.07. What explains the pattern?

    • AThat the debate generates new information in each round
    • BThat each round rereads the same evidence with new noise: averaging readings reduces the noise and the verdict converges to a fixed function of the signals, which can be written without an LLM
    • CThat with more rounds the model learns from its mistakes
    • DThat costs go down with the number of rounds
    Show answer

    Answer B. That each round rereads the same evidence with new noise: averaging readings reduces the noise and the verdict converges to a fixed function of the signals, which can be written without an LLM

    The simulated agents do not invent signal: they read the four signals with an error and round them to a rating. Repeating the reading and averaging is an ensemble that approaches the expected value of the ratings. The limit is a deterministic rule, which costs zero calls and does not vary between runs.

  3. Between two runs of the pipeline with one round, 20% of the decisions (buy, sell or stay out) change. What does that imply for evaluating an agent of this kind?

    • AThat a single run of the backtest is not enough: it has to be repeated with several seeds or samples and the distribution of the result reported
    • BThat setting the temperature to zero is enough to make it stop happening
    • CThat 20% of the decisions are mistakes
    • DNothing: the average Sharpe is the only thing that matters
    Show answer

    Answer A. That a single run of the backtest is not enough: it has to be repeated with several seeds or samples and the distribution of the result reported

    With one round, the net Sharpe of ten runs goes from −0.22 to 0.38: the result of one run is a sample from a wide distribution. Lesson 6.4.4 shows that with real LLMs, even with deterministic decoding, the actions also change between runs.

  4. The social media signal has the highest weekly IC of the four (0.018, t = 3.8), but its long-short portfolio loses 1.9% a year net. Why?

    • ABecause the IC is computed wrong
    • BBecause social media have no effect in this data
    • CBecause the portfolio is long only
    • DBecause the signal changes almost completely every week: it turns over 147% a week and 10 basis points per unit of turnover eat up the whole gross return of 5.8%
    Show answer

    Answer D. Because the signal changes almost completely every week: it turns over 147% a week and 10 basis points per unit of turnover eat up the whole gross return of 5.8%

    The generator's retail sentiment is persistent for only a few days (daily AR(1) with ρ = 0.7) and the quintile portfolios are rebuilt almost entirely every week. A signal with a high IC and a short life is worth little if it is traded with costs (lesson 2.4.7).

  5. Does disagreement among the four analysts predict that the decision will fail?

    • AYes: with high disagreement the hit rate drops to 30%
    • BNot in this data: the hit rate is between 50.5% and 52% at the three levels of disagreement, because the four sources measure independent effects and disagreeing is normal
    • CYes, and that is why the risk manager should trade only when there is disagreement
    • DIt cannot be measured without a real LLM
    Show answer

    Answer B. Not in this data: the hit rate is between 50.5% and 52% at the three levels of disagreement, because the four sources measure independent effects and disagreeing is normal

    The four signals come from different planted effects (value and surprise, momentum and reversal, news drift, retail sentiment). One saying buy and another saying sell does not mean either is wrong. Filtering by disagreement, as the lesson's risk manager does, reduces positions but does not improve the hit rate.

Key points

  • TradingAgents organizes a simulated desk: four analysts (fundamental, sentiment, news, technical), bull and bear researchers who debate for n rounds, a trader, a risk team with three profiles and a manager who approves. Its original evaluation covers three months of 2024 with no declared costs.
  • An analyst agent can only pass on the signal that is in its data: in the course, the four signals have weekly ICs of 0.010 to 0.018 against known planted effects.
  • Debate among agents that read with noise works as an average: more rounds bring the verdict closer to a deterministic rule that can be written without an LLM, while calls grow by 2 per round and company.
  • Decisions change between runs (20% with one round) and the net Sharpe of ten runs goes from −0.22 to 0.38: a single-run backtest does not describe the system.
  • With 10 basis points per unit of turnover, fast signals lose money; the fundamental signal alone and the rule with no LLM are the only ones with a net t above 2, and after trying more than ten configurations, Deflated Sharpe before believing anything (lesson 2.5.4).

Resources