---
title: "5.1.3 Economic metrics for trading models"
description: "Why accuracy, AUC and R² mislead; IC, ICIR, weighted accuracy, the signal's Sharpe after costs and calibration, measured on the level's panel."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/economic-metrics-for-trading-models/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/economic-metrics-for-trading-models.png"
---
# 5.1.3 · Economic metrics for trading models: beyond accuracy
> A trading model is not judged by how often it is right, but by the money left by the portfolio built with it, after costs. In the level's panel, the signal with the worst accuracy is the one that earns the most net, and a LightGBM with an AUC above 0.5 gives probabilities worse than a coin. This lesson measures each metric on the same signals and shows which one to use for each decision.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.1 · ML fundamentals applied to finance
- Duration: 65 min
- Prerequisites: [2.4.7 · Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-transaction-cost-modeling/), [2.6.1 · CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index](https://pythonparatrading.com/en/python-trading-course/level-2/return-and-risk-metrics/), [4.2.2 · Kelly criterion and volatility targeting: how much to risk](https://pythonparatrading.com/en/python-trading-course/level-4/kelly-criterion-and-volatility-targeting/), [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/), [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/)
## By the end of this lesson
- Explain with data why accuracy, AUC and R² do not rank trading models well.
- Compute the weekly IC, the ICIR and the effective breadth, and relate them to the fundamental law of active management.
- Build a signal's long-short decile portfolio and measure its gross Sharpe, turnover, net Sharpe and breakeven cost.
- Reduce turnover by smoothing the signal and measure the effect on the net Sharpe.
- Evaluate a classifier's calibration with a reliability diagram, Brier score and log loss, recalibrate it and connect the error with Kelly sizing.
## The question each metric answers
The usual ML metrics answer statistical questions: how often is the class right? How wrong is it on average,
squared? Does it rank positives and negatives well? A trading system asks a different question: **how much money
does the portfolio built with these predictions leave, after paying to build it?** The two families of questions do
not rank models the same way, and the rest of the lesson measures it with six signals on the same panel and the same
period.
| Metric | What it measures | What it ignores |
|---|---|---|
| Accuracy | Fraction of correct signs (or classes) | Magnitudes, majority class, costs |
| AUC | Probability of ranking a positive above a negative | Magnitudes, scale, costs |
| R², squared error | Error of the prediction in its own units | That for ranking only the order counts |
| IC, ICIR | Quality of the cross-sectional order and its stability | Costs and turnover |
| Signal Sharpe, net | Portfolio return per unit of risk after costs | Capacity, tails (looked at separately) |
| Brier, log loss, reliability | Whether the probabilities mean what they say | The order (a calibrated model may rank nothing) |
## Accuracy misleads: the majority class
The first problem with accuracy is its reference point. The daily returns of the `indice` asset in `vol5_diario`
have a positive drift and are, by construction, **unpredictable**: their conditional mean is constant (the level
README uses it as a control). A classifier that always says “up”:
```python
import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, balanced_accuracy_score
v = pd.read_parquet("vol5_diario.parquet")
ind = v[(v["activo"] == "indice") & (v["fecha"] >= "2018-01-01")]
sube = (ind["rendimiento"] > 0).astype(int)
siempre = np.ones(len(sube), dtype=int)
print(f"Accuracy: {accuracy_score(sube, siempre):.1%}")
print(f"Balanced accuracy: "
f"{balanced_accuracy_score(sube, siempre):.1%}")
```
```output
Accuracy: 56.6%
Balanced accuracy: 50.0%
```
A 56.6% accuracy without any information: it is the frequency of up days. Any directional classifier is compared
with that figure, not with 50%. **Balanced accuracy** (the mean of the accuracy in each class) removes the effect: a
constant classifier gives exactly 0.5. In the panel the label is “above the week's median”, which is balanced by
construction, and the problem shifts to magnitude and costs.
## Six signals, six different rankings
The six signals are evaluated in 2019-2025 on `panel5`:
- **oracle**: the true expected return (`alfa_verdadero`), the ceiling from lesson 5.1.1.
- **ridge**: a ridge with the 21 ranked features, trained on 2012-2018 (as in lesson 5.1.1).
- **baseline**: the sum of four ranks with the sign from the literature (lesson 5.1.2).
- **value**, **ey** and **reversal**: standalone features (`valor_bp`, `ey` and `ret_1s` with its sign flipped).
```python
from sklearn.linear_model import Ridge
f = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
ver = pd.read_parquet("panel5_verdad.parquet")
d = (f.merge(et[["fecha", "ticker", "ret_sig_semana"]],
on=["fecha", "ticker"])
.merge(ver[["fecha", "ticker", "alfa_verdadero"]],
on=["fecha", "ticker"])
.dropna(subset=["ret_sig_semana"]).reset_index(drop=True))
g = d.groupby("fecha")
d["y"] = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean")
d["arriba"] = (d["ret_sig_semana"]
> g["ret_sig_semana"].transform("median")).astype(int)
fuera = ("fecha", "ticker", "sector", "revision_analistas")
feats = [c for c in f.columns if c not in fuera]
X = g[feats].rank(pct=True).sub(0.5).fillna(0)
ent = (d["fecha"] < "2019-01-01").to_numpy()
pru = (d["fecha"] >= "2019-01-08").to_numpy()
rk = g[["mom_12_1", "ret_1s", "valor_bp", "sue"]].rank(pct=True)
d["oraculo"] = d["alfa_verdadero"]
d["ridge"] = Ridge(alpha=10.0).fit(X[ent], d.loc[ent, "y"]).predict(X)
d["baseline"] = (rk["mom_12_1"] - rk["ret_1s"] + rk["valor_bp"]
+ rk["sue"])
d["value"], d["ey_"], d["reversion"] = (d["valor_bp"], d["ey"],
-d["ret_1s"])
senales = ["oraculo", "ridge", "baseline", "value", "ey_",
"reversion"]
t = d[pru].reset_index(drop=True)
print(t["fecha"].nunique(), "weeks,", len(t), "rows")
```
```output
364 weeks, 145600 rows
```
First, the classification metrics. Each signal's prediction is turned into “above the week's median” and compared
with the binary label. **Weighted accuracy** weights each hit by the size of the move, $|y_i|$: a model that gets
the large moves right and misses the small ones comes out better than in plain accuracy. **AUC** uses the
continuous signal and measures the probability that a stock from the good half has a larger signal than one from
the bad half:
```python
from sklearn.metrics import roc_auc_score
def clasificacion(s, col):
alta = s[col] > s.groupby("fecha")[col].transform("median")
ok = alta.astype(int) == s["arriba"]
return {"acierto": ok.mean(),
"acierto_pond": (s["y"].abs() * ok).sum()
/ s["y"].abs().sum(),
"auc": roc_auc_score(s["arriba"], s[col])}
pd.DataFrame({c: clasificacion(t, c) for c in senales}).T.round(4)
```
```output
acierto acierto_pond auc
oraculo 0.5077 0.5108 0.5124
ridge 0.5061 0.5106 0.5087
baseline 0.5056 0.5086 0.5089
value 0.5041 0.5062 0.5055
ey_ 0.5034 0.5051 0.5046
reversion 0.5034 0.5073 0.5052
```
All the signals are between 50.3% and 50.8% accuracy: it is the scale of lesson 5.1.1. Weighted accuracy changes
the order: reversal rises from 50.34% to 50.73% because it is right more often on large moves (a sharp fall in the
previous week corrects more), and it overtakes value and `ey`. AUC ranks almost the same as accuracy. None of the
three metrics says anything about how much it costs to trade each signal, which is what decides the result.
## R² and squared error: scale versus order
A regression model is evaluated by default with R² or squared error. For ranking stocks, the scale of the prediction
is irrelevant; for R², it is almost everything:
```python
from sklearn.metrics import r2_score
def ic_semanal(s, col, y="y"):
r = s.groupby("fecha")[[col, y]].rank()
r = r - r.groupby(s["fecha"]).transform("mean")
num = (r[col] * r[y]).groupby(s["fecha"]).sum()
den = ((r[col] ** 2).groupby(s["fecha"]).sum()
* (r[y] ** 2).groupby(s["fecha"]).sum()) ** 0.5
return num / den
t["oraculo_x5"] = 5 * t["oraculo"]
for c in ["oraculo", "oraculo_x5", "ridge"]:
print(f"{c:11s} R² {r2_score(t['y'], t[c]):+.3%} "
f"IC {ic_semanal(t, c).mean():.4f}")
```
```output
oraculo R² +0.044% IC 0.0254
oraculo_x5 R² -2.531% IC 0.0254
ridge R² +0.040% IC 0.0190
```
The oracle explains 0.044% of the label's variance. Multiplied by 5 it has **exactly** the same IC and would produce
the same decile portfolio, but its R² falls to −2.5%: worse than always predicting zero. The ridge has an R² similar
to the oracle's and a lower IC. R² is useful when the magnitude of the prediction is used as is (for sizing, for the
volatility in lesson 5.8.5); for choosing between ranking signals, it misleads.
## IC, ICIR and the fundamental law
The weekly **IC** measures the quality of the order; the **ICIR**, its mean divided by its standard deviation
across weeks, measures its stability. Annualized ($\times\sqrt{52}$) it is comparable to a Sharpe ratio: if the
portfolio weights each stock in proportion to its rank, its weekly return is roughly proportional to that week's IC
times the dispersion of returns.
The **fundamental law of active management** (Grinold, 1989) gives a bound on the information ratio with $N$
independent bets per period:
$$\mathrm{IR} \approx \mathrm{IC}\cdot\sqrt{\text{breadth}}, \qquad
\text{breadth} = N \times \text{periods per year}.$$
The cell checks it against the measured ICIR and computes the **effective breadth**: if the bets were independent,
the standard deviation of the weekly IC would be $1/\sqrt{N}$; solving for $N$ from the measured standard deviation
gives how many truly independent bets there are each week.
```python
filas = {}
for c in senales:
ic = ic_semanal(t, c)
n = t.groupby("fecha")[c].count().mean()
filas[c] = {"ic": ic.mean(), "desv_ic": ic.std(),
"icir_anual": ic.mean() / ic.std() * np.sqrt(52),
"ley_fund": ic.mean() * np.sqrt(n * 52),
"amplitud_efectiva": 1 / ic.std() ** 2}
pd.DataFrame(filas).T.round(3)
```
```output
ic desv_ic icir_anual ley_fund amplitud_efectiva
oraculo 0.025 0.100 1.829 3.668 99.437
ridge 0.019 0.088 1.562 2.737 130.271
baseline 0.018 0.082 1.567 2.560 149.781
value 0.010 0.072 1.041 1.495 194.029
ey_ 0.009 0.062 1.093 1.365 256.640
reversion 0.016 0.103 1.099 2.258 94.724
```
With 400 stocks, the fundamental law promises the oracle an information ratio of 3.7. The measured ICIR is 1.83:
half. The standard deviation of the IC across weeks is 0.10, twice what 400 independent bets would give
($1/\sqrt{400} = 0.05$), and it is equivalent to about 100. The 400 stocks share the market, sectors and slow
features: the breadth in the law is that of **independent decisions**, not of assets, and counting it wrong inflates
any expectation. The ICIR already includes that dependence, which is why it is the measure of a signal's quality
before costs.
## The signal's Sharpe after costs
The measure that rules is the portfolio. The level's convention (data README): every week, long the top decile of
the signal and short the bottom one, equal-weighted, with a gross notional of 2 (1 long and 1 short); simple
returns; turnover = sum of the absolute weight changes; cost = 10 bp per side on turnover (lesson 2.4.7). The
**breakeven cost** is the cost per side that brings the net return to zero: mean gross return divided by mean
turnover.
```python
def pesos_deciles(s, col):
q = s.groupby("fecha")[col].rank(pct=True)
w = pd.Series(np.select([q > 0.9, q <= 0.1], [1.0, -1.0], 0.0),
index=s.index)
w = 2 * w / w.abs().groupby(s["fecha"]).transform("sum")
return (s[["fecha", "ticker"]].assign(w=w)
.pivot(index="fecha", columns="ticker", values="w")
.fillna(0.0))
R = t.pivot(index="fecha", columns="ticker", values="ret_sig_semana")
def cartera(W, coste_pb=10):
bruto = (W * R).sum(axis=1)
rot = W.diff().abs().sum(axis=1)
rot.iloc[0] = W.iloc[0].abs().sum()
neto = bruto - coste_pb / 1e4 * rot
sr = lambda x: x.mean() / x.std() * np.sqrt(52)
return {"bruto_anual": bruto.mean() * 52, "sharpe_bruto": sr(bruto),
"rotacion_sem": rot.mean(), "sharpe_neto": sr(neto),
"equilibrio_pb": 1e4 * bruto.mean() / rot.mean()}
W = {c: pesos_deciles(t, c) for c in senales}
res = pd.DataFrame({c: cartera(W[c]) for c in senales}).T
res["acierto"] = [clasificacion(t, c)["acierto"] for c in senales]
res.round(3)
```
```output
bruto_anual sharpe_bruto rotacion_sem sharpe_neto equilibrio_pb acierto
oraculo 0.167 1.667 2.649 0.287 12.089 0.508
ridge 0.129 1.404 2.541 -0.029 9.797 0.506
baseline 0.113 1.331 2.030 0.090 10.726 0.506
value 0.076 1.094 0.105 1.015 139.932 0.504
ey_ 0.061 0.889 0.102 0.810 114.183 0.503
reversion 0.133 1.295 3.420 -0.435 7.486 0.503
```
The figure at the top crosses the two extreme columns of the table: accuracy and net Sharpe. The order is reversed
almost completely:
- The **oracle** has the best accuracy (50.77%) and the best gross Sharpe (1.67), but its portfolio turns over 265%
a week: it withstands 12 bp per side and with 10 bp it ends at 0.29.
- The **ridge** and the **baseline** give gross Sharpes of 1.40 and 1.33 and net ones of −0.03 and 0.09: the same
story with less signal. It is what the level README gives for 2015-2025.
- The one-week **reversal** has a gross Sharpe of 1.30 and the worst turnover (342%): with 10 bp it loses money
(−0.43). Its breakeven cost, 7 bp, is lower than the bid-ask spread of many real stocks.
- **Value** and **ey** have some of the lowest accuracies (50.41% and 50.34%) and the **best net Sharpes** (1.01 and
0.81). Their portfolios change 10% a week: they withstand 140 and 114 bp per side.
### Smoothing to turn over less
If a signal has a slow part (lesson 5.1.1: the oracle keeps an IC of 0.011 with weeks of delay), you can trade some
IC for much less turnover. The simplest way is an exponential average of the signal's ranks over the last few
weeks, which only uses past information:
```python
def suavizar(col, vida_media=2):
p = d.pivot(index="fecha", columns="ticker", values=col)
s = p.rank(axis=1, pct=True).ewm(halflife=vida_media).mean()
return s.loc[t["fecha"].unique()]
filas = {}
for c in ["oraculo", "ridge", "baseline", "reversion"]:
S = suavizar(c)
larga = S.stack().rename("s").reset_index()
ts = t[["fecha", "ticker"]].merge(larga, on=["fecha", "ticker"])
filas[c + "_suave"] = {**cartera(pesos_deciles(ts, "s")),
"ic": ic_semanal(ts.assign(y=t["y"]),
"s").mean()}
pd.DataFrame(filas).T.round(3)
```
```output
bruto_anual sharpe_bruto rotacion_sem sharpe_neto equilibrio_pb ic
oraculo_suave 0.120 1.184 1.016 0.661 22.658 0.016
ridge_suave 0.103 1.231 0.999 0.613 19.913 0.012
baseline_suave 0.091 1.202 0.626 0.771 27.878 0.013
reversion_suave 0.059 0.607 2.104 -0.514 5.417 0.008
```
*Fig. — Annual net Sharpe of the decile long-short portfolio by cost per side, 2019-2025: high-turnover signals cross zero quickly.* Four descending lines from zero cost. Reversal: from 1.30 to zero near 7.5 bp. Oracle: from 1.67 to zero near 12 bp. Smoothed oracle (highlighted): from 1.18 to zero near 23 bp. Value: almost flat, from 1.09 to 0.90 at 25 bp.
With a half-life of two weeks, the oracle's IC drops from 0.025 to 0.016 and its gross Sharpe from 1.67 to 1.18,
but turnover falls from 265% to 102% a week: the breakeven cost rises from 12 to 23 bp and the net Sharpe, from 0.29
to 0.66. The ridge goes from −0.03 to 0.61 and the baseline, from 0.09 to 0.77. Reversal, whose IC is all in the
first week, has no slow part to keep. The figure draws the net Sharpe as a function of the cost per side: the lines
of the fast signals fall quickly and cross zero between 7 and 12 bp; the smoothed ones and value hold up much
longer.
Optimizing the smoothing (the half-life, the decile thresholds) with the same test data is another experiment that
counts for the Deflated Sharpe (lessons 2.5.4 and 5.4.4). Here a single value was tried, fixed in advance; lesson
4.1.5 treats turnover as a constraint of the optimization.
## Calibration: when the probability decides the size
Some models are not used to rank but to decide **how much** to bet: the meta-labeling of lesson 5.3.3 sizes each
trade with the secondary model's probability of success, and Kelly (lesson 4.2.2) bets $f = 2p - 1$ on a symmetric
bet. There it is not enough for $p$ to rank well: it has to **mean** what it says. A model is calibrated if, of the
times it says 0.6, it is right 60% of the time.
The experiment: two classifiers that predict whether a stock will end above the median the following week, trained
on 2012-2016. A logistic regression and a fairly flexible LightGBM (63 leaves, 300 trees). With 2017-2018, which
neither has seen, the LightGBM is recalibrated with an **isotonic** regression (a monotonic function of its
probability) through `CalibratedClassifierCV` and `FrozenEstimator`, which leaves it as is and only fits the
recalibration. Everything is evaluated on 2019-2025:
```python
import lightgbm as lgb
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss
a = (d["fecha"] < "2017-01-01").to_numpy()
c = ((d["fecha"] >= "2017-01-13")
& (d["fecha"] < "2019-01-01")).to_numpy()
y = d["arriba"]
logit = LogisticRegression(C=1.0, max_iter=1000).fit(X[a], y[a])
gbm = lgb.LGBMClassifier(
n_estimators=300, learning_rate=0.1, num_leaves=63,
min_child_samples=20, random_state=0, n_jobs=4,
deterministic=True, force_row_wise=True, verbose=-1,
).fit(X[a], y[a])
recal = CalibratedClassifierCV(FrozenEstimator(gbm),
method="isotonic").fit(X[c], y[c])
yp, fiab, tabla = y[pru].to_numpy(), {}, {}
for nombre, m in [("logistica", logit), ("lgbm", gbm),
("lgbm_recal", recal)]:
p = m.predict_proba(X[pru])[:, 1]
tramo = pd.qcut(p, 10, labels=False, duplicates="drop")
fiab[nombre] = pd.DataFrame({"p": p, "obs": yp}).groupby(
tramo).mean()
tabla[nombre] = {"auc": roc_auc_score(yp, p),
"brier": brier_score_loss(yp, p),
"log_loss": log_loss(yp, p),
"kelly_medio": np.abs(2 * p - 1).mean()}
tabla["coin"] = {"auc": 0.5, "brier": 0.25,
"log_loss": np.log(2), "kelly_medio": 0.0}
print(pd.DataFrame(tabla).T.round(4))
print(fiab["lgbm"].T.round(3))
print(fiab["logistica"].T.round(3))
```
```output
auc brier log_loss kelly_medio
logistica 0.5060 0.2500 0.6932 0.0196
lgbm 0.5006 0.2546 0.7028 0.1008
lgbm_recal 0.5019 0.2500 0.6944 0.0091
coin 0.5000 0.2500 0.6931 0.0000
0 1 2 3 4 5 6 7 8 9
p 0.373 0.438 0.463 0.480 0.494 0.507 0.521 0.538 0.563 0.622
obs 0.497 0.499 0.499 0.499 0.500 0.500 0.505 0.511 0.493 0.497
0 1 2 3 4 5 6 7 8 9
p 0.478 0.487 0.492 0.495 0.499 0.502 0.505 0.509 0.513 0.520
obs 0.490 0.495 0.494 0.498 0.500 0.501 0.508 0.505 0.503 0.507
```
*Fig. — Reliability diagram for 2019-2025: mean predicted probability by decile versus the observed frequency of ending above the median, for a logistic regression and a LightGBM.* A dashed diagonal marks perfect calibration. The LightGBM's ten points (highlighted) run from 0.37 to 0.62 in predicted probability, yet their observed frequency stays flat between 0.49 and 0.51: overconfidence. The logistic regression's points cluster between 0.48 and 0.52, near the centre.
The reliability diagram (the figure) puts the mean predicted probability in each decile on the horizontal axis and
the observed frequency on the vertical one; a calibrated model falls on the diagonal. The LightGBM predicts between
0.37 and 0.62 on average per decile and the observed frequency is 0.49-0.51 in all of them: it is
**overconfident**. Its AUC (0.501) barely departs from 0.5, and its Brier score (0.2546) and log loss (0.7028) are
**worse than a coin's** (0.25 and 0.6931). Sized with Kelly, it would bet on average 10% of capital per position
with no edge at all. The logistic regression ranks somewhat better (AUC 0.506) and its probabilities move in a
narrow range (0.48-0.52), and even so its log loss (0.6932) does not beat the coin's: the observed frequency only
ranges from 0.49 to 0.51 and, with this signal-to-noise ratio, a per-stock probability other than 0.5 carries
almost no information. Recalibrated, the LightGBM is compressed around 0.5 and its average Kelly size drops to 0.9%;
its log loss stays slightly above the coin's (0.6944). Recalibration does not give it signal, but it removes the
false confidence.
## Which metric for which decision
The metric is chosen by the use the prediction will be put to:
```mermaid
flowchart TB
U{What is it used for?} --> O[Rank assets]
U --> T[Size positions]
U --> F[Filter trades]
O --> IC[IC and ICIR]
T --> CA[Calibration]
F --> PR[Precision and P&L]
IC --> SN[Net Sharpe]
CA --> SN
PR --> SN
SN --> DSR[Deflated Sharpe]
```
- **Rank assets** (cross-sectional portfolios): weekly IC, its standard deviation and ICIR to compare signals;
Sharpe of the decile portfolio, gross and net, with its turnover and breakeven cost.
- **Size positions** (meta-labeling, Kelly, volatility targeting): calibration (Brier score, log loss, reliability
diagram) in addition to the order; if the prediction is a volatility, QLIKE or error in logs (lesson 5.8.5).
- **Filter trades** from a primary system: precision of the trades let through and mean P&L per trade versus that of
the unfiltered system (lesson 5.3.3).
- **Always**: the net Sharpe of the complete strategy, with its standard error, and discounted for the number of
experiments with the Deflated Sharpe (lessons 2.5.4 and 5.4.4). And the tails and the worst losing streak, which no
Sharpe sees (lesson 2.6.1).
During training you can optimize a statistical loss (squared error, cross-entropy), which is stable and
differentiable; lesson 5.6.4 tries a Sharpe-based loss. But the **choice** between models, hyperparameters and
variants is made with economic metrics, on data the model has not seen.
## Exercises
1. Compute the net Sharpe of the oracle's portfolio and of value's with costs of 0, 5, 10, 20 and 40 bp per side.
From what cost does value beat the oracle?
**Solution.**
```python
costes = [0, 5, 10, 20, 40]
pd.DataFrame({c: [cartera(W[c], k)["sharpe_neto"] for k in costes]
for c in ["oraculo", "value"]},
index=pd.Index(costes, name="coste_pb")).round(2)
```
```output
oraculo value
coste_pb
0 1.67 1.09
5 0.98 1.05
10 0.29 1.01
20 -1.08 0.94
40 -3.78 0.78
```
Without costs, the oracle clearly wins (1.67 versus 1.09). With 5 bp per side value is already better (1.05 versus
0.98), and at 10 bp the difference is huge (1.01 versus 0.29). At 40 bp the oracle loses a lot (−3.78) and value
barely notices the cost (0.78). The crossover is below 5 bp per side: less than it costs to trade a mid-cap stock in
a real market.
2. Repeat the IC and ICIR table with the `ret_sig_21d` label (21-session return) for the oracle and value. Why is it
not advisable to compute their significance treating the weeks as if they were independent?
**Solution.**
```python
y21 = et[["fecha", "ticker", "ret_sig_21d"]]
t21 = t.merge(y21, on=["fecha", "ticker"]).dropna(
subset=["ret_sig_21d"])
for c in ["oraculo", "value"]:
ic = ic_semanal(t21, c, "ret_sig_21d")
ac1 = ic.autocorr(1)
print(f"{c:8s} IC {ic.mean():.4f} annual ICIR "
f"{ic.mean() / ic.std() * np.sqrt(52):.2f} "
f"IC autocorrelation {ac1:.2f}")
```
```output
oraculo IC 0.0207 annual ICIR 1.61 IC autocorrelation 0.47
value IC 0.0209 annual ICIR 2.04 IC autocorrelation 0.76
```
With the 21-session label, the labels of consecutive weeks overlap by about three quarters, and one week's IC looks
very much like the next one's: the first-order autocorrelation of the IC series measures it. The ICIR computed as if
the weeks were independent overstates the significance; you need a standard error corrected for autocorrelation
(Newey-West, lesson 2.1.1) or to use only non-overlapping weeks.
3. A vendor offers you a model with “58% daily directional accuracy on the index”. What three questions would you ask
before looking at anything else?
**Solution.**
First: what is the frequency of up days in the same period? With an index's drift, “always up” already gives
55-57%, so 58% may add nothing; I would ask for the balanced accuracy or the comparison with that reference. Second:
how much does it win or lose when it is right and when it is wrong? Accuracy does not see magnitudes: I would ask
for the strategy's P&L, the accuracy weighted by the size of the move and the net Sharpe with costs and turnover.
Third: how was it validated and how many variants were tried? I would ask for the out-of-sample period, the
validation method (lessons 5.4.1-5.4.3) and the number of experiments for the Deflated Sharpe (5.4.4).
## Self-assessment
1. A classifier that predicts “up” every day is right on 56.6% of the index's days in 2018-2025. What is it worth as a predictor?
- A. A lot: it is right more often than a 52% model
- B. It depends on the horizon
- C. Nothing: it is the frequency of up days (the index's drift); its balanced accuracy is exactly 50% and in this data the return is unpredictable by construction
- D. It is a good baseline for trading
*Answer C: Nothing: it is the frequency of up days (the index's drift); its balanced accuracy is exactly 50% and in this data the return is unpredictable by construction.* Accuracy is compared with the frequency of the majority class, not with 50%. Balanced accuracy (the mean of the accuracy in each class) removes the effect: a constant classifier gives 0.5.
2. In 2019-2025, value as a standalone signal has one of the worst accuracies of the six signals (50.41%) and the best net Sharpe (1.01). Why?
- A. By chance
- B. Because it turns over 10% a week versus 250-340% for the others: its breakeven cost is 140 bp per side, and with 10 bp it loses almost nothing
- C. Because value has the highest IC
- D. Because the accuracy is miscalculated
*Answer B: Because it turns over 10% a week versus 250-340% for the others: its breakeven cost is 140 bp per side, and with 10 bp it loses almost nothing.* Accuracy sees neither magnitude nor turnover. The oracle has more IC and a higher gross Sharpe (1.67), but its portfolio is renewed every week and only withstands 12 bp per side; with 10 bp it ends at 0.29.
3. Multiplying the oracle's prediction by 5 changes its out-of-sample R² from +0.04% to −2.5%. What happens to its long-short decile portfolio?
- A. Nothing: the order of the stocks is the same, so the IC and the portfolio are identical. R² penalizes scale, which does not matter in a rank-based portfolio
- B. It loses money
- C. It earns five times more
- D. Its turnover is multiplied by 5
*Answer A: Nothing: the order of the stocks is the same, so the IC and the portfolio are identical. R² penalizes scale, which does not matter in a rank-based portfolio.* R² measures the squared error of the prediction in its own units. To rank stocks only the order counts; to size positions the scale does count, and then what you look at is calibration.
4. The oracle's weekly IC is 0.025 and there are 400 stocks. The fundamental law gives an information ratio of 0.025·√(400·52) ≈ 3.7, but the measured annual ICIR is 1.8. Where does the difference come from?
- A. From a calculation error
- B. From costs
- C. From the IC being rank-based
- D. From the 400 weekly bets not being independent: they share the market, sectors and features. The standard deviation of the IC across weeks (≈ 0.1) corresponds to about 100 independent bets per week, not 400
*Answer D: From the 400 weekly bets not being independent: they share the market, sectors and features. The standard deviation of the IC across weeks (≈ 0.1) corresponds to about 100 independent bets per week, not 400.* With N independent bets the standard deviation of the IC would be ≈ 1/√N = 0.05. The measured one is twice that, so the effective breadth is a quarter. The breadth in the fundamental law is that of independent decisions, not of assets.
5. A LightGBM gives probabilities that a stock ends above the median between 0.31 and 0.68; in those buckets the observed frequency is always ≈ 0.50. If positions are sized with Kelly (f = 2p − 1), what happens?
- A. Nothing: the AUC is greater than 0.5
- B. Too little is bet
- C. The right amount is bet
- D. On average 10% of capital is bet per position with no real edge: the model is overconfident, its Brier score and log loss are worse than a coin's (0.2546 versus 0.25)
*Answer D: On average 10% of capital is bet per position with no real edge: the model is overconfident, its Brier score and log loss are worse than a coin's (0.2546 versus 0.25).* Uncalibrated probabilities turn noise into size. Recalibrated with 2017-2018 (isotonic), the probabilities are compressed around 0.5 and the average Kelly size drops to 0.9%.
## Key points
- Accuracy is compared with the majority class and does not see magnitudes: accuracy weighted by |return| and balanced accuracy partly correct this.
- The weekly IC (rank correlation) measures what matters for ranking; its mean divided by its standard deviation, annualized (ICIR), resembles the gross Sharpe of the long-short portfolio.
- The fundamental law IR ≈ IC·√breadth uses independent bets: with 400 correlated stocks, the measured effective breadth is about 100 per week.
- The final metric is the portfolio's net Sharpe: the oracle goes from 1.67 gross to 0.29 net with 10 bp per side, and value, with worse accuracy, gives 1.01.
- The breakeven cost (gross return / turnover) says how many bp per side a signal can withstand; smoothing raises the oracle's from 12 to 23 bp and its net Sharpe from 0.29 to 0.66.
- If the probability decides the size, calibration is required (Brier score, log loss, reliability diagram): an overconfident model bets without an edge.
## Resources
### Research
- [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min
### Documentation
- [scikit-learn: Metrics and scoring](https://scikit-learn.org/stable/modules/model_evaluation.html) · scikit-learn.org
- [scikit-learn: Probability calibration](https://scikit-learn.org/stable/modules/calibration.html) · scikit-learn.org
- [scikit-learn: FrozenEstimator](https://scikit-learn.org/stable/modules/generated/sklearn.frozen.FrozenEstimator.html) · scikit-learn.org
### Books
- [Grinold, R. C. (1989): The Fundamental Law of Active Management. The Journal of Portfolio Management 15(3), 30-37](https://doi.org/10.3905/jpm.1989.409211) · doi.org
### Data
- [Level 5 data: panel truth, with each stock's expected return (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_verdad.parquet) · Download · panel5_verdad.parquet
- [Level 5 data: daily returns and realized variance of six assets (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet
Previous lesson: [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/)
Next lesson: [5.2.1 · Price, volume and microstructure features](https://pythonparatrading.com/en/python-trading-course/level-5/price-volume-microstructure-features/)