Python Para Trading

enes

Search

Contact
Contact

LanguageEspañol

Lesson 5.1.3 · 65 min

Economic metrics for trading models: beyond accuracy

A trading model is not judged by how often it is right, but by the money left by the portfolio built with it, after costs. In the level's panel, the signal with the worst accuracy is the one that earns the most net, and a LightGBM with an AUC above 0.5 gives probabilities worse than a coin. This lesson measures each metric on the same signals and shows which one to use for each decision.

Fig. — Hit rate versus net Sharpe (10 bp per side) of six signals on the panel, 2019-2025: one metric does not rank like the other
03/03
Module
5.1 · ML fundamentals applied to finance
Duration
65 min
Prerequisites
2.4.7 · Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing, 2.6.1 · CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index, 4.2.2 · Kelly criterion and volatility targeting: how much to risk, 5.1.1 · Why financial machine learning is different, 5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines
Author
Updated
Level 5 lessons

  1. 5.1 · ML fundamentals applied to finance

    1. 5.1.1 Why financial machine learning is different
    2. 5.1.2 Supervised, unsupervised and reinforcement learning; scikit-learn pipelines
    3. 5.1.3 Economic metrics for trading models: beyond accuracy

  2. 5.2 · Feature engineering

    1. 5.2.1 Price, volume and microstructure features
    2. 5.2.2 Point-in-time fundamentals, macro and alternative data
    3. 5.2.3 Fractional differentiation for features
    4. 5.2.4 Leakage-free normalization and cross-sectional rank
    5. 5.2.5 Feature importance: MDI, MDA, SFI and SHAP

  3. 5.3 · Labelling and sampling

    1. 5.3.1 Fixed-horizon labeling and its problems
    2. 5.3.2 Triple-barrier method: labels that mimic a trade
    3. 5.3.3 Meta-labeling: one model for the side and another for the size
    4. 5.3.4 Label uniqueness, overlap weights and the sequential bootstrap
    5. 5.3.5 Event-based sampling: the symmetric CUSUM filter

  4. 5.4 · Validation for financial series

    1. 5.4.1 Why random k-fold fails on financial data
    2. 5.4.2 Purged k-fold with embargo and CPCV for ML models
    3. 5.4.3 Walk-forward with retraining: windows and frequency
    4. 5.4.4 Backtesting ML signals with costs and Deflated Sharpe

  5. 5.5 · Classical models

    1. 5.5.1 Regularized regressions: Ridge, Lasso, Elastic Net and logistic
    2. 5.5.2 Decision trees, Random Forest and bagging
    3. 5.5.3 Gradient boosting: XGBoost, LightGBM and CatBoost
    4. 5.5.4 Optuna and the risk of overfitting the validation
    5. 5.5.5 Unsupervised learning: PCA, clustering and autoencoders
    6. 5.5.6 Learning to rank for stock selection

  6. 5.6 · Deep learning with Keras 3

    1. 5.6.1 Keras 3: one model, several compute engines
    2. 5.6.2 An MLP on a stock panel: regularize and compare
    3. 5.6.3 1D CNN, LSTM and GRU: windows, scaling and leaks
    4. 5.6.4 Callbacks, a Sharpe loss and the model in production

  7. 5.7 · Deep learning with PyTorch

    1. 5.7.1 PyTorch: tensors, autograd and the training loop
    2. 5.7.2 Time series architectures: TCN, N-BEATS, PatchTST and iTransformer
    3. 5.7.3 Probabilistic models: quantile loss, DeepAR and intervals
    4. 5.7.4 Graph neural networks for asset relationships
    5. 5.7.5 End-to-end portfolio optimization: differentiable layers and Sharpe loss

  8. 5.8 · Time series foundation models

    1. 5.8.1 Time series foundation models: pretrain once, forecast without training
    2. 5.8.2 Foundation model families: architecture, data and license
    3. 5.8.3 In practice: context, quantiles, covariates and fine-tuning
    4. 5.8.4 Foundation models in finance: what the evidence says
    5. 5.8.5 Lab: foundation models vs GARCH, HAR and LightGBM

  9. 5.9 · Reinforcement learning

    1. 5.9.1 MDP, Q-learning, PPO and Gymnasium environments
    2. 5.9.2 RL for execution, market making and dynamic hedging
    3. 5.9.3 Why reinforcement learning overfits in directional trading

  10. 5.10 · Interpretability, robustness and model risk

    1. 5.10.1 SHAP, partial dependence and the stability of feature importances
    2. 5.10.2 Data drift and concept drift: detecting them and what to do
    3. 5.10.3 Ensembles, signal combination and synthetic data

The question each metric answers

The usual ML metrics answer statistical questions: how often is the class right? How wrong is it on average, squared? Does it rank positives and negatives well? A trading system asks a different question: how much money does the portfolio built with these predictions leave, after paying to build it? The two families of questions do not rank models the same way, and the rest of the lesson measures it with six signals on the same panel and the same period.

MetricWhat it measuresWhat it ignores
AccuracyFraction of correct signs (or classes)Magnitudes, majority class, costs
AUCProbability of ranking a positive above a negativeMagnitudes, scale, costs
R², squared errorError of the prediction in its own unitsThat for ranking only the order counts
IC, ICIRQuality of the cross-sectional order and its stabilityCosts and turnover
Signal Sharpe, netPortfolio return per unit of risk after costsCapacity, tails (looked at separately)
Brier, log loss, reliabilityWhether the probabilities mean what they sayThe order (a calibrated model may rank nothing)

Accuracy misleads: the majority class

The first problem with accuracy is its reference point. The daily returns of the indice asset in vol5_diario have a positive drift and are, by construction, unpredictable: their conditional mean is constant (the level README uses it as a control). A classifier that always says “up”:

IN [01]python
import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, balanced_accuracy_score

v = pd.read_parquet("vol5_diario.parquet")
ind = v[(v["activo"] == "indice") & (v["fecha"] >= "2018-01-01")]
sube = (ind["rendimiento"] > 0).astype(int)
siempre = np.ones(len(sube), dtype=int)
print(f"Accuracy: {accuracy_score(sube, siempre):.1%}")
print(f"Balanced accuracy: "
      f"{balanced_accuracy_score(sube, siempre):.1%}")
OUT [01]Always “up”: accuracy and balanced accuracy
Accuracy: 56.6%
Balanced accuracy: 50.0%

A 56.6% accuracy without any information: it is the frequency of up days. Any directional classifier is compared with that figure, not with 50%. Balanced accuracy (the mean of the accuracy in each class) removes the effect: a constant classifier gives exactly 0.5. In the panel the label is “above the week's median”, which is balanced by construction, and the problem shifts to magnitude and costs.

Six signals, six different rankings

The six signals are evaluated in 2019-2025 on panel5:

  • oracle: the true expected return (alfa_verdadero), the ceiling from lesson 5.1.1.
  • ridge: a ridge with the 21 ranked features, trained on 2012-2018 (as in lesson 5.1.1).
  • baseline: the sum of four ranks with the sign from the literature (lesson 5.1.2).
  • value, ey and reversal: standalone features (valor_bp, ey and ret_1s with its sign flipped).
IN [02]python
from sklearn.linear_model import Ridge

f = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
ver = pd.read_parquet("panel5_verdad.parquet")
d = (f.merge(et[["fecha", "ticker", "ret_sig_semana"]],
             on=["fecha", "ticker"])
      .merge(ver[["fecha", "ticker", "alfa_verdadero"]],
             on=["fecha", "ticker"])
      .dropna(subset=["ret_sig_semana"]).reset_index(drop=True))
g = d.groupby("fecha")
d["y"] = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean")
d["arriba"] = (d["ret_sig_semana"]
               > g["ret_sig_semana"].transform("median")).astype(int)
fuera = ("fecha", "ticker", "sector", "revision_analistas")
feats = [c for c in f.columns if c not in fuera]
X = g[feats].rank(pct=True).sub(0.5).fillna(0)
ent = (d["fecha"] < "2019-01-01").to_numpy()
pru = (d["fecha"] >= "2019-01-08").to_numpy()

rk = g[["mom_12_1", "ret_1s", "valor_bp", "sue"]].rank(pct=True)
d["oraculo"] = d["alfa_verdadero"]
d["ridge"] = Ridge(alpha=10.0).fit(X[ent], d.loc[ent, "y"]).predict(X)
d["baseline"] = (rk["mom_12_1"] - rk["ret_1s"] + rk["valor_bp"]
                 + rk["sue"])
d["value"], d["ey_"], d["reversion"] = (d["valor_bp"], d["ey"],
                                        -d["ret_1s"])
senales = ["oraculo", "ridge", "baseline", "value", "ey_",
           "reversion"]
t = d[pru].reset_index(drop=True)
print(t["fecha"].nunique(), "weeks,", len(t), "rows")
OUT [02]The six signals of 2019-2025
364 weeks, 145600 rows

First, the classification metrics. Each signal's prediction is turned into “above the week's median” and compared with the binary label. Weighted accuracy weights each hit by the size of the move, ∣yi∣|y_i|: a model that gets the large moves right and misses the small ones comes out better than in plain accuracy. AUC uses the continuous signal and measures the probability that a stock from the good half has a larger signal than one from the bad half:

IN [03]python
from sklearn.metrics import roc_auc_score


def clasificacion(s, col):
    alta = s[col] > s.groupby("fecha")[col].transform("median")
    ok = alta.astype(int) == s["arriba"]
    return {"acierto": ok.mean(),
            "acierto_pond": (s["y"].abs() * ok).sum()
            / s["y"].abs().sum(),
            "auc": roc_auc_score(s["arriba"], s[col])}


pd.DataFrame({c: clasificacion(t, c) for c in senales}).T.round(4)
OUT [03]Accuracy, weighted accuracy and AUC
           acierto  acierto_pond     auc
oraculo     0.5077        0.5108  0.5124
ridge       0.5061        0.5106  0.5087
baseline    0.5056        0.5086  0.5089
value       0.5041        0.5062  0.5055
ey_         0.5034        0.5051  0.5046
reversion   0.5034        0.5073  0.5052

All the signals are between 50.3% and 50.8% accuracy: it is the scale of lesson 5.1.1. Weighted accuracy changes the order: reversal rises from 50.34% to 50.73% because it is right more often on large moves (a sharp fall in the previous week corrects more), and it overtakes value and ey. AUC ranks almost the same as accuracy. None of the three metrics says anything about how much it costs to trade each signal, which is what decides the result.

R² and squared error: scale versus order

A regression model is evaluated by default with R² or squared error. For ranking stocks, the scale of the prediction is irrelevant; for R², it is almost everything:

IN [04]python
from sklearn.metrics import r2_score


def ic_semanal(s, col, y="y"):
    r = s.groupby("fecha")[[col, y]].rank()
    r = r - r.groupby(s["fecha"]).transform("mean")
    num = (r[col] * r[y]).groupby(s["fecha"]).sum()
    den = ((r[col] ** 2).groupby(s["fecha"]).sum()
           * (r[y] ** 2).groupby(s["fecha"]).sum()) ** 0.5
    return num / den


t["oraculo_x5"] = 5 * t["oraculo"]
for c in ["oraculo", "oraculo_x5", "ridge"]:
    print(f"{c:11s} R² {r2_score(t['y'], t[c]):+.3%}   "
          f"IC {ic_semanal(t, c).mean():.4f}")
OUT [04]The same order, three R² values
oraculo     R² +0.044%   IC 0.0254
oraculo_x5  R² -2.531%   IC 0.0254
ridge       R² +0.040%   IC 0.0190

The oracle explains 0.044% of the label's variance. Multiplied by 5 it has exactly the same IC and would produce the same decile portfolio, but its R² falls to −2.5%: worse than always predicting zero. The ridge has an R² similar to the oracle's and a lower IC. R² is useful when the magnitude of the prediction is used as is (for sizing, for the volatility in lesson 5.8.5); for choosing between ranking signals, it misleads.

IC, ICIR and the fundamental law

The weekly IC measures the quality of the order; the ICIR, its mean divided by its standard deviation across weeks, measures its stability. Annualized (×52\times\sqrt{52}) it is comparable to a Sharpe ratio: if the portfolio weights each stock in proportion to its rank, its weekly return is roughly proportional to that week's IC times the dispersion of returns.

The fundamental law of active management (Grinold, 1989) gives a bound on the information ratio with NN independent bets per period:

\text{breadth} = N \times \text{periods per year}.$$ The cell checks it against the measured ICIR and computes the **effective breadth**: if the bets were independent, the standard deviation of the weekly IC would be $1/\sqrt{N}$; solving for $N$ from the measured standard deviation gives how many truly independent bets there are each week. :::cell{title="IC, ICIR, fundamental law and effective breadth"} ```python filas = {} for c in senales: ic = ic_semanal(t, c) n = t.groupby("fecha")[c].count().mean() filas[c] = {"ic": ic.mean(), "desv_ic": ic.std(), "icir_anual": ic.mean() / ic.std() * np.sqrt(52), "ley_fund": ic.mean() * np.sqrt(n * 52), "amplitud_efectiva": 1 / ic.std() ** 2} pd.DataFrame(filas).T.round(3) ``` ```output ic desv_ic icir_anual ley_fund amplitud_efectiva oraculo 0.025 0.100 1.829 3.668 99.437 ridge 0.019 0.088 1.562 2.737 130.271 baseline 0.018 0.082 1.567 2.560 149.781 value 0.010 0.072 1.041 1.495 194.029 ey_ 0.009 0.062 1.093 1.365 256.640 reversion 0.016 0.103 1.099 2.258 94.724 ``` ::: With 400 stocks, the fundamental law promises the oracle an information ratio of 3.7. The measured ICIR is 1.83: half. The standard deviation of the IC across weeks is 0.10, twice what 400 independent bets would give ($1/\sqrt{400} = 0.05$), and it is equivalent to about 100. The 400 stocks share the market, sectors and slow features: the breadth in the law is that of **independent decisions**, not of assets, and counting it wrong inflates any expectation. The ICIR already includes that dependence, which is why it is the measure of a signal's quality before costs. ## The signal's Sharpe after costs The measure that rules is the portfolio. The level's convention (data README): every week, long the top decile of the signal and short the bottom one, equal-weighted, with a gross notional of 2 (1 long and 1 short); simple returns; turnover = sum of the absolute weight changes; cost = 10 bp per side on turnover (lesson 2.4.7). The **breakeven cost** is the cost per side that brings the net return to zero: mean gross return divided by mean turnover. :::cell{title="Long-short decile portfolio: gross, turnover and net"} ```python def pesos_deciles(s, col): q = s.groupby("fecha")[col].rank(pct=True) w = pd.Series(np.select([q > 0.9, q <= 0.1], [1.0, -1.0], 0.0), index=s.index) w = 2 * w / w.abs().groupby(s["fecha"]).transform("sum") return (s[["fecha", "ticker"]].assign(w=w) .pivot(index="fecha", columns="ticker", values="w") .fillna(0.0)) R = t.pivot(index="fecha", columns="ticker", values="ret_sig_semana") def cartera(W, coste_pb=10): bruto = (W * R).sum(axis=1) rot = W.diff().abs().sum(axis=1) rot.iloc[0] = W.iloc[0].abs().sum() neto = bruto - coste_pb / 1e4 * rot sr = lambda x: x.mean() / x.std() * np.sqrt(52) return {"bruto_anual": bruto.mean() * 52, "sharpe_bruto": sr(bruto), "rotacion_sem": rot.mean(), "sharpe_neto": sr(neto), "equilibrio_pb": 1e4 * bruto.mean() / rot.mean()} W = {c: pesos_deciles(t, c) for c in senales} res = pd.DataFrame({c: cartera(W[c]) for c in senales}).T res["acierto"] = [clasificacion(t, c)["acierto"] for c in senales] res.round(3) ``` ```output bruto_anual sharpe_bruto rotacion_sem sharpe_neto equilibrio_pb acierto oraculo 0.167 1.667 2.649 0.287 12.089 0.508 ridge 0.129 1.404 2.541 -0.029 9.797 0.506 baseline 0.113 1.331 2.030 0.090 10.726 0.506 value 0.076 1.094 0.105 1.015 139.932 0.504 ey_ 0.061 0.889 0.102 0.810 114.183 0.503 reversion 0.133 1.295 3.420 -0.435 7.486 0.503 ``` ::: The figure at the top crosses the two extreme columns of the table: accuracy and net Sharpe. The order is reversed almost completely: - The **oracle** has the best accuracy (50.77%) and the best gross Sharpe (1.67), but its portfolio turns over 265% a week: it withstands 12 bp per side and with 10 bp it ends at 0.29. - The **ridge** and the **baseline** give gross Sharpes of 1.40 and 1.33 and net ones of −0.03 and 0.09: the same story with less signal. It is what the level README gives for 2015-2025. - The one-week **reversal** has a gross Sharpe of 1.30 and the worst turnover (342%): with 10 bp it loses money (−0.43). Its breakeven cost, 7 bp, is lower than the bid-ask spread of many real stocks. - **Value** and **ey** have some of the lowest accuracies (50.41% and 50.34%) and the **best net Sharpes** (1.01 and 0.81). Their portfolios change 10% a week: they withstand 140 and 114 bp per side. <aside class="callout" data-kind="aviso"> A net Sharpe of 1.01 with a single feature over seven years is not a recommendation: it is a result on synthetic data in which value has a planted effect. In real data, value is one of the best known and most studied anomalies (lesson 5.1.1: what gets published loses return), and seven years give a standard error of the Sharpe of about $1/\sqrt{7} \approx 0.38$. What is general is the mechanism: among similar signals, the one that turns over less keeps more of the gross return. </aside> ### Smoothing to turn over less If a signal has a slow part (lesson 5.1.1: the oracle keeps an IC of 0.011 with weeks of delay), you can trade some IC for much less turnover. The simplest way is an exponential average of the signal's ranks over the last few weeks, which only uses past information: :::cell{title="Smoothed signals: exponential average of the ranks"} ```python def suavizar(col, vida_media=2): p = d.pivot(index="fecha", columns="ticker", values=col) s = p.rank(axis=1, pct=True).ewm(halflife=vida_media).mean() return s.loc[t["fecha"].unique()] filas = {} for c in ["oraculo", "ridge", "baseline", "reversion"]: S = suavizar(c) larga = S.stack().rename("s").reset_index() ts = t[["fecha", "ticker"]].merge(larga, on=["fecha", "ticker"]) filas[c + "_suave"] = {**cartera(pesos_deciles(ts, "s")), "ic": ic_semanal(ts.assign(y=t["y"]), "s").mean()} pd.DataFrame(filas).T.round(3) ``` ```output bruto_anual sharpe_bruto rotacion_sem sharpe_neto equilibrio_pb ic oraculo_suave 0.120 1.184 1.016 0.661 22.658 0.016 ridge_suave 0.103 1.231 0.999 0.613 19.913 0.012 baseline_suave 0.091 1.202 0.626 0.771 27.878 0.013 reversion_suave 0.059 0.607 2.104 -0.514 5.417 0.008 ``` ::: ::figure{vega="n5-m1-costes-equilibrio"} With a half-life of two weeks, the oracle's IC drops from 0.025 to 0.016 and its gross Sharpe from 1.67 to 1.18, but turnover falls from 265% to 102% a week: the breakeven cost rises from 12 to 23 bp and the net Sharpe, from 0.29 to 0.66. The ridge goes from −0.03 to 0.61 and the baseline, from 0.09 to 0.77. Reversal, whose IC is all in the first week, has no slow part to keep. The figure draws the net Sharpe as a function of the cost per side: the lines of the fast signals fall quickly and cross zero between 7 and 12 bp; the smoothed ones and value hold up much longer. Optimizing the smoothing (the half-life, the decile thresholds) with the same test data is another experiment that counts for the Deflated Sharpe (lessons 2.5.4 and 5.4.4). Here a single value was tried, fixed in advance; lesson 4.1.5 treats turnover as a constraint of the optimization. ## Calibration: when the probability decides the size Some models are not used to rank but to decide **how much** to bet: the meta-labeling of lesson 5.3.3 sizes each trade with the secondary model's probability of success, and Kelly (lesson 4.2.2) bets $f = 2p - 1$ on a symmetric bet. There it is not enough for $p$ to rank well: it has to **mean** what it says. A model is calibrated if, of the times it says 0.6, it is right 60% of the time. The experiment: two classifiers that predict whether a stock will end above the median the following week, trained on 2012-2016. A logistic regression and a fairly flexible LightGBM (63 leaves, 300 trees). With 2017-2018, which neither has seen, the LightGBM is recalibrated with an **isotonic** regression (a monotonic function of its probability) through `CalibratedClassifierCV` and `FrozenEstimator`, which leaves it as is and only fits the recalibration. Everything is evaluated on 2019-2025: :::cell{title="Calibration of two classifiers and isotonic recalibration"} ```python import lightgbm as lgb from sklearn.calibration import CalibratedClassifierCV from sklearn.frozen import FrozenEstimator from sklearn.linear_model import LogisticRegression from sklearn.metrics import brier_score_loss, log_loss a = (d["fecha"] < "2017-01-01").to_numpy() c = ((d["fecha"] >= "2017-01-13") & (d["fecha"] < "2019-01-01")).to_numpy() y = d["arriba"] logit = LogisticRegression(C=1.0, max_iter=1000).fit(X[a], y[a]) gbm = lgb.LGBMClassifier( n_estimators=300, learning_rate=0.1, num_leaves=63, min_child_samples=20, random_state=0, n_jobs=4, deterministic=True, force_row_wise=True, verbose=-1, ).fit(X[a], y[a]) recal = CalibratedClassifierCV(FrozenEstimator(gbm), method="isotonic").fit(X[c], y[c]) yp, fiab, tabla = y[pru].to_numpy(), {}, {} for nombre, m in [("logistica", logit), ("lgbm", gbm), ("lgbm_recal", recal)]: p = m.predict_proba(X[pru])[:, 1] tramo = pd.qcut(p, 10, labels=False, duplicates="drop") fiab[nombre] = pd.DataFrame({"p": p, "obs": yp}).groupby( tramo).mean() tabla[nombre] = {"auc": roc_auc_score(yp, p), "brier": brier_score_loss(yp, p), "log_loss": log_loss(yp, p), "kelly_medio": np.abs(2 * p - 1).mean()} tabla["coin"] = {"auc": 0.5, "brier": 0.25, "log_loss": np.log(2), "kelly_medio": 0.0} print(pd.DataFrame(tabla).T.round(4)) print(fiab["lgbm"].T.round(3)) print(fiab["logistica"].T.round(3)) ``` ```output auc brier log_loss kelly_medio logistica 0.5060 0.2500 0.6932 0.0196 lgbm 0.5006 0.2546 0.7028 0.1008 lgbm_recal 0.5019 0.2500 0.6944 0.0091 coin 0.5000 0.2500 0.6931 0.0000 0 1 2 3 4 5 6 7 8 9 p 0.373 0.438 0.463 0.480 0.494 0.507 0.521 0.538 0.563 0.622 obs 0.497 0.499 0.499 0.499 0.500 0.500 0.505 0.511 0.493 0.497 0 1 2 3 4 5 6 7 8 9 p 0.478 0.487 0.492 0.495 0.499 0.502 0.505 0.509 0.513 0.520 obs 0.490 0.495 0.494 0.498 0.500 0.501 0.508 0.505 0.503 0.507 ``` ::: ::figure{vega="n5-m1-calibracion"} The reliability diagram (the figure) puts the mean predicted probability in each decile on the horizontal axis and the observed frequency on the vertical one; a calibrated model falls on the diagonal. The LightGBM predicts between 0.37 and 0.62 on average per decile and the observed frequency is 0.49-0.51 in all of them: it is **overconfident**. Its AUC (0.501) barely departs from 0.5, and its Brier score (0.2546) and log loss (0.7028) are **worse than a coin's** (0.25 and 0.6931). Sized with Kelly, it would bet on average 10% of capital per position with no edge at all. The logistic regression ranks somewhat better (AUC 0.506) and its probabilities move in a narrow range (0.48-0.52), and even so its log loss (0.6932) does not beat the coin's: the observed frequency only ranges from 0.49 to 0.51 and, with this signal-to-noise ratio, a per-stock probability other than 0.5 carries almost no information. Recalibrated, the LightGBM is compressed around 0.5 and its average Kelly size drops to 0.9%; its log loss stays slightly above the coin's (0.6944). Recalibration does not give it signal, but it removes the false confidence. <aside class="callout" data-kind="riesgo"> Recalibration needs data the model has not seen (here, 2017-2018) and has to be repeated when the model is retrained. Calibrating with the training data corrects nothing: an overfitted model is well calibrated on the data it memorized. And calibration is not stable if the relationship changes (lesson 5.10.2): a probability calibrated in calm periods may not be calibrated in a rebound. </aside> ## Which metric for which decision The metric is chosen by the use the prediction will be put to: ```mermaid flowchart TB U{What is it used for?} --> O[Rank assets] U --> T[Size positions] U --> F[Filter trades] O --> IC[IC and ICIR] T --> CA[Calibration] F --> PR[Precision and P&L] IC --> SN[Net Sharpe] CA --> SN PR --> SN SN --> DSR[Deflated Sharpe] ``` - **Rank assets** (cross-sectional portfolios): weekly IC, its standard deviation and ICIR to compare signals; Sharpe of the decile portfolio, gross and net, with its turnover and breakeven cost. - **Size positions** (meta-labeling, Kelly, volatility targeting): calibration (Brier score, log loss, reliability diagram) in addition to the order; if the prediction is a volatility, QLIKE or error in logs (lesson 5.8.5). - **Filter trades** from a primary system: precision of the trades let through and mean P&L per trade versus that of the unfiltered system (lesson 5.3.3). - **Always**: the net Sharpe of the complete strategy, with its standard error, and discounted for the number of experiments with the Deflated Sharpe (lessons 2.5.4 and 5.4.4). And the tails and the worst losing streak, which no Sharpe sees (lesson 2.6.1). During training you can optimize a statistical loss (squared error, cross-entropy), which is stable and differentiable; lesson 5.6.4 tries a Sharpe-based loss. But the **choice** between models, hyperparameters and variants is made with economic metrics, on data the model has not seen. ## Exercises 1. Compute the net Sharpe of the oracle's portfolio and of value's with costs of 0, 5, 10, 20 and 40 bp per side. From what cost does value beat the oracle? ::::solucion :::cell{title="Net Sharpe by cost per side"} ```python costes = [0, 5, 10, 20, 40] pd.DataFrame({c: [cartera(W[c], k)["sharpe_neto"] for k in costes] for c in ["oraculo", "value"]}, index=pd.Index(costes, name="coste_pb")).round(2) ``` ```output oraculo value coste_pb 0 1.67 1.09 5 0.98 1.05 10 0.29 1.01 20 -1.08 0.94 40 -3.78 0.78 ``` ::: Without costs, the oracle clearly wins (1.67 versus 1.09). With 5 bp per side value is already better (1.05 versus 0.98), and at 10 bp the difference is huge (1.01 versus 0.29). At 40 bp the oracle loses a lot (−3.78) and value barely notices the cost (0.78). The crossover is below 5 bp per side: less than it costs to trade a mid-cap stock in a real market. :::: 2. Repeat the IC and ICIR table with the `ret_sig_21d` label (21-session return) for the oracle and value. Why is it not advisable to compute their significance treating the weeks as if they were independent? ::::solucion :::cell{title="IC with the 21-session label"} ```python y21 = et[["fecha", "ticker", "ret_sig_21d"]] t21 = t.merge(y21, on=["fecha", "ticker"]).dropna( subset=["ret_sig_21d"]) for c in ["oraculo", "value"]: ic = ic_semanal(t21, c, "ret_sig_21d") ac1 = ic.autocorr(1) print(f"{c:8s} IC {ic.mean():.4f} annual ICIR " f"{ic.mean() / ic.std() * np.sqrt(52):.2f} " f"IC autocorrelation {ac1:.2f}") ``` ```output oraculo IC 0.0207 annual ICIR 1.61 IC autocorrelation 0.47 value IC 0.0209 annual ICIR 2.04 IC autocorrelation 0.76 ``` ::: With the 21-session label, the labels of consecutive weeks overlap by about three quarters, and one week's IC looks very much like the next one's: the first-order autocorrelation of the IC series measures it. The ICIR computed as if the weeks were independent overstates the significance; you need a standard error corrected for autocorrelation (Newey-West, lesson 2.1.1) or to use only non-overlapping weeks. :::: 3. A vendor offers you a model with “58% daily directional accuracy on the index”. What three questions would you ask before looking at anything else? :::solucion First: what is the frequency of up days in the same period? With an index's drift, “always up” already gives 55-57%, so 58% may add nothing; I would ask for the balanced accuracy or the comparison with that reference. Second: how much does it win or lose when it is right and when it is wrong? Accuracy does not see magnitudes: I would ask for the strategy's P&L, the accuracy weighted by the size of the move and the net Sharpe with costs and turnover. Third: how was it validated and how many variants were tried? I would ask for the out-of-sample period, the validation method (lessons 5.4.1-5.4.3) and the number of experiments for the Deflated Sharpe (5.4.4). :::

Each answer is folded under its question

Self-assessment

  1. A classifier that predicts “up” every day is right on 56.6% of the index's days in 2018-2025. What is it worth as a predictor?

    • AA lot: it is right more often than a 52% model
    • BIt depends on the horizon
    • CNothing: it is the frequency of up days (the index's drift); its balanced accuracy is exactly 50% and in this data the return is unpredictable by construction
    • DIt is a good baseline for trading
    Show answer

    Answer C. Nothing: it is the frequency of up days (the index's drift); its balanced accuracy is exactly 50% and in this data the return is unpredictable by construction

    Accuracy is compared with the frequency of the majority class, not with 50%. Balanced accuracy (the mean of the accuracy in each class) removes the effect: a constant classifier gives 0.5.

  2. In 2019-2025, value as a standalone signal has one of the worst accuracies of the six signals (50.41%) and the best net Sharpe (1.01). Why?

    • ABy chance
    • BBecause it turns over 10% a week versus 250-340% for the others: its breakeven cost is 140 bp per side, and with 10 bp it loses almost nothing
    • CBecause value has the highest IC
    • DBecause the accuracy is miscalculated
    Show answer

    Answer B. Because it turns over 10% a week versus 250-340% for the others: its breakeven cost is 140 bp per side, and with 10 bp it loses almost nothing

    Accuracy sees neither magnitude nor turnover. The oracle has more IC and a higher gross Sharpe (1.67), but its portfolio is renewed every week and only withstands 12 bp per side; with 10 bp it ends at 0.29.

  3. Multiplying the oracle's prediction by 5 changes its out-of-sample R² from +0.04% to −2.5%. What happens to its long-short decile portfolio?

    • ANothing: the order of the stocks is the same, so the IC and the portfolio are identical. R² penalizes scale, which does not matter in a rank-based portfolio
    • BIt loses money
    • CIt earns five times more
    • DIts turnover is multiplied by 5
    Show answer

    Answer A. Nothing: the order of the stocks is the same, so the IC and the portfolio are identical. R² penalizes scale, which does not matter in a rank-based portfolio

    R² measures the squared error of the prediction in its own units. To rank stocks only the order counts; to size positions the scale does count, and then what you look at is calibration.

  4. The oracle's weekly IC is 0.025 and there are 400 stocks. The fundamental law gives an information ratio of 0.025·√(400·52) ≈ 3.7, but the measured annual ICIR is 1.8. Where does the difference come from?

    • AFrom a calculation error
    • BFrom costs
    • CFrom the IC being rank-based
    • DFrom the 400 weekly bets not being independent: they share the market, sectors and features. The standard deviation of the IC across weeks (≈ 0.1) corresponds to about 100 independent bets per week, not 400
    Show answer

    Answer D. From the 400 weekly bets not being independent: they share the market, sectors and features. The standard deviation of the IC across weeks (≈ 0.1) corresponds to about 100 independent bets per week, not 400

    With N independent bets the standard deviation of the IC would be ≈ 1/√N = 0.05. The measured one is twice that, so the effective breadth is a quarter. The breadth in the fundamental law is that of independent decisions, not of assets.

  5. A LightGBM gives probabilities that a stock ends above the median between 0.31 and 0.68; in those buckets the observed frequency is always ≈ 0.50. If positions are sized with Kelly (f = 2p − 1), what happens?

    • ANothing: the AUC is greater than 0.5
    • BToo little is bet
    • CThe right amount is bet
    • DOn average 10% of capital is bet per position with no real edge: the model is overconfident, its Brier score and log loss are worse than a coin's (0.2546 versus 0.25)
    Show answer

    Answer D. On average 10% of capital is bet per position with no real edge: the model is overconfident, its Brier score and log loss are worse than a coin's (0.2546 versus 0.25)

    Uncalibrated probabilities turn noise into size. Recalibrated with 2017-2018 (isotonic), the probabilities are compressed around 0.5 and the average Kelly size drops to 0.9%.

Key points

  • Accuracy is compared with the majority class and does not see magnitudes: accuracy weighted by |return| and balanced accuracy partly correct this.
  • The weekly IC (rank correlation) measures what matters for ranking; its mean divided by its standard deviation, annualized (ICIR), resembles the gross Sharpe of the long-short portfolio.
  • The fundamental law IR ≈ IC·√breadth uses independent bets: with 400 correlated stocks, the measured effective breadth is about 100 per week.
  • The final metric is the portfolio's net Sharpe: the oracle goes from 1.67 gross to 0.29 net with 10 bp per side, and value, with worse accuracy, gives 1.01.
  • The breakeven cost (gross return / turnover) says how many bp per side a signal can withstand; smoothing raises the oracle's from 12 to 23 bp and its net Sharpe from 0.29 to 0.66.
  • If the probability decides the size, calibration is required (Brier score, log loss, reliability diagram): an overconfident model bets without an edge.

Resources

Research

  1. Alpha Factors and Alpha Trading in PythonResearch-0007 · 5 min

Documentation

Books

Data

Back to level 5