Why financial machine learning is different.

What changes compared with ML in other fields

The techniques of this level (trees, boosting, neural networks, transformers) come from fields where they work very well: classifying images, translating text, recommending products. A cat is a cat today and in ten years, the label is in the image and there are millions of examples. None of those three things is true when you predict a return.

The practical consequence is that almost the whole method changes: how you label (module 5.3), how you validate (5.4), what you measure (5.1.3) and which result is considered credible. This lesson measures each difference on data where the answer is known.

The ceiling: an oracle that is barely right

The level's test bench is panel5: 400 fictitious stocks, 730 weeks from 2012 to 2025, 22 features known at each Friday's close and the following week's return. It is synthetic data with a generating model published in the README: the extra expected return of each stock and week, alfa_verdadero, comes from a set of planted effects (momentum, reversal, value, earnings surprise, low volatility, sentiment) whose size was chosen to mimic that of a real market. That column is the oracle: the best possible prediction with the information available each Friday. No model can do better out of sample.

import numpy as np
import pandas as pd

f = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
ver = pd.read_parquet("panel5_verdad.parquet")
d = (f.merge(et[["fecha", "ticker", "ret_sig_semana"]],
             on=["fecha", "ticker"])
      .merge(ver[["fecha", "ticker", "alfa_verdadero"]],
             on=["fecha", "ticker"])
      .dropna(subset=["ret_sig_semana"]))
print(d.shape, d["fecha"].nunique(), "weeks")
d[["fecha", "ticker", "mom_12_1", "alfa_verdadero",
   "ret_sig_semana"]].head(3)

(291600, 27) 729 weeks
       fecha ticker  mom_12_1  alfa_verdadero  ret_sig_semana
0 2012-01-13   BAGO   -0.2174       -0.000268       -0.059579
1 2012-01-13   BAKA    0.0510       -0.000103       -0.024737
2 2012-01-13   BAKU   -0.2482       -0.001394       -0.072296

The natural measure of a cross-sectional prediction is the information coefficient (IC): the rank correlation (Spearman) between the prediction and the realized return, computed every week across the 400 stocks. Ranks are used because what matters for building a portfolio is the order, and because returns have fat tails that would dominate a Pearson correlation. Lesson 5.1.3 develops it; here the function is enough:

def ic_semanal(df, col, y="ret_sig_semana"):
    """Rank correlation of col with y, week by week."""
    s = df[["fecha", col, y]].dropna()
    r = s.groupby("fecha")[[col, y]].rank()
    r = r - r.groupby(s["fecha"]).transform("mean")
    num = (r[col] * r[y]).groupby(s["fecha"]).sum()
    den = ((r[col] ** 2).groupby(s["fecha"]).sum()
           * (r[y] ** 2).groupby(s["fecha"]).sum()) ** 0.5
    return num / den


def resumen(ic):
    return pd.Series({"ic_medio": ic.mean(), "desv": ic.std(),
                      "t": ic.mean() / ic.std() * np.sqrt(len(ic)),
                      "semanas_pos": (ic > 0).mean()})


ic_or = ic_semanal(d, "alfa_verdadero")
resumen(ic_or).round(4)

ic_medio       0.0292
desv           0.0929
t              8.4714
semanas_pos    0.6461
dtype: float64

The perfect model has a mean IC of 0.029. With 729 weeks, that is highly significant (t = 8.5), but week by week the standard deviation of the IC (0.093) is three times its mean: in 35% of weeks the oracle ranks the stocks worse than chance. The figure at the top draws the distribution: a bell centered almost at zero, with the mean barely shifted to the right.

Hit rate and explained variance tell the same story in other words:

med = d.groupby("fecha")[["ret_sig_semana", "alfa_verdadero"]]
dev = d[["ret_sig_semana", "alfa_verdadero"]] - med.transform(
    "median")
acierto = ((dev["ret_sig_semana"] > 0)
           == (dev["alfa_verdadero"] > 0)).mean()
print(f"Hit rate against the median: {acierto:.2%}")
print(f"Explained variance (IC²): {ic_or.mean() ** 2:.3%}")
disp = d.groupby("fecha")["ret_sig_semana"].std().mean()
sd_alfa = d.groupby("fecha")["alfa_verdadero"].std().mean()
print(f"Weekly dispersion: {disp:.2%}; "
      f"alpha std: {sd_alfa * 1e4:.1f} bp")

Hit rate against the median: 50.95%
Explained variance (IC²): 0.085%
Weekly dispersion: 3.62%; alpha std: 12.3 bp

A 50.95% hit rate and 0.09% of the variance. The dispersion of returns across stocks in a typical week is 3.6%; that of the expected return, 12 basis points: thirty times less. For comparison, an acceptable image classifier is right more than 95% of the time. In finance, a model that is right 55% of the time consistently on a panel like this one is not good: it is impossible.

Grouping by oracle deciles shows where the signal is and how much noise surrounds it:

g = d.groupby("fecha")
exceso = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean")
decil = g["alfa_verdadero"].transform(
    lambda s: pd.qcut(s, 10, labels=False))
(exceso.groupby(decil).agg(["mean", "std"]) * 1e4).round(1)

                mean    std
alfa_verdadero
0              -14.9  424.3
1              -13.4  384.3
2               -3.9  364.8
3               -1.6  363.0
4               -0.4  354.4
5               -1.4  350.7
6                2.3  349.6
7                6.2  345.7
8                7.2  345.3
9               19.9  369.1

The relationship is monotonic: the top decile returns about 20 bp per week more than the mean and the bottom one about 15 bp less. But within each decile the standard deviation is between 345 and 424 bp. The difference between the extremes, about 35 bp, is one tenth of the noise of a single stock in a single week. That is why a financial prediction is not evaluated with one case or one week, but with many aggregated bets, and why the portfolio, not the prediction, is the unit of measurement (lesson 5.1.3).

Scarce data: 291,600 rows are not 291,600 tests

The panel has almost 300,000 labeled rows. For an ML model they look like a lot; for inference they are far fewer, for two reasons. The first: slow features barely change from one week to the next, so a stock's row in week 300 almost repeats the information in week 299. The rank correlation of each feature with itself one week later measures it:

def persistencia(col):
    p = d.pivot(index="fecha", columns="ticker", values=col)
    p = p.rank(axis=1)
    return p.corrwith(p.shift(1), axis=1).mean()


cols = ["valor_bp", "mom_12_1", "vol_63d", "sue", "ruido_3",
        "ret_1s", "ruido_1", "alfa_verdadero"]
pd.Series({c: persistencia(c) for c in cols}).round(3)

valor_bp          0.999
mom_12_1          0.975
vol_63d           0.977
sue               0.941
ruido_3           0.977
ret_1s            0.019
ruido_1          -0.001
alfa_verdadero    0.515
dtype: float64

Value (valor_bp) has a persistence of 0.999: over 14 years, the ranking of stocks by value changes so slowly that, for practical purposes, it provides a few dozen independent observations of its effect, not hundreds of thousands. Momentum, volatility and SUE are between 0.94 and 0.98. Only the one-week reversal (ret_1s) and the i.i.d. noise renew every week. And ruido_3, an AR(1) with ρ = 0.98 and no effect at all, is as persistent as momentum: a slow feature with no effect can correlate with returns for years by pure chance.

The second reason is that all the stocks in a week share the same market and the same sectors: a week's noise does not cancel out across stocks as much as it would if they were independent. Together, the two effects mean that tests with 14 years of data have less power than the number of rows suggests. The IC table of the 22 features, alongside what the generating model says about each one, puts it to the test:

efecto = {"ret_1s": "yes", "mom_12_1": "yes", "vol_63d": "yes",
          "valor_bp": "yes", "sue": "yes", "sentimiento": "yes",
          "volumen_rel": "interaction",
          "dias_desde_informe": "interaction",
          "revision_analistas": "LEAK"}
feats = [c for c in f.columns
         if c not in ("fecha", "ticker", "sector")]
tabla = pd.DataFrame({c: resumen(ic_semanal(d, c)) for c in feats}).T
tabla["efecto"] = [efecto.get(c, "no") for c in tabla.index]
tabla.sort_values("t")[["ic_medio", "t", "efecto"]].round(4)

                    ic_medio        t       efecto
ret_1s               -0.0159  -4.2881          yes
rsi_14               -0.0083  -2.6280           no
ret_1m               -0.0077  -2.1404           no
ret_3m               -0.0050  -1.5616           no
ruido_1              -0.0024  -1.2751           no
dist_max_52s         -0.0031  -0.9992           no
volumen_rel          -0.0014  -0.7362  interaction
ruido_3              -0.0010  -0.5558           no
ruido_5              -0.0008  -0.4661           no
dias_desde_informe   -0.0005  -0.2746  interaction
log_capitalizacion   -0.0005  -0.2103           no
vol_21d               0.0020   0.3569           no
vol_63d               0.0032   0.5593          yes
ruido_2               0.0015   0.7851           no
ruido_4               0.0044   1.8392           no
mom_12_1              0.0077   2.4404          yes
roe                   0.0057   2.7132           no
sentimiento           0.0066   3.0800          yes
sue                   0.0079   3.7415          yes
valor_bp              0.0095   3.9281          yes
ey                    0.0087   4.0399           no
revision_analistas    0.1743  92.1366         LEAK

Three readings of the table:

• True effects that do not show. vol_63d has a planted effect and its IC gives t = 0.6: low volatility is masked because volatile stocks have more beta and the market rises on average (the README measures it with Fama-MacBeth). sentimiento and sue, which have an effect, rank below ey.
• Features with no effect of their own that look good. ey (t = 4.0) and roe (t = 2.7) inherit the effect of value and SUE, with which they are correlated; rsi_14 (t = −2.6) and ret_1m (t = −2.1), that of reversal. A univariate test does not distinguish cause from resemblance. The multiple testing corrections of lesson 2.1.8 account for the number of attempts, but not for this confusion.
• The leak. revision_analistas gives an IC of 0.174 with t = 92: six times the oracle's ceiling. It is not a great feature: the vendor dates it at the start of the week it covers and it contains the first two days of the return being predicted. A result above the ceiling is a bug, not a finding (lesson 5.2.2 teaches how to audit the dates).

A simple calculation measures the scarcity of data. If a signal has a mean IC $\overline{IC}$ with weekly standard deviation $s$, the t-statistic after $n$ weeks is $\overline{IC}\sqrt{n}/s$, and reaching t = 2 takes $n = (2s/\overline{IC})^2$ weeks. For a strategy with annual Sharpe $\mathrm{SR}$, the years needed are $(2/\mathrm{SR})^2$:

sem = tabla.loc[["valor_bp", "sue", "sentimiento", "mom_12_1"],
                ["ic_medio", "desv"]].astype(float)
sem.columns = ["ic", "desv"]
sem.loc["oracle"] = [ic_or.mean(), ic_or.std()]
sem["años_para_t2"] = (2 * sem["desv"] / sem["ic"]) ** 2 / 52
print(sem.round(3))
sharpe = pd.Series([0.3, 0.5, 1.0, 2.0], name="sharpe_anual")
print((2 / sharpe) ** 2)

                ic   desv  años_para_t2
valor_bp     0.009  0.065         3.634
sue          0.008  0.057         4.006
sentimiento  0.007  0.058         5.911
mom_12_1     0.008  0.085         9.416
oracle       0.029  0.093         0.781
0    44.444444
1    16.000000
2     4.000000
3     1.000000
Name: sharpe_anual, dtype: float64

The oracle, which combines all the effects, stands out from the noise in less than a year. A single feature with an IC of 0.008 or 0.009 needs three and a half to six years, and momentum, whose IC varies more across weeks, more than nine. And these are lower bounds: the formula treats weeks as independent, and with features this persistent they are not. A strategy with an annual Sharpe of 0.5 (good in the real world) needs 16 years of history to give t = 2, and that assumes the relationship does not change in those 16 years. Which is the next difference.

Non-stationarity: the relationship changes

In computer vision, the function that maps pixels to the label is fixed. In panel5, the coefficient of each effect changes every week: it is its base value multiplied by a regime factor and by an AR(1) term that weakens it and sometimes reverses it. In addition, momentum is multiplied by −2.5 in rebounds after a crisis: this is the momentum crash, documented in real markets (the stocks that fell the most are the ones that rebound the most). The file panel5_coef_verdad.csv gives the true coefficient for each week:

coef = pd.read_csv("panel5_coef_verdad.csv", parse_dates=["fecha"])
coef = coef.set_index("fecha")
c_mom = coef["c_mom"] * 1e4          # bp per week and unit
print(c_mom.groupby(coef["regimen"]).agg(["mean", "count"])
      .round(2))
ic_mom = ic_semanal(d, "mom_12_1")
anual = pd.DataFrame({"c_mom_pb": c_mom, "ic_mom": ic_mom})
anual.groupby(anual.index.year).mean().round(4)

          mean  count
regimen
calma     4.71    576
estres    2.02     89
rebote  -10.44     65
       c_mom_pb  ic_mom
fecha
2012     4.9549  0.0133
2013     5.6347  0.0175
2014     5.5424  0.0207
2015     3.4455  0.0153
2016     0.7704 -0.0134
2017     6.0316  0.0188
2018     4.5862  0.0050
2019     0.7888 -0.0069
2020    -2.0040 -0.0101
2021     4.2767  0.0003
2022    -1.4078  0.0177
2023     4.0448  0.0057
2024     3.8732  0.0134
2025     2.0747  0.0107

In calm periods (576 weeks), momentum returns 4.7 bp per week and unit; in stress, 2.0; in rebounds, −10.4. The year 2020 has a negative mean coefficient (−2.0) because of the March to June rebound, and so does 2022 (−1.4). The measured momentum IC follows the coefficient with noise: it is negative in 2016, 2019 and 2020, years with a low or negative coefficient, but in 2022, with a negative true coefficient, it comes out positive (0.018): one year of data is not even enough to get the sign right. A model trained on 2012-2015 learns a “good” momentum that does damage in 2016 and in 2020, and nothing in the training data warns it.

There are two kinds of change and it pays to separate them from now on. Data drift changes the distribution of the features (a feature shifts or spreads out), and concept drift changes the relationship between features and label, as here. The second is the dangerous one, because it cannot be seen by looking at the features: it only shows up in the results, late. Lesson 5.10.2 detects it with data where the date of each change is known; rolling versus expanding retraining windows are covered in 5.4.3.

Overfitting: a flexible model learns the noise

With little signal, a lot of noise and changing relationships, a flexible model easily finds patterns in the noise. The experiment: three models with the 21 features without the leak, transformed to cross-sectional rank (lesson 5.2.4), trained on 2012-2018 and evaluated on 2019-2025. The label is the next week's return minus its cross-sectional mean. A ridge (linear and regularized), a moderate LightGBM (8 leaves, at least 2,000 rows per leaf) and a flexible LightGBM (512 leaves, 5 rows per leaf, 400 trees). Training is deliberately short: the flexible one takes about 15 seconds on CPU.

import lightgbm as lgb
from sklearn.linear_model import Ridge

feats_x = [c for c in feats if c != "revision_analistas"]
X = d.groupby("fecha")[feats_x].rank(pct=True).sub(0.5).fillna(0)
y = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean")
d["y"] = y
ent = d["fecha"] < "2019-01-01"
pru = d["fecha"] >= "2019-01-08"     # one-week gap

comun = dict(random_state=0, n_jobs=4, deterministic=True,
             force_row_wise=True, verbose=-1)
modelos = {
    "ridge": Ridge(alpha=10.0),
    "lgbm_moderado": lgb.LGBMRegressor(
        n_estimators=200, learning_rate=0.02, num_leaves=8,
        min_child_samples=2000, subsample=0.5, subsample_freq=1,
        **comun),
    "lgbm_flexible": lgb.LGBMRegressor(
        n_estimators=400, learning_rate=0.1, num_leaves=512,
        min_child_samples=5, **comun),
}


def ic_medio(pred, mascara):
    s = d.loc[mascara, ["fecha", "y"]].assign(p=pred)
    return ic_semanal(s, "p", "y").mean()


filas = {"oracle": [ic_medio(d.loc[ent, "alfa_verdadero"], ent),
                    ic_medio(d.loc[pru, "alfa_verdadero"], pru)]}
for nombre, m in modelos.items():
    m.fit(X[ent], y[ent])
    filas[nombre] = [ic_medio(m.predict(X[ent]), ent),
                     ic_medio(m.predict(X[pru]), pru)]
pd.DataFrame(filas, index=["ic_2012_2018", "ic_2019_2025"]).T.round(4)

               ic_2012_2018  ic_2019_2025
oracle               0.0327        0.0254
ridge                0.0258        0.0190
lgbm_moderado        0.0658        0.0141
lgbm_flexible        0.9626       -0.0022

The oracle marks the ceiling of each period: 0.033 in training and 0.025 in test. The ridge stays below it in both (0.026 and 0.019): it learns little and what it learns holds. The moderate LightGBM gives 0.066 in sample, twice the ceiling, and 0.014 out of sample: it has learned some signal and quite a lot of noise. The flexible one gives 0.963 in sample (it ranks almost perfectly returns that are 99.9% noise) and −0.002 out of sample. It is not that boosting is useless: lesson 5.5.3 tunes it with honest validation. It is that in this data, uncontrolled flexibility turns into memorized noise long before it turns into signal.

This yields a sanity rule used throughout the level. If you know, even roughly, the most a signal can deliver:

flowchart TB
  R[Result] --> A{In-sample IC far above the ceiling?}
  A -- yes --> S[Overfitting]
  A -- no --> B{Out-of-sample IC above the ceiling?}
  B -- yes --> F[Leak or bug]
  B -- no --> C{Profitable after costs?}
  C -- no --> N[Worthless signal]
  C -- yes --> V[Validate further: 5.4]

In real data there is no oracle, but there are orders of magnitude: the R² in the literature, the Sharpe of the best known strategies, the IC of published anomalies. A result far above that scale is investigated as a bug before it is celebrated.

Competition: the signal expires and gets shared

The last difference is that the market reacts. If a signal makes money, others find it, trade it and use it up. A first effect shows up in the data: the signal's horizon is short, and whoever exploits it late gets less. The cell uses each week's oracle to predict the following week's return, as always, and also the return one, two, three and four weeks later:

a = d.pivot(index="fecha", columns="ticker",
            values="alfa_verdadero").rank(axis=1)
r = d.pivot(index="fecha", columns="ticker",
            values="ret_sig_semana")
retraso = {}
for k in range(5):
    ic_k = a.corrwith(r.shift(-k).rank(axis=1), axis=1).dropna()
    retraso[k] = resumen(ic_k)
print(pd.DataFrame(retraso).T.rename_axis("semanas_tarde").round(4))
for col in ("ret_1s", "valor_bp"):
    x = d.pivot(index="fecha", columns="ticker",
                values=col).rank(axis=1)
    ics = [x.corrwith(r.shift(-k).rank(axis=1), axis=1).mean()
           for k in range(4)]
    print(col, np.round(ics, 4))

               ic_medio    desv       t  semanas_pos
semanas_tarde
0                0.0292  0.0929  8.4714       0.6461
1                0.0110  0.0900  3.2974       0.5701
2                0.0111  0.0875  3.4129       0.5695
3                0.0105  0.0907  3.1194       0.5565
4                0.0075  0.0875  2.2963       0.5462
ret_1s [-0.0159 -0.0001  0.0004 -0.0012]
valor_bp [0.0095 0.0089 0.0087 0.0087]

With a one-week delay, the IC drops from 0.029 to 0.011: almost two thirds are lost. What remains does not run out quickly (0.011 and 0.010 with two and three weeks of delay) because it is the slow part of the signal: value, momentum and SUE change slowly, and the expected return three weeks out resembles next week's. The last two lines separate them: the value IC barely moves with the delay (from 0.0095 to 0.0087), and that of the one-week reversal (ret_1s, −0.016) disappears completely with the first delay. In a real market, that split is the result of competition: fast signals are collected by those who arrive first, and a weekly system is left with the slow part.

On a scale of years, the effect was measured by McLean and Pontiff (2016) with 97 variables published in the academic literature that predicted returns: the return of their portfolios is 26% lower outside the original paper's sample and 58% lower after publication. The authors read the 26% as an upper bound on the part due to data mining and the difference (32 points) as the effect of investors who learn from the publication. For ML, the lesson is twofold: what worked in the documented past is what most competitors know, and a model that “discovers” a known anomaly has discovered nothing.

What changes in the method

The four differences translate into a workflow that is the backbone of the level:

flowchart TB
  D[Point-in-time data] --> E[Labels and weights]
  E --> V[Purged validation]
  V --> M[Model and baseline]
  M --> K[Economic metrics]
  K --> B[Backtest and DSR]
  B --> W[Monitor drift]

• Point-in-time data (5.2.2): each feature, with the date on which it is known. The revision_analistas leak is not detected by any validation; only by auditing the dates.
• Labels and weights (5.3): labels that overlap in time mean fewer independent samples; uniqueness and the sequential bootstrap correct for it.
• Purged validation (5.4.1-5.4.3): random k-fold mixes neighbors in time and promises accuracy that does not exist.
• Model and baseline (5.5-5.8): each model is compared with a simple reference (a ridge, a single feature, HAR for volatility) and with the ceiling when it is known.
• Economic metrics (5.1.3) and backtest with Deflated Sharpe (5.4.4): a model is judged by the portfolio it produces, after costs, and discounting how many experiments were run (lesson 2.5.4).
• Monitor drift (5.10.2): the relationship will change; you need to know when.

Exercises

1. Compute the oracle's mean IC by regime (calma, estres, rebote) with the regimen column of panel5_coef_verdad.csv. In which regime is it hardest to predict even when the expected return is known?

reg = coef["regimen"].reindex(ic_or.index)
ic_or.groupby(reg).agg(["mean", "std", "count"]).round(4)

           mean     std  count
regimen
calma    0.0286  0.0889    575
estres   0.0301  0.0948     89
rebote   0.0330  0.1223     65

2. With the formula $n = (2s/\overline{IC})^2$, compute how many weeks a signal with the ridge's mean out-of-sample IC (0.019) and a weekly standard deviation of 0.09 would need to give t = 2, and how many for t = 3, the threshold proposed by the multiple testing literature (lesson 2.1.8).

ic_r, s_r = 0.019, 0.09
for t_obj in (2, 3):
    n = (t_obj * s_r / ic_r) ** 2
    print(f"t = {t_obj}: {n:.0f} weeks ({n / 52:.1f} years)")

t = 2: 90 weeks (1.7 years)
t = 3: 202 weeks (3.9 years)

3. A colleague presents a classifier trained on panel5 that predicts whether a stock will end above the median 21 sessions ahead, with a hit rate of 59.7% in 5-fold shuffled cross-validation. Explain why you do not believe it and what you would ask for.