--- title: "5.1.1 Why financial machine learning is different" description: "Little signal, a lot of noise, relationships that change and competitors that learn: the ceiling of a model measured on a panel with a known answer." url: "https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/why-financial-machine-learning-is-different.png" --- # 5.1.1 · Why financial machine learning is different > In computer vision a model is right 99% of the time; in a stock panel, the perfect model is right 51% of the time. This lesson measures that ceiling with an oracle that knows the true expected return and shows the four differences that force a change of method: the signal-to-noise ratio, the scarcity of independent data, non-stationarity and competition. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.1 · ML fundamentals applied to finance - Duration: 60 min - Prerequisites: [0.7.2 · Probability and distributions, correlation versus causation and simple regression](https://pythonparatrading.com/en/python-trading-course/level-0/probability-distributions-and-regression/), [2.1.8 · The multiple testing problem: Bonferroni, FDR and the “factor zoo”](https://pythonparatrading.com/en/python-trading-course/level-2/multiple-testing-bonferroni-fdr-factor-zoo/), [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/), [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/) ## By the end of this lesson - Measure the predictability ceiling of a panel with an oracle: weekly IC, hit rate, explained variance and how they vary across weeks. - Explain why hundreds of thousands of rows are little independent data and compute how many years it takes to tell a signal from noise. - Show with the true coefficients that the relationship between features and returns changes with the regime and over time. - Recognize overfitting and leakage by comparing in-sample and out-of-sample IC with the oracle's. - Quantify how much of the signal is lost by arriving late and relate it to competition and to the decay of published anomalies. ## What changes compared with ML in other fields The techniques of this level (trees, boosting, neural networks, transformers) come from fields where they work very well: classifying images, translating text, recommending products. A cat is a cat today and in ten years, the label is in the image and there are millions of examples. None of those three things is true when you predict a return. | Trait | Vision, language | Financial returns | |---|---|---| | Signal-to-noise ratio | High: the label is almost in the data | Extremely low: the perfect model explains tenths of a point of the variance | | Stationarity | The phenomenon does not change | Relationships change with the regime and disappear once exploited | | Data | Millions of independent examples; more can be generated | A single history; observations correlated over time and across assets | | Adversary | None: the cat does not react to the classifier | Other participants compete for the same signal and use it up | | Cost of error | Symmetric, measured with accuracy or loss | Asymmetric and economic: costs, tails, capacity | The practical consequence is that almost the whole method changes: how you label (module 5.3), how you validate (5.4), what you measure (5.1.3) and which result is considered credible. This lesson measures each difference on data where the answer is known. ## The ceiling: an oracle that is barely right The level's test bench is `panel5`: 400 fictitious stocks, 730 weeks from 2012 to 2025, 22 features known at each Friday's close and the following week's return. It is **synthetic** data with a generating model published in the README: the extra expected return of each stock and week, `alfa_verdadero`, comes from a set of planted effects (momentum, reversal, value, earnings surprise, low volatility, sentiment) whose size was chosen to mimic that of a real market. That column is the **oracle**: the best possible prediction with the information available each Friday. No model can do better out of sample. ```python import numpy as np import pandas as pd f = pd.read_parquet("panel5_features.parquet") et = pd.read_parquet("panel5_etiquetas.parquet") ver = pd.read_parquet("panel5_verdad.parquet") d = (f.merge(et[["fecha", "ticker", "ret_sig_semana"]], on=["fecha", "ticker"]) .merge(ver[["fecha", "ticker", "alfa_verdadero"]], on=["fecha", "ticker"]) .dropna(subset=["ret_sig_semana"])) print(d.shape, d["fecha"].nunique(), "weeks") d[["fecha", "ticker", "mom_12_1", "alfa_verdadero", "ret_sig_semana"]].head(3) ``` ```output (291600, 27) 729 weeks fecha ticker mom_12_1 alfa_verdadero ret_sig_semana 0 2012-01-13 BAGO -0.2174 -0.000268 -0.059579 1 2012-01-13 BAKA 0.0510 -0.000103 -0.024737 2 2012-01-13 BAKU -0.2482 -0.001394 -0.072296 ``` The natural measure of a cross-sectional prediction is the **information coefficient (IC)**: the rank correlation (Spearman) between the prediction and the realized return, computed every week across the 400 stocks. Ranks are used because what matters for building a portfolio is the order, and because returns have fat tails that would dominate a Pearson correlation. Lesson 5.1.3 develops it; here the function is enough: ```python def ic_semanal(df, col, y="ret_sig_semana"): """Rank correlation of col with y, week by week.""" s = df[["fecha", col, y]].dropna() r = s.groupby("fecha")[[col, y]].rank() r = r - r.groupby(s["fecha"]).transform("mean") num = (r[col] * r[y]).groupby(s["fecha"]).sum() den = ((r[col] ** 2).groupby(s["fecha"]).sum() * (r[y] ** 2).groupby(s["fecha"]).sum()) ** 0.5 return num / den def resumen(ic): return pd.Series({"ic_medio": ic.mean(), "desv": ic.std(), "t": ic.mean() / ic.std() * np.sqrt(len(ic)), "semanas_pos": (ic > 0).mean()}) ic_or = ic_semanal(d, "alfa_verdadero") resumen(ic_or).round(4) ``` ```output ic_medio 0.0292 desv 0.0929 t 8.4714 semanas_pos 0.6461 dtype: float64 ``` The perfect model has a mean IC of 0.029. With 729 weeks, that is highly significant (t = 8.5), but week by week the standard deviation of the IC (0.093) is three times its mean: in 35% of weeks the oracle ranks the stocks **worse than chance**. The figure at the top draws the distribution: a bell centered almost at zero, with the mean barely shifted to the right. Hit rate and explained variance tell the same story in other words: ```python med = d.groupby("fecha")[["ret_sig_semana", "alfa_verdadero"]] dev = d[["ret_sig_semana", "alfa_verdadero"]] - med.transform( "median") acierto = ((dev["ret_sig_semana"] > 0) == (dev["alfa_verdadero"] > 0)).mean() print(f"Hit rate against the median: {acierto:.2%}") print(f"Explained variance (IC²): {ic_or.mean() ** 2:.3%}") disp = d.groupby("fecha")["ret_sig_semana"].std().mean() sd_alfa = d.groupby("fecha")["alfa_verdadero"].std().mean() print(f"Weekly dispersion: {disp:.2%}; " f"alpha std: {sd_alfa * 1e4:.1f} bp") ``` ```output Hit rate against the median: 50.95% Explained variance (IC²): 0.085% Weekly dispersion: 3.62%; alpha std: 12.3 bp ``` A 50.95% hit rate and 0.09% of the variance. The dispersion of returns across stocks in a typical week is 3.6%; that of the expected return, 12 basis points: thirty times less. For comparison, an acceptable image classifier is right more than 95% of the time. In finance, a model that is right 55% of the time consistently on a panel like this one is not good: it is impossible. Grouping by oracle deciles shows where the signal is and how much noise surrounds it: ```python g = d.groupby("fecha") exceso = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean") decil = g["alfa_verdadero"].transform( lambda s: pd.qcut(s, 10, labels=False)) (exceso.groupby(decil).agg(["mean", "std"]) * 1e4).round(1) ``` ```output mean std alfa_verdadero 0 -14.9 424.3 1 -13.4 384.3 2 -3.9 364.8 3 -1.6 363.0 4 -0.4 354.4 5 -1.4 350.7 6 2.3 349.6 7 6.2 345.7 8 7.2 345.3 9 19.9 369.1 ``` *Fig. — Next-week return relative to the panel mean by oracle decile, in basis points, 2012-2025.* Ten bars rising from decile 1 to 10: from −14.9 bp to +19.9 bp per week (decile 10, highlighted). A label notes that the standard deviation within each decile is 345 to 424 bp, about twenty times the gap between means. The relationship is monotonic: the top decile returns about 20 bp per week more than the mean and the bottom one about 15 bp less. But within each decile the standard deviation is between 345 and 424 bp. The difference between the extremes, about 35 bp, is one tenth of the noise of a single stock in a single week. That is why a financial prediction is not evaluated with one case or one week, but with many aggregated bets, and why the portfolio, not the prediction, is the unit of measurement (lesson 5.1.3). ## Scarce data: 291,600 rows are not 291,600 tests The panel has almost 300,000 labeled rows. For an ML model they look like a lot; for inference they are far fewer, for two reasons. The first: slow features barely change from one week to the next, so a stock's row in week 300 almost repeats the information in week 299. The rank correlation of each feature with itself one week later measures it: ```python def persistencia(col): p = d.pivot(index="fecha", columns="ticker", values=col) p = p.rank(axis=1) return p.corrwith(p.shift(1), axis=1).mean() cols = ["valor_bp", "mom_12_1", "vol_63d", "sue", "ruido_3", "ret_1s", "ruido_1", "alfa_verdadero"] pd.Series({c: persistencia(c) for c in cols}).round(3) ``` ```output valor_bp 0.999 mom_12_1 0.975 vol_63d 0.977 sue 0.941 ruido_3 0.977 ret_1s 0.019 ruido_1 -0.001 alfa_verdadero 0.515 dtype: float64 ``` Value (`valor_bp`) has a persistence of 0.999: over 14 years, the ranking of stocks by value changes so slowly that, for practical purposes, it provides a few dozen independent observations of its effect, not hundreds of thousands. Momentum, volatility and SUE are between 0.94 and 0.98. Only the one-week reversal (`ret_1s`) and the i.i.d. noise renew every week. And `ruido_3`, an AR(1) with ρ = 0.98 and no effect at all, is as persistent as momentum: a slow feature with no effect can correlate with returns for years by pure chance. The second reason is that all the stocks in a week share the same market and the same sectors: a week's noise does not cancel out across stocks as much as it would if they were independent. Together, the two effects mean that tests with 14 years of data have less power than the number of rows suggests. The IC table of the 22 features, alongside what the generating model says about each one, puts it to the test: ```python efecto = {"ret_1s": "yes", "mom_12_1": "yes", "vol_63d": "yes", "valor_bp": "yes", "sue": "yes", "sentimiento": "yes", "volumen_rel": "interaction", "dias_desde_informe": "interaction", "revision_analistas": "LEAK"} feats = [c for c in f.columns if c not in ("fecha", "ticker", "sector")] tabla = pd.DataFrame({c: resumen(ic_semanal(d, c)) for c in feats}).T tabla["efecto"] = [efecto.get(c, "no") for c in tabla.index] tabla.sort_values("t")[["ic_medio", "t", "efecto"]].round(4) ``` ```output ic_medio t efecto ret_1s -0.0159 -4.2881 yes rsi_14 -0.0083 -2.6280 no ret_1m -0.0077 -2.1404 no ret_3m -0.0050 -1.5616 no ruido_1 -0.0024 -1.2751 no dist_max_52s -0.0031 -0.9992 no volumen_rel -0.0014 -0.7362 interaction ruido_3 -0.0010 -0.5558 no ruido_5 -0.0008 -0.4661 no dias_desde_informe -0.0005 -0.2746 interaction log_capitalizacion -0.0005 -0.2103 no vol_21d 0.0020 0.3569 no vol_63d 0.0032 0.5593 yes ruido_2 0.0015 0.7851 no ruido_4 0.0044 1.8392 no mom_12_1 0.0077 2.4404 yes roe 0.0057 2.7132 no sentimiento 0.0066 3.0800 yes sue 0.0079 3.7415 yes valor_bp 0.0095 3.9281 yes ey 0.0087 4.0399 no revision_analistas 0.1743 92.1366 LEAK ``` Three readings of the table: - **True effects that do not show.** `vol_63d` has a planted effect and its IC gives t = 0.6: low volatility is masked because volatile stocks have more beta and the market rises on average (the README measures it with Fama-MacBeth). `sentimiento` and `sue`, which have an effect, rank below `ey`. - **Features with no effect of their own that look good.** `ey` (t = 4.0) and `roe` (t = 2.7) inherit the effect of value and SUE, with which they are correlated; `rsi_14` (t = −2.6) and `ret_1m` (t = −2.1), that of reversal. A univariate test does not distinguish cause from resemblance. The multiple testing corrections of lesson 2.1.8 account for the number of attempts, but not for this confusion. - **The leak.** `revision_analistas` gives an IC of 0.174 with t = 92: six times the oracle's ceiling. It is not a great feature: the vendor dates it at the start of the week it covers and it contains the first two days of the return being predicted. A result above the ceiling is a bug, not a finding (lesson 5.2.2 teaches how to audit the dates). A simple calculation measures the scarcity of data. If a signal has a mean IC $\overline{IC}$ with weekly standard deviation $s$, the t-statistic after $n$ weeks is $\overline{IC}\sqrt{n}/s$, and reaching t = 2 takes $n = (2s/\overline{IC})^2$ weeks. For a strategy with annual Sharpe $\mathrm{SR}$, the years needed are $(2/\mathrm{SR})^2$: ```python sem = tabla.loc[["valor_bp", "sue", "sentimiento", "mom_12_1"], ["ic_medio", "desv"]].astype(float) sem.columns = ["ic", "desv"] sem.loc["oracle"] = [ic_or.mean(), ic_or.std()] sem["años_para_t2"] = (2 * sem["desv"] / sem["ic"]) ** 2 / 52 print(sem.round(3)) sharpe = pd.Series([0.3, 0.5, 1.0, 2.0], name="sharpe_anual") print((2 / sharpe) ** 2) ``` ```output ic desv años_para_t2 valor_bp 0.009 0.065 3.634 sue 0.008 0.057 4.006 sentimiento 0.007 0.058 5.911 mom_12_1 0.008 0.085 9.416 oracle 0.029 0.093 0.781 0 44.444444 1 16.000000 2 4.000000 3 1.000000 Name: sharpe_anual, dtype: float64 ``` The oracle, which combines all the effects, stands out from the noise in less than a year. A single feature with an IC of 0.008 or 0.009 needs three and a half to six years, and momentum, whose IC varies more across weeks, more than nine. And these are lower bounds: the formula treats weeks as independent, and with features this persistent they are not. A strategy with an annual Sharpe of 0.5 (good in the real world) needs 16 years of history to give t = 2, and that assumes the relationship does not change in those 16 years. Which is the next difference. ## Non-stationarity: the relationship changes In computer vision, the function that maps pixels to the label is fixed. In `panel5`, the coefficient of each effect changes every week: it is its base value multiplied by a regime factor and by an AR(1) term that weakens it and sometimes reverses it. In addition, momentum is multiplied by −2.5 in **rebounds** after a crisis: this is the momentum *crash*, documented in real markets (the stocks that fell the most are the ones that rebound the most). The file `panel5_coef_verdad.csv` gives the true coefficient for each week: ```python coef = pd.read_csv("panel5_coef_verdad.csv", parse_dates=["fecha"]) coef = coef.set_index("fecha") c_mom = coef["c_mom"] * 1e4 # bp per week and unit print(c_mom.groupby(coef["regimen"]).agg(["mean", "count"]) .round(2)) ic_mom = ic_semanal(d, "mom_12_1") anual = pd.DataFrame({"c_mom_pb": c_mom, "ic_mom": ic_mom}) anual.groupby(anual.index.year).mean().round(4) ``` ```output mean count regimen calma 4.71 576 estres 2.02 89 rebote -10.44 65 c_mom_pb ic_mom fecha 2012 4.9549 0.0133 2013 5.6347 0.0175 2014 5.5424 0.0207 2015 3.4455 0.0153 2016 0.7704 -0.0134 2017 6.0316 0.0188 2018 4.5862 0.0050 2019 0.7888 -0.0069 2020 -2.0040 -0.0101 2021 4.2767 0.0003 2022 -1.4078 0.0177 2023 4.0448 0.0057 2024 3.8732 0.0134 2025 2.0747 0.0107 ``` *Fig. — True momentum coefficient in the panel, week by week, in basis points per week and unit of the feature: it changes with the regime and flips in rebounds.* Weekly line from 2012 to 2025 that mostly moves between 0 and 9 bp, with sharp drops below zero in the rebounds of 2016, 2019, 2020, 2022 and 2025. The minimum, highlighted, is −25.0 bp in the week of 2020-05-15, in the 2020 rebound. In calm periods (576 weeks), momentum returns 4.7 bp per week and unit; in stress, 2.0; in rebounds, −10.4. The year 2020 has a negative mean coefficient (−2.0) because of the March to June rebound, and so does 2022 (−1.4). The measured momentum IC follows the coefficient with noise: it is negative in 2016, 2019 and 2020, years with a low or negative coefficient, but in 2022, with a negative true coefficient, it comes out positive (0.018): one year of data is not even enough to get the sign right. A model trained on 2012-2015 learns a “good” momentum that does damage in 2016 and in 2020, and nothing in the training data warns it. There are two kinds of change and it pays to separate them from now on. **Data drift** changes the distribution of the features (a feature shifts or spreads out), and **concept drift** changes the relationship between features and label, as here. The second is the dangerous one, because it cannot be seen by looking at the features: it only shows up in the results, late. Lesson 5.10.2 detects it with data where the date of each change is known; rolling versus expanding retraining windows are covered in 5.4.3. ## Overfitting: a flexible model learns the noise With little signal, a lot of noise and changing relationships, a flexible model easily finds patterns in the noise. The experiment: three models with the 21 features without the leak, transformed to cross-sectional rank (lesson 5.2.4), trained on 2012-2018 and evaluated on 2019-2025. The label is the next week's return minus its cross-sectional mean. A ridge (linear and regularized), a moderate LightGBM (8 leaves, at least 2,000 rows per leaf) and a flexible LightGBM (512 leaves, 5 rows per leaf, 400 trees). Training is deliberately short: the flexible one takes about 15 seconds on CPU. ```python import lightgbm as lgb from sklearn.linear_model import Ridge feats_x = [c for c in feats if c != "revision_analistas"] X = d.groupby("fecha")[feats_x].rank(pct=True).sub(0.5).fillna(0) y = d["ret_sig_semana"] - g["ret_sig_semana"].transform("mean") d["y"] = y ent = d["fecha"] < "2019-01-01" pru = d["fecha"] >= "2019-01-08" # one-week gap comun = dict(random_state=0, n_jobs=4, deterministic=True, force_row_wise=True, verbose=-1) modelos = { "ridge": Ridge(alpha=10.0), "lgbm_moderado": lgb.LGBMRegressor( n_estimators=200, learning_rate=0.02, num_leaves=8, min_child_samples=2000, subsample=0.5, subsample_freq=1, **comun), "lgbm_flexible": lgb.LGBMRegressor( n_estimators=400, learning_rate=0.1, num_leaves=512, min_child_samples=5, **comun), } def ic_medio(pred, mascara): s = d.loc[mascara, ["fecha", "y"]].assign(p=pred) return ic_semanal(s, "p", "y").mean() filas = {"oracle": [ic_medio(d.loc[ent, "alfa_verdadero"], ent), ic_medio(d.loc[pru, "alfa_verdadero"], pru)]} for nombre, m in modelos.items(): m.fit(X[ent], y[ent]) filas[nombre] = [ic_medio(m.predict(X[ent]), ent), ic_medio(m.predict(X[pru]), pru)] pd.DataFrame(filas, index=["ic_2012_2018", "ic_2019_2025"]).T.round(4) ``` ```output ic_2012_2018 ic_2019_2025 oracle 0.0327 0.0254 ridge 0.0258 0.0190 lgbm_moderado 0.0658 0.0141 lgbm_flexible 0.9626 -0.0022 ``` *Fig. — Mean IC of three models and the oracle out of sample (2019-2025, bars) and in sample (2012-2018, right-hand column): the more flexible the model, the wider the gap between in and out.* Horizontal bars of out-of-sample IC: oracle 0.025, ridge 0.019 (highlighted), moderate LightGBM 0.014 and flexible LightGBM −0.002. On the right, each one's in-sample IC: 0.033, 0.026, 0.066 and 0.963. The oracle marks the ceiling of each period: 0.033 in training and 0.025 in test. The ridge stays below it in both (0.026 and 0.019): it learns little and what it learns holds. The moderate LightGBM gives 0.066 in sample, twice the ceiling, and 0.014 out of sample: it has learned some signal and quite a lot of noise. The flexible one gives 0.963 in sample (it ranks almost perfectly returns that are 99.9% noise) and −0.002 out of sample. It is not that boosting is useless: lesson 5.5.3 tunes it with honest validation. It is that in this data, uncontrolled flexibility turns into memorized noise long before it turns into signal. This yields a sanity rule used throughout the level. If you know, even roughly, the most a signal can deliver: ```mermaid flowchart TB R[Result] --> A{In-sample IC far above the ceiling?} A -- yes --> S[Overfitting] A -- no --> B{Out-of-sample IC above the ceiling?} B -- yes --> F[Leak or bug] B -- no --> C{Profitable after costs?} C -- no --> N[Worthless signal] C -- yes --> V[Validate further: 5.4] ``` In real data there is no oracle, but there are orders of magnitude: the R² in the literature, the Sharpe of the best known strategies, the IC of published anomalies. A result far above that scale is investigated as a bug before it is celebrated. ## Competition: the signal expires and gets shared The last difference is that the market reacts. If a signal makes money, others find it, trade it and use it up. A first effect shows up in the data: the signal's horizon is short, and whoever exploits it late gets less. The cell uses each week's oracle to predict the following week's return, as always, and also the return one, two, three and four weeks later: ```python a = d.pivot(index="fecha", columns="ticker", values="alfa_verdadero").rank(axis=1) r = d.pivot(index="fecha", columns="ticker", values="ret_sig_semana") retraso = {} for k in range(5): ic_k = a.corrwith(r.shift(-k).rank(axis=1), axis=1).dropna() retraso[k] = resumen(ic_k) print(pd.DataFrame(retraso).T.rename_axis("semanas_tarde").round(4)) for col in ("ret_1s", "valor_bp"): x = d.pivot(index="fecha", columns="ticker", values=col).rank(axis=1) ics = [x.corrwith(r.shift(-k).rank(axis=1), axis=1).mean() for k in range(4)] print(col, np.round(ics, 4)) ``` ```output ic_medio desv t semanas_pos semanas_tarde 0 0.0292 0.0929 8.4714 0.6461 1 0.0110 0.0900 3.2974 0.5701 2 0.0111 0.0875 3.4129 0.5695 3 0.0105 0.0907 3.1194 0.5565 4 0.0075 0.0875 2.2963 0.5462 ret_1s [-0.0159 -0.0001 0.0004 -0.0012] valor_bp [0.0095 0.0089 0.0087 0.0087] ``` With a one-week delay, the IC drops from 0.029 to 0.011: almost two thirds are lost. What remains does not run out quickly (0.011 and 0.010 with two and three weeks of delay) because it is the slow part of the signal: value, momentum and SUE change slowly, and the expected return three weeks out resembles next week's. The last two lines separate them: the value IC barely moves with the delay (from 0.0095 to 0.0087), and that of the one-week reversal (`ret_1s`, −0.016) disappears completely with the first delay. In a real market, that split is the result of competition: fast signals are collected by those who arrive first, and a weekly system is left with the slow part. On a scale of years, the effect was measured by McLean and Pontiff (2016) with 97 variables published in the academic literature that predicted returns: the return of their portfolios is 26% lower outside the original paper's sample and 58% lower after publication. The authors read the 26% as an upper bound on the part due to data mining and the difference (32 points) as the effect of investors who learn from the publication. For ML, the lesson is twofold: what worked in the documented past is what most competitors know, and a model that “discovers” a known anomaly has discovered nothing. ## What changes in the method The four differences translate into a workflow that is the backbone of the level: ```mermaid flowchart TB D[Point-in-time data] --> E[Labels and weights] E --> V[Purged validation] V --> M[Model and baseline] M --> K[Economic metrics] K --> B[Backtest and DSR] B --> W[Monitor drift] ``` - **Point-in-time data** (5.2.2): each feature, with the date on which it is known. The `revision_analistas` leak is not detected by any validation; only by auditing the dates. - **Labels and weights** (5.3): labels that overlap in time mean fewer independent samples; uniqueness and the sequential bootstrap correct for it. - **Purged validation** (5.4.1-5.4.3): random k-fold mixes neighbors in time and promises accuracy that does not exist. - **Model and baseline** (5.5-5.8): each model is compared with a simple reference (a ridge, a single feature, HAR for volatility) and with the ceiling when it is known. - **Economic metrics** (5.1.3) and **backtest with Deflated Sharpe** (5.4.4): a model is judged by the portfolio it produces, after costs, and discounting how many experiments were run (lesson 2.5.4). - **Monitor drift** (5.10.2): the relationship will change; you need to know when. ## Exercises 1. Compute the oracle's mean IC by regime (`calma`, `estres`, `rebote`) with the `regimen` column of `panel5_coef_verdad.csv`. In which regime is it hardest to predict even when the expected return is known? **Solution.** ```python reg = coef["regimen"].reindex(ic_or.index) ic_or.groupby(reg).agg(["mean", "std", "count"]).round(4) ``` ```output mean std count regimen calma 0.0286 0.0889 575 estres 0.0301 0.0948 89 rebote 0.0330 0.1223 65 ``` The oracle's mean IC is similar in the three regimes (0.029 in calm, 0.030 in stress and 0.033 in rebound), but its noise is not: the weekly standard deviation of the IC rises from 0.089 in calm to 0.095 in stress and 0.122 in rebounds, where momentum changes sign and common returns dominate. Relative to its noise, the rebound is the hardest regime (0.033/0.122 = 0.27 versus 0.32 in calm), and it also has only 65 weeks. Knowing the expected return does not remove the noise: what changes across regimes is how much the signal weighs against it. 2. With the formula $n = (2s/\overline{IC})^2$, compute how many weeks a signal with the ridge's mean out-of-sample IC (0.019) and a weekly standard deviation of 0.09 would need to give t = 2, and how many for t = 3, the threshold proposed by the multiple testing literature (lesson 2.1.8). **Solution.** ```python ic_r, s_r = 0.019, 0.09 for t_obj in (2, 3): n = (t_obj * s_r / ic_r) ** 2 print(f"t = {t_obj}: {n:.0f} weeks ({n / 52:.1f} years)") ``` ```output t = 2: 90 weeks (1.7 years) t = 3: 202 weeks (3.9 years) ``` For t = 2, a good year and a half is enough; for t = 3, almost four years. And the calculation assumes that the IC is stable and that weeks are independent: with changing relationships and persistent features, the real time is longer. 3. A colleague presents a classifier trained on `panel5` that predicts whether a stock will end above the median 21 sessions ahead, with a hit rate of 59.7% in 5-fold shuffled cross-validation. Explain why you do not believe it and what you would ask for. **Solution.** The oracle, which knows the true expected return, is right 51% of the time: 59.7% is nine points above the ceiling, so there is a bug in the validation or a leak. Here it is the validation: 21-session labels overlap by about four weeks and slow features barely change, so when shuffling, the model sees in training the same stock one week before and after, with almost the same label, and memorizes it. The level README measures it: 59.7% with shuffled k-fold, 50.1% with purged k-fold and with walk-forward. I would ask for purged validation with an embargo or a walk-forward (lessons 5.4.1 and 5.4.2), an audit of each feature's dates and the result in economic metrics after costs (5.1.3). ## Self-assessment 1. A model trained on `panel5` gives a mean out-of-sample IC of 0.15. The oracle, which knows the true expected return, gives 0.029. What do you conclude? - A. That the model has found a signal the generator did not know about - B. That the model is five times better than the oracle - C. That there is a leak or a bug: no model can beat the true expected return out of sample - D. That the oracle is miscalculated *Answer C: That there is a leak or a bug: no model can beat the true expected return out of sample.* The oracle is the best possible prediction with the information available at the time. Beating it systematically is only possible by using later information. In the data, `revision_analistas` gives an IC of 0.174 for that reason: it includes the first two days of the week it predicts. 2. The oracle correctly predicts whether a stock ends above the median in 50.95% of cases. Why is that hit rate enough for a portfolio with a high gross Sharpe? - A. Because it bets on 400 stocks at once every week, for 729 weeks: a tiny edge per bet accumulates with breadth - B. Because the hit rate is mismeasured - C. Because the oracle is right more often on large stocks - D. It is not enough: you cannot make money with 51% *Answer A: Because it bets on 400 stocks at once every week, for 729 weeks: a tiny edge per bet accumulates with breadth.* The fundamental law of active management (lesson 5.1.3) relates the information ratio to IC·√breadth. What it is not enough for is paying the costs of turning over everything every week: the level README gives a net Sharpe of 0.38 for the oracle. 3. A LightGBM with 512 leaves gives an IC of 0.96 in training (2012-2018) and −0.002 in 2019-2025. What is the diagnosis? - A. Information leakage - B. Overfitting: the in-sample IC is far above the oracle's ceiling (0.033 in that period), so the model has memorized noise - C. A regime change in 2019 - D. Missing data in 2019-2025 *Answer B: Overfitting: the in-sample IC is far above the oracle's ceiling (0.033 in that period), so the model has memorized noise.* In sample, any IC above the ceiling is learned noise. The moderate LightGBM also exceeds it (0.066) and out of sample ends at 0.014, below the ridge (0.019). 4. `vol_63d` has a true effect and its IC gives t = 0.6; `rsi_14` has no effect of its own and gives t = −2.6. What does this teach? - A. That the generator is wrong - B. That you should use the RSI - C. That t = 2 is too high a threshold - D. That with 14 years and this signal-to-noise ratio, a univariate test confuses true effects with features correlated with another effect (the RSI inherits reversal) and fails to detect effects masked by another (beta masks low volatility) *Answer D: That with 14 years and this signal-to-noise ratio, a univariate test confuses true effects with features correlated with another effect (the RSI inherits reversal) and fails to detect effects masked by another (beta masks low volatility).* The features resemble each other and the noise is huge. The README gives the detail: in Fama-MacBeth, low volatility is not recovered with the raw return because volatile stocks have more beta. 5. With the oracle's prediction used one week late, the mean IC drops from 0.029 to 0.011. How does this relate to competition? - A. It does not - B. It indicates that the oracle has a leak - C. That you should predict four weeks ahead - D. The signal has a short horizon: whoever exploits it first takes most of it. With many participants looking for the same thing, the latecomer trades on the slow part that remains *Answer D: The signal has a short horizon: whoever exploits it first takes most of it. With many participants looking for the same thing, the latecomer trades on the slow part that remains.* It is the decay of the signal with delay. On a scale of years, McLean and Pontiff (2016) measure something similar with 97 published anomalies: 26% lower returns out of sample and 58% lower after publication. ## Key points - In the level's panel, the perfect model (the true expected return) has a weekly IC of 0.029, is right 50.95% of the time and explains 0.09% of the variance. That is the ceiling, and it is that of a real market. - Even the oracle has a negative IC in 35% of weeks: the signal only shows up by aggregating many bets over a long time. - Many rows are not much data: slow features change little from one week to the next and 14 years are not enough to separate true effects from false ones in a univariate test. - Relationships change: the true momentum coefficient is positive in calm periods and negative in rebounds; a fixed model learns an average that does not repeat. - Sanity rule: an in-sample IC far above the ceiling is overfitting; an out-of-sample IC above the ceiling is a leak. - The signal expires: using the oracle one week late leaves 38% of the IC; published anomalies lose more than half of their return. ## Resources ### Research - [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min - [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min ### Books - [McLean, R. D. and Pontiff, J. (2016): Does Academic Research Destroy Stock Return Predictability? The Journal of Finance 71(1)](https://doi.org/10.1111/jofi.12365) · doi.org - [Israel, R., Kelly, B. and Moskowitz, T. (2020): Can Machines “Learn” Finance? Journal of Investment Management 18(2), 23-36](https://www.aqr.com/Insights/Research/Journal-Article/Can-Machines-Learn-Finance) · aqr.com - [Gu, S., Kelly, B. and Xiu, D. (2020): Empirical Asset Pricing via Machine Learning. The Review of Financial Studies 33(5), 2223-2273 (NBER version w25398)](https://www.nber.org/papers/w25398) · nber.org - [López de Prado, M. (2018): Advances in Financial Machine Learning. Wiley](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com ### Data - [Level 5 data: weekly features of the 400-stock panel (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet Next lesson: [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/)