---
title: "5.2.4 Leakage-free normalization and cross-sectional rank"
description: "Z-scoring each stock with its full history gives a feature with no effect an IC above the oracle's. Windows without the future, ranks and date leaks."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/leakage-free-normalization-cross-sectional-rank/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/leakage-free-normalization-cross-sectional-rank.png"
---
# 5.2.4 · Leakage-free normalization and cross-sectional rank
> Scaling features looks like a technical step without consequences, and in financial series it is one of the most effective sources of leakage: a mean computed with the whole sample carries the future of each stock inside it. This lesson measures how much that leak pays in the level's panel, shows the alternatives without the future (expanding and rolling windows, ranks within each week) and audits the feature that the vendor dates too early.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.2 · Feature engineering
- Duration: 60 min
- Prerequisites: [1.7.4 · Multi-timeframe analysis without look-ahead](https://pythonparatrading.com/en/python-trading-course/level-1/multi-timeframe-analysis-without-look-ahead/), [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/), [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/), [5.2.1 · Price, volume and microstructure features](https://pythonparatrading.com/en/python-trading-course/level-5/price-volume-microstructure-features/)
## By the end of this lesson
- Measure the leak of normalizing each stock with full-sample statistics (z-score and min-max) and compare it with the oracle's ceiling.
- Program expanding and rolling normalizations with `groupby` and check them with an automatic test: the value on a date cannot change when the future is cut off.
- Distinguish time-series normalization (relative to the stock's own history) from cross-sectional normalization (relative to the other stocks that week) and choose between z-score, winsorization, rank and normal rank.
- Audit a feature with a date leak: session-by-session IC over the following week and the lag test.
## Scaling is not a neutral step
Almost all the models in the following lessons need the features on a comparable scale: a penalized regression
punishes all coefficients equally, a neural network learns badly with inputs of different orders of magnitude and a
k-nearest neighbors measures distances. That is why, between computing a feature (lessons 5.2.1 to 5.2.3) and
training, there is almost always a normalization. In a generic ML tutorial it is one line:
`StandardScaler().fit_transform(X)`. In a financial panel you have to decide **relative to what** each value is
normalized and **with which data** the statistics are computed:
| Normalization | Relative to what | Data for the statistics | Uses the future? |
|---|---|---|---|
| z-score or min-max with the stock's whole sample | Its own history | All dates, including later ones | **Yes** |
| Expanding z-score | Its own history | From the start up to the date | No |
| Rolling z-score | Its recent history | The last $n$ dates up to the date | No |
| z-score, rank or normal rank within each week | The other stocks that week | That date only | No |
| The model's global scaler | The whole training set | The training set (or, done wrong, everything) | Depends on where it is fitted |
This lesson measures each row of the table in the level 5 panel: 400 fictitious stocks, 730 weeks from 2012 to 2025,
22 weekly features and a published ground truth (`public/curso/datos/nivel-5/README.md`). The data are synthetic and
the effects are planted: they serve to know how much a leak pays, not to estimate anything about the real market. The
last section changes problem: a feature that no normalization can fix, because the vendor dates it before it is known.
## The panel and the yardstick
The measure for the whole module is the **IC**: the rank correlation, week by week, between a feature and the
following week's return (`ret_sig_semana`), averaged over the weeks; its $t$ statistic is the mean divided by the
standard deviation and multiplied by the square root of the number of weeks (lesson 5.1.3).
```python
import numpy as np
import pandas as pd
f = pd.read_parquet("panel5_features.parquet")
e = pd.read_parquet("panel5_etiquetas.parquet")
v = pd.read_parquet("panel5_verdad.parquet")
d = (f.merge(e[["fecha", "ticker", "ret_sig_semana"]],
on=["fecha", "ticker"])
.merge(v[["fecha", "ticker", "alfa_verdadero"]],
on=["fecha", "ticker"])
.dropna(subset=["ret_sig_semana"])
.sort_values(["ticker", "fecha"])
.reset_index(drop=True))
def ic(df, col, y="ret_sig_semana"):
x = df[["fecha", col, y]].dropna()
r = x.groupby("fecha")[[col, y]].rank()
s = r[col].groupby(x["fecha"]).corr(r[y])
return pd.Series({"ic": s.mean(),
"t": s.mean() / s.std() * np.sqrt(s.count()),
"weeks": s.count()})
print(d.shape)
pd.DataFrame({c: ic(d, c) for c in
["alfa_verdadero", "valor_bp", "log_capitalizacion"]}
).T.round(4)
```
```output
(291600, 27)
ic t weeks
alfa_verdadero 0.0292 8.4714 729.0
valor_bp 0.0095 3.9281 729.0
log_capitalizacion -0.0005 -0.2103 729.0
```
The ceiling is the IC of the **oracle**, `alfa_verdadero`: the part of next week's return that the generating model
plants, known only because the data are synthetic. No model can go beyond it out of sample. `valor_bp` (the log of
the ratio between book equity and market cap) has a planted effect and a small, positive IC; `log_capitalizacion` has
no effect and its IC is zero. Those two are the guinea pigs of the next section.
## The full-sample leak
The most common way to z-score a feature of a time series is to subtract its mean and divide by its standard
deviation, computed with the whole series:
$$
z_{i,t} = \frac{x_{i,t} - \overline{x}_i}{s_i}, \qquad \overline{x}_i = \frac{1}{T}\sum_{u=1}^{T} x_{i,u}.
$$
The sum runs up to $T$, the end of the sample. If $x$ depends on the price, like market cap or the book-to-market
ratio, $\overline{x}_i$ depends on all of the stock's future prices. A stock that is going to rise a lot has a high
future mean market cap, so today its market cap sits **below** its mean: a negative $z_{i,t}$ announces rises. Min-max
($\frac{x - \min}{\max - \min}$ with the minimum and maximum of the whole series, the default scaling in many neural
network tutorials) has the same flaw: it says at which point of its future path the price is.
```python
def z_completa(df, col):
s = df.groupby("ticker")[col]
return (df[col] - s.transform("mean")) / s.transform("std")
def minmax_completa(df, col):
s = df.groupby("ticker")[col]
mn, mx = s.transform("min"), s.transform("max")
return (df[col] - mn) / (mx - mn)
def z_expansiva(df, col, minimo=52):
s = df.groupby("ticker")[col]
m = s.transform(lambda x: x.expanding(minimo).mean())
sd = s.transform(lambda x: x.expanding(minimo).std())
return (df[col] - m) / sd
def z_movil(df, col, ventana=52):
s = df.groupby("ticker")[col]
m = s.transform(lambda x: x.rolling(ventana).mean())
sd = s.transform(lambda x: x.rolling(ventana).std())
return (df[col] - m) / sd
metodos = {"raw": lambda df, c: df[c],
"z full sample": z_completa,
"min-max full sample": minmax_completa,
"z expanding (52 w min.)": z_expansiva,
"z rolling (52 w)": z_movil}
filas = {}
for col in ["log_capitalizacion", "valor_bp"]:
t = d[["fecha", "ticker", "ret_sig_semana"]].copy()
for nombre, fn in metodos.items():
t[nombre] = fn(d, col)
t = t.dropna() # same weeks for all: after the 52nd
for nombre in metodos:
filas[(col, nombre)] = ic(t, nombre)[["ic", "t"]]
pd.DataFrame(filas).T.round(4)
```
```output
ic t
log_capitalizacion raw 0.0000 0.0080
z full sample -0.0490 -15.2101
min-max full sample -0.0451 -14.3760
z expanding (52 w min.) 0.0007 0.2304
z rolling (52 w) -0.0009 -0.3030
valor_bp raw 0.0095 3.7176
z full sample 0.0501 15.7421
min-max full sample 0.0438 13.5968
z expanding (52 w min.) 0.0017 0.5449
z rolling (52 w) 0.0020 0.6477
```
The figure at the top draws the table. With the full-sample z-score, market cap, which has no effect at all, ends up
with an IC of −0.049 with $t = -15$: almost twice the oracle's ceiling, in absolute value. Value goes from 0.010 to
0.050, five times its real effect. Min-max gives the same. In a backtest, either of them would look like the discovery
of the year, and both are information from the prices of the following years.
With expanding or rolling statistics, which on each date use only the past, market cap goes back to zero (0.001 and
−0.001). That is the known answer: it has no effect.
## What changes when normalizing against one's own past
The same table has a second, subtler reading. With the expanding z-score, `valor_bp` falls from 0.0095 to 0.0017 and
is no longer significant, and not because of a leak. The planted effect is **cross-sectional**: stocks that are cheap
relative to the others return a little more the following week. Z-scoring each stock against its own history answers
another question, whether the stock is cheap relative to itself, and that question has no effect in the panel.
```mermaid
flowchart TB
X[x of stock i] --> P{Compare with?}
P -- its past --> T[Time series]
P -- others at t --> C[Cross-sectional]
T --> T1[Unusual for it?]
C --> C1[High today?]
```
There is no correct normalization in the abstract: the correct one is the one that keeps the information the effect
uses. A volatility feature for a single-asset time-series model (lesson 5.8.5) is compared with the asset's past; a
value or momentum feature for picking stocks, with the other stocks on the same date. And the two can be combined: a
stock's time-series z-score, ranked afterwards across stocks.
## An automatic test against the future
Reviewing the code by eye is not enough: in a pipeline of several hundred lines, a full-sample mean hides well. The
property that defines a transformation without the future can be checked mechanically: the value on date $t$
computed with the data up to $t$ is the same as computed with all the data.
```python
def usa_el_futuro(fn, df, col, corte="2018-12-28"):
antes = df["fecha"] <= corte
con_todo = fn(df, col)[antes]
recortado = fn(df[antes], col)
return not np.allclose(con_todo, recortado, equal_nan=True)
{nombre: usa_el_futuro(fn, d, "valor_bp")
for nombre, fn in metodos.items()}
```
```output
{'raw': False, 'z full sample': True, 'min-max full sample': True, 'z expanding (52 w min.)': False, 'z rolling (52 w)': False}
```
It is a cheap unit test worth attaching to every feature and normalization function in a project: it runs in
seconds, does not depend on any model and detects the leak before it reaches a backtest. It also detects the less
obvious variants, such as `rolling(..., center=True)` (the first exercise), a `bfill` that fills gaps with the next
value or a `resample` with the label on the wrong edge (lesson 1.7.4).
## The model's global scaler
The mistake most often cited in the scikit-learn documentation is fitting the scaler on all the data before splitting
into training and test. In this panel you can measure what it costs. The experiment is the README's reference model:
a ridge (α = 10) on the 21 features without the leak, retrained every January from 2015 to 2025 with all previous
weeks and with the label minus its weekly mean. One version fits the `StandardScaler` inside each training window;
the other, once with all years:
```python
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
d["y"] = (d["ret_sig_semana"]
- d.groupby("fecha")["ret_sig_semana"].transform("mean"))
FEATS = [c for c in f.columns
if c not in ("fecha", "ticker", "sector",
"revision_analistas")]
anio = d["fecha"].dt.year.to_numpy()
def walk_forward(X, escalar_con_todo=False):
X = X.copy()
if escalar_con_todo:
X[:] = StandardScaler().fit_transform(X.fillna(X.median()))
pred = pd.Series(np.nan, index=X.index)
for a in range(2015, 2026):
tr, te = anio < a, anio == a
med = X[tr].median()
Xtr, Xte = X[tr].fillna(med), X[te].fillna(med)
if not escalar_con_todo:
esc = StandardScaler().fit(Xtr)
Xtr, Xte = esc.transform(Xtr), esc.transform(Xte)
m = Ridge(alpha=10).fit(Xtr, d.loc[tr, "y"])
pred.loc[te] = m.predict(Xte)
return ic(d.assign(pred=pred), "pred", "y")[["ic", "t"]]
pd.DataFrame({
"scaler in each training window": walk_forward(d[FEATS]),
"scaler with all years": walk_forward(d[FEATS], True),
}).T.round(4)
```
```output
ic t
scaler in each training window 0.0159 4.5775
scaler with all years 0.0159 4.5724
```
The IC is the same to the fourth decimal (0.0159); only the second decimal of $t$ changes. It is not that the
practice is correct: it is that this leak **cannot be exploited**. A global scaler applies the same affine
transformation to all rows, and a linear model undoes it with its coefficients; trees, which only look at the order of
each feature, do not even notice it. The scaler leak does matter when its statistics vary a lot between training and
test and the model is not invariant to them (networks with saturating activations, k-nearest neighbors, fixed
thresholds on the scaled feature), and that is why the rule is still the one from lesson 5.1.2: everything that is
fitted goes inside the `Pipeline` and inside each window.
The difference with the previous section is the one worth remembering. The global mean is **the same** for all rows;
the mean over a stock's whole history is **different for each stock** and depends on its future. The second reorders
the stocks within each week according to what is going to happen to them, and that a model does exploit.
## Cross-sectional transformations
For picking stocks, the natural normalization is the one across stocks within each week. Let $x_{i,t}$ be the feature
of stock $i$ in week $t$ and $n_t$ the number of stocks with data:
- **Cross-sectional z-score**: $(x_{i,t} - \overline{x}_t)/s_t$, with that week's mean and standard deviation.
- **Winsorized z-score**: the previous one clipped at $\pm 3$, so that an extreme value does not dominate.
- **Rank**: $\text{rank}_{i,t}/n_t - 0.5$, uniform between $-0.5$ and $0.5$ whatever the distribution.
- **Normal rank** (*rank-gauss*): $\Phi^{-1}\big((\text{rank}_{i,t} - 0.5)/n_t\big)$, the rank mapped to a standard
normal. It is the form $g(\cdot)$ with which the panel's generator plants the effects.
All of them are computed with the data of a single date, so none can use the future. What they remove is something
else: the level shifts of the whole market. The 63-session volatility shows it well:
```python
from scipy import stats
sem = d.groupby("fecha")
def cs_z(col):
media = sem[col].transform("mean")
return (d[col] - media) / sem[col].transform("std")
def cs_rango(col):
return sem[col].rank(pct=True) - 0.5
def cs_gauss(col):
n = sem[col].transform("count")
return pd.Series(stats.norm.ppf((sem[col].rank() - 0.5) / n),
index=d.index)
anual = pd.DataFrame({
"raw median": d.groupby(d["fecha"].dt.year)["vol_63d"].median(),
"median of the rank": cs_rango("vol_63d")
.groupby(d["fecha"].dt.year).median(),
})
anual.loc[[2017, 2019, 2020, 2022, 2024]].round(3)
```
```output
raw median median of the rank
fecha
2017 0.246 0.001
2019 0.278 0.001
2020 0.354 0.001
2022 0.356 0.001
2024 0.274 0.001
```
*Fig. — Raw 63-session volatility of the panel's 400 stocks: median and 10th and 90th percentiles each month, 2012-2025. In cross-sectional ranks the median would be the same every week.* Three monthly lines from 2012 to 2025. The median, highlighted, moves between 0.23 (July 2019) and 0.50 (May 2020), rising in 2015-2016, 2018, 2020, 2022 and 2025. The 10th and 90th percentiles follow the same level shifts: the 90th percentile reaches 0.65 in 2020.
Raw, the panel's median volatility changes with the regimes of the market model: from 0.246 in 2017 to 0.354 in 2020
(the February and March crisis) and 0.356 in 2022. A model trained on raw volatility in calm years sees values in 2020
that it had hardly seen, and its prediction for all stocks shifts at once. With the rank, the median is always the
same (0.001, and not exactly 0 because each week there is an even number of stocks) and only the order between
stocks remains, which is what a long-short portfolio uses. If the market level matters (aggregate volatility is a
useful regime variable), it is added separately, as a feature common to all stocks that week, not mixed with the
cross-sectional one.
The second thing ranks remove is extremes. `ret_1s` has fat tails and `ruido_5` is built with very fat tails; excess
kurtosis measures it:
```python
pd.DataFrame({c: {"raw": stats.kurtosis(d[c], nan_policy="omit"),
"cross-sectional z": stats.kurtosis(cs_z(c),
nan_policy="omit"),
"normal rank": stats.kurtosis(cs_gauss(c),
nan_policy="omit")}
for c in ["ret_1s", "ruido_5", "vol_63d"]}).round(2)
```
```output
ret_1s ruido_5 vol_63d
raw 2.93 3.62 4.87
cross-sectional z 3.49 3.42 6.13
normal rank -0.06 -0.06 -0.06
```
The cross-sectional z-score removes each week's level, but not the tails: a few stocks with huge moves still weigh
much more in the regression than the rest. The normal rank leaves all features with the same distribution, a standard
normal truncated at its extremes.
## Which transformation predicts best?
With the same walk-forward as before, changing only how the features enter:
```python
transformaciones = {
"raw": d[FEATS],
"cross-sectional z": pd.DataFrame({c: cs_z(c) for c in FEATS}),
"winsorized z ±3": pd.DataFrame({c: cs_z(c).clip(-3, 3)
for c in FEATS}),
"rank": pd.DataFrame({c: cs_rango(c) for c in FEATS}),
"normal rank": pd.DataFrame({c: cs_gauss(c) for c in FEATS}),
}
pd.DataFrame({k: walk_forward(X) for k, X in transformaciones.items()}
).T.round(4)
```
```output
ic t
raw 0.0159 4.5775
cross-sectional z 0.0164 4.5211
winsorized z ±3 0.0177 4.8441
rank 0.0177 4.5728
normal rank 0.0184 4.8721
```
The order is the expected one, from the raw feature (0.016) to the normal rank (0.018), and the normal rank is the
true form of the panel's effects. But the table should be read with its error: with $t$ between 4.5 and 4.9, the
standard error of each IC is about 0.0035, larger than any of the differences. In this panel, the cross-sectional
transformation is not what separates a good model from a bad one; what it does is make the model **stable**: the same
coefficient means the same in 2017 and in 2020, no extreme value moves the regression and a new feature enters on the
same scale as the others. With models that are retrained rarely or with more fat-tailed features, the difference
grows.
## A feature that is too good
The README's feature table has a row that looks like none of the others:
```python
pd.DataFrame({c: ic(d, c) for c in
["revision_analistas", "alfa_verdadero", "ret_1s"]}
).T.round(4)
```
```output
ic t weeks
revision_analistas 0.1743 92.1366 729.0
alfa_verdadero 0.0292 8.4714 729.0
ret_1s -0.0159 -4.2881 729.0
```
An IC of 0.174, six times the oracle's, with $t = 92$. No normalization changes it: as ranks or raw, it is still
there. The explanation is in the level's README: the (fictitious) vendor publishes analyst revisions and **dates them
at the start of the week they cover**, but the value incorporates what happened in the first two sessions of that
week. In the Friday row, the model receives a number that in reality is not known until the following Tuesday.
It is the same date problem as in lesson 5.2.2 (fundamentals dated at the quarter close instead of at publication),
with one difference: here there is no second column with the right date. It has to be inferred from the data.
```mermaid
flowchart TB
V[Friday: row] --> L[Monday and Tuesday]
L --> M[Value published]
M --> E[Next Friday]
V -. value date .-> M
```
## Auditing when a value is known
The audit has two steps. The first: decompose the label. If the feature truly anticipates the following week, its IC
is spread over the five sessions; if it carries specific sessions inside it, it is concentrated in them.
```python
p = pd.read_parquet("panel5_precios.parquet",
columns=["fecha", "ticker", "cierre"])
P = p.pivot(index="fecha", columns="ticker", values="cierre")
R = P.pct_change()
def matriz(col):
return d.pivot(index="fecha", columns="ticker", values=col)
F = matriz("revision_analistas")
pos = P.index.get_indexer(F.index)
def ic_matriz(X, Y):
X, Y = X.where(Y.notna()), Y.where(X.notna())
c = X.rank(axis=1).corrwith(Y.rank(axis=1), axis=1)
return pd.Series({"ic": c.mean(),
"t": c.mean() / c.std() * np.sqrt(c.count())})
def a_matriz(valores):
return pd.DataFrame(valores, index=F.index, columns=P.columns)
tramos = {"week ending on the date":
a_matriz(P.to_numpy()[pos] / P.to_numpy()[pos - 5] - 1)}
for k in range(1, 6):
idx = np.minimum(pos + k, len(R) - 1)
vals = R.to_numpy()[idx]
vals[pos + k >= len(R)] = np.nan
tramos[f"next session {k}"] = a_matriz(vals)
tramos["label (next week)"] = matriz("ret_sig_semana")
pd.DataFrame({k: ic_matriz(F, Y) for k, Y in tramos.items()}
).T.round({"ic": 3, "t": 1})
```
```output
ic t
week ending on the date -0.004 -2.2
next session 1 0.189 97.1
next session 2 0.178 93.1
next session 3 -0.003 -1.6
next session 4 0.002 0.8
next session 5 -0.002 -1.0
label (next week) 0.174 92.1
```
*Fig. — IC of revision_analistas with the return of each window: it sits entirely in the first two sessions of the following week and vanishes when the feature is lagged one week.* Horizontal bars. Past week −0.004; session 1 of the following week 0.189 and session 2 0.178, highlighted; sessions 3, 4 and 5 −0.003, 0.002 and −0.002; whole following week 0.174; the feature lagged one week, −0.002.
The IC lies entirely in sessions 1 and 2 (0.189 and 0.178) and is zero in sessions 3, 4 and 5 and in the past week.
An economic effect does not know on which session the week starts: an analyst revision that truly anticipated returns
would do so spread out, and decaying. A correlation that starts and ends on exact sessions is the fingerprint of a
wrongly set date.
The second step: **lag** the feature until the moment it is known for sure and see what remains. A real effect
survives the lag; a leak does not.
```python
L = matriz("ret_sig_semana")
V = matriz("valor_bp")
pd.DataFrame({
f"lag {k} wk.": {
"revision_analistas": ic_matriz(F.shift(k), L)["ic"],
"valor_bp": ic_matriz(V.shift(k), L)["ic"]}
for k in [0, 1, 2, 4]}).round(4)
```
```output
lag 0 wk. lag 1 wk. lag 2 wk. lag 4 wk.
revision_analistas 0.1743 -0.0016 0.0002 -0.0012
valor_bp 0.0095 0.0089 0.0087 0.0088
```
Lagged one week, `revision_analistas` has an IC of −0.002: nothing. Value lagged one, two or four weeks keeps its IC
of 0.009: it is a slow, genuine effect. Without it, the ridge of the previous section stays at 0.016-0.018; with it
as is, the README measures an IC of 0.175 for the same ridge and a long-short portfolio with a net Sharpe of 17.7, the
figure nobody should believe.
## Typical mistakes
- **Scaling the whole series before building windows.** The pattern of many LSTM tutorials: `MinMaxScaler` over the
whole price and then the sequences. Each window knows where the future maximum is (lesson 5.6.3).
- **`rolling(center=True)` or `shift(-k)` in a feature.** Both look ahead. The cut-off-the-future test catches them.
- **Filling gaps backward.** `bfill` copies the next known value into dates on which it was not yet known.
- **Normalizing with today's universe.** Computing the ranks with the stocks that exist at the end of the sample
introduces survivorship bias into the feature (lesson 1.4.4); the rank on each date is computed with the stocks that
were trading on that date.
- **Trusting the vendor's date.** A row's date is that of the period the value refers to, not necessarily that of its
publication. Without a publication date, lag with a margin and audit.
## Exercises
1. A centered moving average (`rolling(52, center=True)`) smooths better than one that only looks back. Write the
centered rolling z-score, run it through `usa_el_futuro` and measure its IC with market cap.
**Solution.**
```python
def z_centrada(df, col, ventana=52):
s = df.groupby("ticker")[col]
m = s.transform(lambda x: x.rolling(ventana, center=True).mean())
sd = s.transform(lambda x: x.rolling(ventana, center=True).std())
return (df[col] - m) / sd
t = d[["fecha", "ret_sig_semana"]].assign(
zc=z_centrada(d, "log_capitalizacion")).dropna()
print("uses the future:", usa_el_futuro(z_centrada, d,
"log_capitalizacion"))
ic(t, "zc")[["ic", "t"]].round(4)
```
```output
uses the future: True
ic -0.2148
t -68.4267
dtype: float64
```
The centered window uses 26 future weeks on each date, and the test detects it. Its IC with market cap, which has no
effect, is −0.215 with $t = -68$, four times that of the full-sample z-score: the mean of the next 26 weeks depends
much more on next week's return than the 14-year mean does, in which that week weighs little.
2. The panel's value effect is spread across sectors. Compute the normal rank of `valor_bp` within each sector and
week (sector-neutralized) and compare its IC with that of the normal rank over the whole universe.
**Solution.**
```python
gs = d.groupby(["fecha", "sector"])["valor_bp"]
n_s = gs.transform("count")
d["vbp_sector"] = stats.norm.ppf((gs.rank() - 0.5) / n_s)
d["vbp_gauss"] = cs_gauss("valor_bp")
pd.DataFrame({c: ic(d, c) for c in ["vbp_gauss", "vbp_sector"]}
).T[["ic", "t"]].round(4)
```
```output
ic t
vbp_gauss 0.0095 3.9281
vbp_sector 0.0077 4.5619
```
Neutralized value gives a somewhat lower IC (0.0077 versus 0.0095) but one that is more stable week to week
($t$ = 4.6 versus 3.9). In the panel the effect is planted on the rank over the whole universe, so the part of value
that compares sectors with each other also pays, and is lost when neutralizing; in exchange, the feature stops
carrying each sector's common return, which is noise for this effect. With real data neutralization usually matters
more, because there are entire sectors that are structurally “cheap” (banks, energy) whose return depends on their own
cycle: neutralizing separates the value bet from the sector bet.
3. A sentiment vendor delivers every morning at 7:00 (New York time) a file with one row per stock and the previous
day's date, computed with the news up to 23:59. Your features are from each Friday's close. What knowledge date do
you assign to each row and what would you check before using it?
**Solution.**
The row dated Thursday is known on Friday at 7:00, before Friday's close: it can be used in Friday's feature. The
Friday row, on the other hand, is known on Saturday: it goes into the following Friday's feature, not into that
Friday's. The general rule is to store the file's arrival date (`fecha_conocimiento`) and join with `merge_asof` on
it, never on the value's date (lesson 5.2.2). Before using it: decompose the IC by session as in this lesson (if it
concentrates on the following Monday, something arrives late), lag it one day and one week and see how much IC
survives, and check in the vendor's history whether old rows were rewritten afterwards (a history “cleaned” after the
fact is another leak).
## Self-assessment
1. You z-score `log_capitalizacion` for each stock with the mean and standard deviation of its 14 years of data, and its IC with next week's return goes from 0.000 to −0.049. What has happened?
- A. The normalization has brought to light a hidden size effect
- B. It is noise: with 400 stocks an IC of −0.049 is not significant
- C. The full-sample mean includes the stock's future prices: a stock that is going to rise has a market cap today below its future mean, and the feature reveals it
- D. Z-scoring changes the sign of rank correlations
*Answer C: The full-sample mean includes the stock's future prices: a stock that is going to rise has a market cap today below its future mean, and the feature reveals it.* The IC of the oracle, the best possible forecast, is 0.029. An IC of −0.049 with t = −15 on a feature with no effect can only come from information later than the date. With expanding mean and standard deviation, the IC goes back to 0.001.
2. Which test automatically detects that a normalization uses future data?
- A. The IC being higher than that of the unnormalized feature
- B. Cutting the data at a date and checking that the values before the cut do not change
- C. The normalized feature having mean 0 and standard deviation 1
- D. A cross-validation with shuffled k-fold
*Answer B: Cutting the data at a date and checking that the values before the cut do not change.* A transformation without the future gives, on date t, the same value with the data up to t as with all of it. The full-sample z-score fails that test; the expanding and rolling ones pass it.
3. In the walk-forward with ridge, fitting the `StandardScaler` on all years before splitting gives exactly the same out-of-sample IC as fitting it only on the training set. Why?
- A. Because the global scaler does not use test data
- B. Because ridge ignores the scale of the features
- C. Because the features were already ranks
- D. Because an affine transformation common to all rows does not change the order of a linear model's predictions: the leak exists but cannot be exploited. The harmful one is the one that varies with each row's future
*Answer D: Because an affine transformation common to all rows does not change the order of a linear model's predictions: the leak exists but cannot be exploited. The harmful one is the one that varies with each row's future.* The linear model offsets any common change of scale and origin with its coefficients. A per-stock mean over its whole history, on the other hand, puts into each row different information from after its date.
4. `valor_bp` has an IC of 0.0095 raw and 0.0017 z-scored with each stock's expanding history. Why does it drop if that normalization has no leak?
- A. Because the value effect is cross-sectional (cheap stocks versus expensive ones) and z-scoring each stock against its own past removes it: the feature ends up measuring whether the stock is cheap relative to itself
- B. Because the expanding window has a bug
- C. Because the first 52 weeks are lost
- D. Because value has no effect in the panel
*Answer A: Because the value effect is cross-sectional (cheap stocks versus expensive ones) and z-scoring each stock against its own past removes it: the feature ends up measuring whether the stock is cheap relative to itself.* Time-series normalization changes what the feature means. For a cross-sectional effect, the right normalization is the one across stocks within each week.
5. `revision_analistas` has an IC of 0.189 with the first session of the following week, 0.178 with the second and almost 0 with the third, fourth and fifth. Lagged one week, its IC is −0.002. What do you conclude?
- A. That it is a very fast signal that has to be traded at the open
- B. That cross-sectional normalization spoils it
- C. That the value is known after its date: it incorporates the first two days of the week to be predicted, and lagged to when it is really known it has no effect
- D. That purged k-fold would have removed it
*Answer C: That the value is known after its date: it incorporates the first two days of the week to be predicted, and lagged to when it is really known it has no effect.* A real effect decays slowly when lagged (value lagged one week keeps its IC). One that jumps from 0.17 to 0 with one week of lag and is concentrated in specific future sessions is a date leak, which no validation detects.
## Key points
- A mean, standard deviation, minimum or maximum computed over a stock's whole sample carries its future inside: in the panel, the full-history z-score gives a feature with no effect an IC of −0.049, above the oracle's ceiling (0.029).
- The alternatives without the future are expanding and rolling windows (`expanding`, `rolling` inside `groupby`). Any transformation is tested by cutting off the future: the values before the cut cannot change.
- Normalizing against the stock's own history changes the meaning of the feature: value stops being “cheap relative to the others” and its effect disappears.
- Cross-sectional transformations within each week (z-score, winsorization, rank, normal rank) remove market-wide level shifts and extremes; in the panel the normal rank gives the best out-of-sample IC (0.018 versus 0.016 raw), an improvement within the error.
- A global scaler fitted before splitting is a formal leak that a linear model cannot exploit; the harmful one varies with each row's future.
- No normalization fixes a feature dated before it is known. It is detected by measuring its IC session by session and lagging it: `revision_analistas` goes from 0.174 to −0.002.
## Resources
### Research
- [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min
### Documentation
- [scikit-learn: common pitfalls and recommended practices (leakage in preprocessing)](https://scikit-learn.org/stable/common_pitfalls.html) · scikit-learn.org
- [scikit-learn: preprocessing data (StandardScaler, QuantileTransformer)](https://scikit-learn.org/stable/modules/preprocessing.html) · scikit-learn.org
- [pandas: DataFrame.rolling](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.rolling.html) · pandas.pydata.org
- [pandas: DataFrame.expanding](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.expanding.html) · pandas.pydata.org
### Data
- [Level 5 data: panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet
- [Level 5 data: panel daily prices (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_precios.parquet) · Download · panel5_precios.parquet
Previous lesson: [5.2.3 · Fractional differentiation for features](https://pythonparatrading.com/en/python-trading-course/level-5/fractional-differentiation-for-features/)
Next lesson: [5.2.5 · Feature importance: MDI, MDA, SFI and SHAP](https://pythonparatrading.com/en/python-trading-course/level-5/feature-importance-mdi-mda-sfi-shap/)