---
title: "5.4.1 Why random k-fold fails"
description: "A random forest scores 59.7% with shuffled k-fold and 50.1% with purging, on a panel whose ceiling is 51%: where the leak comes from."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/why-random-k-fold-fails-finance/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/why-random-k-fold-fails-finance.png"
---

# 5.4.1 · Why random k-fold fails on financial data

> Shuffling the rows before validating is the first thing any machine learning course teaches, and on a stock panel with 21-session labels it is the fastest way to manufacture a model that does not exist. This lesson reproduces the case in the level's README, measures its two ingredients and shows that the wrong validation does not only inflate the result: it also picks the wrong model.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.4 · Validation for financial series
- Duration: 60 min
- Prerequisites: [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/), [5.3.1 · Fixed-horizon labeling and its problems](https://pythonparatrading.com/en/python-trading-course/level-5/fixed-horizon-labeling/), [5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines](https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/)

## By the end of this lesson

- Reproduce the contrast on the level 5 panel: shuffled k-fold against blocked k-fold and purged k-fold, and place it against the oracle's ceiling.
- Measure the two ingredients of the leak: label overlap (agreement of the label with that of the same stock k weeks earlier) and feature persistence.
- Check that grouping by date is not enough on a panel, because the leak comes from neighboring weeks, not from neighboring stocks.
- See how a leaky validation rewards the model that memorizes most and why an honest validation does not distinguish between hyperparameters when there is no signal to learn.

## A hit rate that cannot exist

Lesson 2.5.3 measured the leak of shuffled k-fold on a series of pure noise: an AUC of 0.93 where there was nothing to
predict. Here the problem is that of a real machine learning model: a panel of 400 stocks with 22 weekly features, a
small planted signal and a classification question like those of 5.3.1, “will this stock end up above its week's
median over the next 21 sessions?”. The README of the level's data publishes the result of a random forest with
three validations; this lesson reproduces it and takes it apart.

<aside class="callout" data-kind="aviso">

`panel5_*` are synthetic data with a known answer: the alpha of each stock and week is in `panel5_verdad.parquet`,
and the level's README describes the model that generates it. They are for measuring methods, not for searching for
strategies.

</aside>

The set is the README's: 80 fixed stocks (chosen with the generator's seed) over the 725 weeks that have features and
a complete 21-session label, 21 features without `revision_analistas` (the planted leak, left for the end) and a
binary label relative to the median of the 400 stocks of that week. Being relative, the common market move disappears
from the answer: what is asked is to rank stocks, not to guess the index.

```python
import zlib
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold

feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
d = feat.merge(etiq, on=["fecha", "ticker"])

# complete weeks: features and 21-session label for all 400
completa = (d.groupby("fecha")[["ret_1s", "ret_sig_21d"]]
            .transform(lambda s: s.notna().all()).all(axis=1))
d = d[completa].copy()
for col, med in (("ret_sig_21d", "med_21d"),
                 ("ret_sig_semana", "med_sem")):
    d[med] = d.groupby("fecha")[col].transform("median")

# the README's 80 stocks (the generator's seed)
tickers = sorted(d["ticker"].unique())
rng = np.random.default_rng([2029, zlib.crc32(b"n5_panel_cv")])
elegidas = [tickers[i] for i in
            np.sort(rng.choice(400, 80, replace=False))]
d = (d[d["ticker"].isin(elegidas)]
     .sort_values(["fecha", "ticker"]).reset_index(drop=True))

FEATS = [c for c in feat.columns if c not in
         ("fecha", "ticker", "sector", "revision_analistas")]
X = np.nan_to_num(d[FEATS].to_numpy(dtype=float))
y = (d["ret_sig_21d"] > d["med_21d"]).astype(int).to_numpy()
semana = d["fecha"].rank(method="dense").astype(int).to_numpy() - 1
print(f"{len(d)} rows = {d['ticker'].nunique()} stocks x "
      f"{semana.max() + 1} weeks; {len(FEATS)} features; "
      f"share of ones {y.mean():.3f}")
```
```output
58000 rows = 80 stocks x 725 weeks; 21 features; share of ones 0.498
```

The model is the README's: a forest of 60 trees with leaves of at least 5 observations, each tree trained on 30% of
the rows. It fixes `random_state` and the number of threads, so the result is the same on every run. The three
validations, with 5 folds each:

- **shuffled k-fold**: `KFold(5, shuffle=True)`, the one in any tutorial.
- **blocked k-fold**: `KFold(5)` without shuffling; since the rows are sorted by date, each fold is 145 consecutive
  weeks.
- **purged k-fold**: the same blocks, removing from training the 4 weeks on each side of the test block (the README's
  rule; lesson 5.4.2 programs it properly, with the end dates of each label).

```python
def bosque(hoja=5):
    return RandomForestClassifier(
        n_estimators=60, min_samples_leaf=hoja, max_features="sqrt",
        max_samples=0.3, n_jobs=4, random_state=0)

def acierto(divisiones, Xv=X, yv=y, hoja=5):
    """Mean out-of-sample hit rate of the folds."""
    res = []
    for ent, pru in divisiones:
        m = bosque(hoja).fit(Xv[ent], yv[ent])
        res.append((m.predict(Xv[pru]) == yv[pru]).mean())
    return float(np.mean(res))

barajado = list(KFold(5, shuffle=True, random_state=0).split(X))
bloques = list(KFold(5).split(X))
purgado = []
for b in np.array_split(np.arange(semana.max() + 1), 5):
    lejos = (semana < b[0] - 4) | (semana > b[-1] + 4)
    purgado.append((np.flatnonzero(lejos),
                    np.flatnonzero(np.isin(semana, b))))

res = {"shuffled k-fold": acierto(barajado),
       "blocked k-fold": acierto(bloques),
       "purged k-fold": acierto(purgado)}
for nombre, a in res.items():
    print(f"{nombre:<20}{a:.1%}")
```
```output
shuffled k-fold     59.7%
blocked k-fold      50.0%
purged k-fold       50.1%
```

Shuffled, the forest gets almost six out of ten right; with blocks, half, like a coin. The question is which of the
two figures is the truth, and on this panel there is no need to trust either: the ceiling can be computed.

## The ceiling: what the oracle would get right

`panel5_verdad.parquet` contains `alfa_verdadero`, the extra expected return of each stock and week due to the
planted effects. It is the best possible forecast of the predictable part of the return: a perfect model would
recover all of it and could do nothing more, because the rest is noise. If each stock is classified above or below the
median alpha of its week, the hit rate against the label is the maximum any model can reach out of sample (with the
same label precision: the 21-session label also picks up alphas from later weeks, which this week's oracle does not
see, so it is an approximate ceiling).

```python
verdad = pd.read_parquet("panel5_verdad.parquet",
                         columns=["fecha", "ticker", "alfa_verdadero"])
verdad["med_alfa"] = (verdad.groupby("fecha")["alfa_verdadero"]
                      .transform("median"))
o = d[["fecha", "ticker"]].merge(verdad, on=["fecha", "ticker"])
acierta = ((o["alfa_verdadero"] > o["med_alfa"]).to_numpy() == y)
reciente = (d["fecha"] >= "2019-01-01").to_numpy()
print(f"oracle: {acierta.mean():.1%} "
      f"(2019-2025: {acierta[reciente].mean():.1%})")
error_iid = np.sqrt(0.25 / len(y))
print(f"standard error if the rows were independent: "
      f"{error_iid:.2%}")
```
```output
oracle: 51.1% (2019-2025: 50.5%)
standard error if the rows were independent: 0.21%
```

The oracle's hit rate is 51.1%. It is the figure that puts everything in its place: the 59.7% of shuffled k-fold is
eight points above what the existing information allows, with a standard error (if the rows were independent, which
they are not) of two tenths of a point. It is not an excellent model: it is a broken validation. The 50.0% and 50.1%
of the blocks, below the ceiling, are what the forest really achieves, which is almost nothing.

The header figure summarizes the four figures, with the ceiling as a reference.

<aside class="callout" data-kind="riesgo">

With real data there is no oracle, but there is an order of magnitude: hit rates of 51-53% and ICs of a few
hundredths are the norm in cross-sectional return prediction (lesson 5.1.1). A 60% weekly hit rate on stocks is an
alarm, not a result. Before celebrating it, look for the leak.

</aside>

## Anatomy of the overlap: five weeks per label

The label of a row is the return from the close of `fecha` to that of 21 sessions later, `fin_sig_21d`. The features
are weekly. Therefore, a Friday's label spans that week and the following four, and the following week's label shares
four of its five weeks with it:

```python
una = d[d["ticker"] == elegidas[0]].set_index("fecha")
tabla = una.loc["2019-03-01":"2019-03-29",
                ["fin_sig_21d", "ret_sig_21d", "med_21d"]].copy()
tabla["etiqueta"] = (tabla["ret_sig_21d"]
                     > tabla.pop("med_21d")).astype(int)
print(elegidas[0])
tabla.round({"ret_sig_21d": 4})
```
```output
BAKU
           fin_sig_21d  ret_sig_21d  etiqueta
fecha
2019-03-01  2019-04-01      -0.0860         0
2019-03-08  2019-04-08      -0.0042         1
2019-03-15  2019-04-15      -0.0771         0
2019-03-22  2019-04-23      -0.0905         0
2019-03-29  2019-04-30      -0.0156         1
```

*Fig. — Five consecutive 21-session labels of one panel stock (March 2019): each shares four of its five weeks with the next.* Five horizontal bars, one per row date from 1 to 29 March 2019, running from the row date to the end of its label 21 sessions later (1 April to 30 April). The 15 March one, highlighted, is the test row; the others, training rows, overlap it almost entirely. Labels: 0, 1, 0, 0, 1.

The horizon is counted in sessions, not days: the March 22 label ends on April 23 because the NYSE was closed on April
19, 2019 (Good Friday). The figure draws those five labels as intervals. If the March 15 one falls in the test set and
those of the two previous and following weeks in the training set, the forest has four rows of the same stock to
train on whose answer is computed mostly with the same returns. In this example two of the four have the same label
as the one of the 15th; across the whole panel, the proportion is much higher for adjacent weeks. How similar those
answers are is measured directly: the agreement of each row's label with that of the same stock k weeks earlier.

```python
y_sem = (d["ret_sig_semana"] > d["med_sem"]).astype(int).to_numpy()
etiquetas = pd.DataFrame({"ticker": d["ticker"],
                          "21 sessions": y, "1 week": y_sem})
acuerdo = {}
for k in range(1, 7):
    antes = etiquetas.groupby("ticker")[["21 sessions",
                                         "1 week"]].shift(k)
    hay = antes.notna().all(axis=1)
    acuerdo[k] = (antes[hay] == etiquetas.loc[hay, ["21 sessions",
                                                    "1 week"]]
                  ).mean()
pd.DataFrame(acuerdo).T.rename_axis("k").round(3)
```
```output
   21 sessions  1 week
k
1        0.777   0.496
2        0.680   0.503
3        0.601   0.501
4        0.528   0.502
5        0.502   0.501
6        0.504   0.498
```

*Fig. — Share of panel rows whose label matches the same stock's label k weeks earlier: 21-session (overlapping) label against the one-week label.* Two curves over k from 1 to 6 weeks. With the 21-session label, agreement is 77.7% with the previous week (highlighted), 68.0% at k = 2, 60.1% at k = 3, 52.8% at k = 4 and 50% from k = 5, when labels stop overlapping. With the one-week label, agreement is 50% from k = 1.

With the 21-session label, the row of the previous week has the same answer 77.7% of the time; two weeks earlier,
68%; the agreement drops to one half in the fifth week, when the two labels no longer share any return. With the
one-week label, which does not overlap, the agreement is 50% from the first week. It is the same stock and the same
features: the only thing that changes is whether the answer is shared.

## The second ingredient: features that barely change

For the model to exploit that agreement it has to **recognize** that the test row is the neighbor of a training row.
It manages to if the features of one week look a lot like those of the previous week. The weekly autocorrelation of
each feature, within each stock, measures it:

```python
persist = {c: d.groupby("ticker")[c]
           .apply(lambda s: s.autocorr(1)).mean() for c in FEATS}
persist = pd.Series(persist, name="weekly autocorr.")
print(persist.sort_values(ascending=False).round(2).to_string())
```
```output
valor_bp              0.99
log_capitalizacion    0.99
roe                   0.99
ey                    0.99
ruido_3               0.97
mom_12_1              0.97
vol_63d               0.97
dist_max_52s          0.96
sue                   0.94
ret_3m                0.92
ruido_2               0.89
vol_21d               0.88
ruido_5               0.77
ret_1m                0.77
rsi_14                0.63
dias_desde_informe    0.63
sentimiento           0.59
volumen_rel           0.46
ruido_4               0.33
ret_1s                0.03
ruido_1               0.00
```

*Fig. — Persistence of the 21 panel features: week-to-week autocorrelation within each stock.* Horizontal bars sorted from highest to lowest. valor_bp, log_capitalizacion, roe and ey: 0.99. ruido_3 (an AR(1) noise with no effect, highlighted), mom_12_1 and vol_63d: 0.97. dist_max_52s 0.96, sue 0.94, ret_3m 0.92, ruido_2 0.89, vol_21d 0.88, ruido_5 and ret_1m 0.77, rsi_14 and dias_desde_informe 0.63, sentimiento 0.59, volumen_rel 0.46, ruido_4 0.33, ret_1s 0.03 and ruido_1 0.00.

Almost all the features are slow: value, size, the accounting ratios and 12-month momentum have a weekly
autocorrelation of 0.94 to 0.99, and `ruido_3`, an AR(1) noise with no effect at all, 0.97. Together they form a
fingerprint that identifies each stock in each stretch of time. Only `ret_1s` (0.03) and `ruido_1` (0.00) are fully
renewed every week; `ruido_4` and `volumen_rel`, halfway. For a forest with small leaves, finding the neighbor is as
easy as recognizing a face is for a human.

```mermaid
flowchart TB
  A[Overlapping label] --> C[Same answer]
  B[Slow features] --> D[Recognizable neighbor]
  C --> E[Shuffled k-fold]
  D --> E
  E --> F[Inflated hit rate]
```

## The two ingredients, separately

If the explanation is correct, removing either of the two ingredients makes the leak disappear, and the second alone
is enough to manufacture a hit rate with columns that have no effect at all. Three experiments, always with shuffled
k-fold:

- the **one-week** label (no overlap) with the same 21 features;
- the 21-session label with **four persistent features with no effect**: `ruido_2`, `ruido_3`, `ruido_5` and
  `log_capitalizacion`;
- the 21-session label with **three fast features**: `ret_1s`, `ruido_1` and `volumen_rel`.

```python
def columnas(cols):
    return np.nan_to_num(d[cols].to_numpy(dtype=float))

lentas = ["ruido_2", "ruido_3", "ruido_5", "log_capitalizacion"]
rapidas = ["ret_1s", "ruido_1", "volumen_rel"]
exp = {
    "21 sessions, 21 features": res["shuffled k-fold"],
    "1 week, 21 features": acierto(barajado, yv=y_sem),
    "21 sessions, 4 slow, no effect":
        acierto(barajado, Xv=columnas(lentas)),
    "21 sessions, 3 fast": acierto(barajado, Xv=columnas(rapidas)),
}
for nombre, a in exp.items():
    print(f"{nombre:<34}{a:.1%}")
```
```output
21 sessions, 21 features          59.7%
1 week, 21 features               50.0%
21 sessions, 4 slow, no effect    53.5%
21 sessions, 3 fast               49.8%
```

| Label | Features | Shuffled k-fold | What is missing |
|---|---|---|---|
| 21 sessions | all 21 | 59.7% | nothing: full leak |
| 1 week | all 21 | 50.0% | label overlap |
| 21 sessions | 4 slow, no effect | 53.5% | the signal (not needed) |
| 21 sessions | 3 fast | 49.8% | persistence |

With the one-week label, shuffled k-fold becomes honest: 50.0%, the same result as with blocks. With fast features,
too. And with four columns that have no effect at all on returns (three noises and size, which on this panel earns no
premium), the forest reaches 53.5%: above the oracle, with no information whatsoever about the future. The leak does
not need a signal; it needs a fingerprint and a shared answer.

## Grouping by date is not enough

A panel has two dimensions, and the usual intuition is to protect the cross-sectional one: the 80 stocks of the same
week should not be split between training and test, because they share that week's market and sector moves. That is
what `GroupKFold` does with the week as the group: each whole week falls into a single fold. With `shuffle=True`, the
weeks are assigned to folds at random.

```python
gkf = GroupKFold(5, shuffle=True, random_state=0)
por_semana = list(gkf.split(X, y, groups=semana))
comunes = [np.intersect1d(semana[a], semana[b]).size
           for a, b in por_semana]
print(f"weeks on both sides, per fold: {comunes}")
print(f"GroupKFold by week, shuffled: {acierto(por_semana):.1%}")
```
```output
weeks on both sides, per fold: [0, 0, 0, 0, 0]
GroupKFold by week, shuffled: 59.2%
```

No week is on both sides, and the hit rate barely moves: 59.2%. The leak was not cross-sectional. The label is
relative to the week's median, so the common component was already out of the answer; what the forest exploits is the
**same stock** in the **surrounding weeks**, which still fall into training when the weeks are shuffled. On a
financial panel the group that matters is time, and not as a group label but as an order: the folds have to be
contiguous blocks, with a margin around them.

| Scheme | What it protects | Hit rate |
|---|---|---|
| `KFold(shuffle=True)` | nothing | 59.7% |
| `GroupKFold(shuffle=True)` by week | stocks of the same week | 59.2% |
| `KFold` by time blocks | almost everything; the boundary remains | 50.0% |
| blocks with a 4-week purge | the boundary as well | 50.1% |

<aside class="callout" data-kind="nota">

The cross-sectional dimension can also leak, in other designs: if the label were the absolute return (without
subtracting the median), the stocks of the same week would share the market move, and splitting them between training
and test would teach the model that week's index. Time blocks solve both things at once, because each whole week ends
up on one side.

</aside>

## The wrong validation picks the wrong model

The leak does not only inflate a figure. If the validation is used to choose hyperparameters (lesson 5.5.4), it picks
whatever exploits the leak best. The hyperparameter that controls how much a forest can memorize is the minimum leaf
size: with one-observation leaves, each tree can isolate a stock in a specific stretch of weeks. Four sizes, with the
two validations:

```python
hojas = {}
for hoja in (1, 5, 50, 500):
    if hoja == 5:  # already computed above
        hojas[hoja] = (res["shuffled k-fold"], res["purged k-fold"])
    else:
        hojas[hoja] = (acierto(barajado, hoja=hoja),
                       acierto(purgado, hoja=hoja))
pd.DataFrame(hojas, index=["shuffled", "purged"]).T.rename_axis(
    "min leaf").map(lambda a: f"{a:.1%}")
```
```output
         shuffled purged
min leaf
1           60.4%  50.3%
5           59.7%  50.1%
50          55.7%  50.2%
500         52.3%  50.2%
```

*Fig. — Accuracy of the panel random forest by minimum leaf size: shuffled validation rewards memorising; purged validation does not tell sizes apart.* Two curves over minimum leaf size on a log scale (1, 5, 50 and 500). Shuffled k-fold: 60.4% with leaves of 1 (highlighted), 59.7% with 5, 55.7% with 50 and 52.3% with 500. Purged k-fold: between 50.1% and 50.3%, flat, below the oracle line at 51.1%.

Shuffled k-fold has a clear, monotonic preference: the smaller the leaf, the higher the hit rate, from 52.3% with
leaves of 500 to 60.4% with leaves of 1. A hyperparameter search with that validation would pick the forest that
memorizes most and present it with 60%. Purged k-fold gives between 50.1% and 50.3% with all four sizes: none stands
out, because the forest does not find in these features a signal it can exploit out of sample. It is the right
answer, however disappointing, and the one to take seriously: lesson 5.5.1 shows that a linear regression with the
features as ranks does pick up part of the signal (IC of 0.0185 in the README), and 5.4.2 checks it with purged
validation.

## What to do in practice

1. **No shuffling of rows** in a problem with labels that span several periods or with persistent features, which in
   finance means almost all of them. Folds contiguous in time.
2. **Purge and embargo** around each test block with the real start and end dates of each label, not with an
   approximate rule (lesson 5.4.2).
3. **Walk-forward** for the history that would have been lived in real time, with retraining (lesson 5.4.3).
4. **Compare with a ceiling and with a baseline**: on synthetic data, the oracle; on real data, the order of magnitude
   in the literature and a simple linear model. A result far above both is a leak until proven otherwise.
5. **Audit when each piece of data is known** before validating anything. Purging protects against label overlap, not
   against a misdated feature.

<aside class="callout" data-kind="riesgo">

The level's README measures it with `revision_analistas`, a feature that the vendor dates at the start of the week
but that contains the first two days of the following one: with it, the same forest gives 51.5% in purged k-fold and
51.8% in walk-forward, above the oracle in both honest validations. No validation detects it, because the leak is
inside each row. It is discovered with the point-in-time audit of lesson 5.2.2.

</aside>

## Exercises

1. With the 21-session label, shuffled k-fold gives 53.5% with four slow features with no effect. Is a single one
   enough? Repeat it with only `log_capitalizacion` and with only `ruido_3`, and explain the difference with their
   persistence.

**Solution.**

```python
for col in ("log_capitalizacion", "ruido_3"):
    a = acierto(barajado, Xv=columnas([col]))
    print(f"{col:<20}autocorr. {persist[col]:.2f}  "
          f"shuffled {a:.1%}")
```
```output
log_capitalizacion  autocorr. 0.99  shuffled 51.1%
ruido_3             autocorr. 0.97  shuffled 50.4%
```

With a single column the fingerprint is much poorer: 51.1% with size and 50.4% with `ruido_3`. Many stocks have
similar values in one dimension and the forest cannot tell the neighbor apart from other rows with the opposite
answer. The leak grows with the number of persistent features because each one adds a coordinate to the fingerprint;
with four it already beats the oracle. That is why panels with many accounting and risk features, which barely
change, are the most exposed.

2. The 58,000 rows are not 58,000 independent observations. Estimate the effective number of independent labels (each
   stock has a new label roughly every five weeks) and the corresponding standard error of the hit rate. Does the
   conclusion about the 59.7% change?

**Solution.**

```python
n_ef = len(y) / 5  # one independent label every ~5 weeks
for nombre, n in (("rows", len(y)), ("effective", n_ef)):
    print(f"{nombre:<10} n = {n:>8,.0f}  standard error "
          f"{np.sqrt(0.25 / n):.2%}")
print(f"59.7% against the 51.1% ceiling: "
      f"{(0.597 - 0.511) / np.sqrt(0.25 / n_ef):.0f} standard errors")
```
```output
rows       n =   58,000  standard error 0.21%
effective  n =   11,600  standard error 0.46%
59.7% against the 51.1% ceiling: 19 standard errors
```

With about 11,600 effective labels, the standard error of the hit rate rises from 0.21% to 0.46%, and the
cross-sectional correlation is still missing (the stocks of a week are not independent of each other), which makes it
larger still. Even so, 59.7% against a ceiling of 51.1% is almost twenty standard errors: it is not bad luck. The
practical lesson is the reverse: in an honest validation, differences of a few tenths between models (50.1% against
50.3%) are noise.

3. A colleague presents a weekly stock classification model with a 57% hit rate validated with `TimeSeriesSplit`
   without `gap` and one-month labels. What three questions would you ask before believing them?

**Solution.**

First, whether the features are correctly dated: when each piece of data is known (publication date against period
date, revisions, vendors that date at the start of the week). It is the leak that no validation detects. Second, how
the overlap is handled: `TimeSeriesSplit` without `gap` leaves in training the last rows before each test, whose
one-month labels overlap with the first ones of the test; with small folds, that boundary weighs. A gap of at least
the label horizon is needed (lessons 5.4.2 and 5.4.3). Third, what it is compared with: the hit rate of a linear
baseline and that of a noise-only model (permuted columns) with the same validation. A weekly 57% on stocks is far
above what is published with real data; without those three answers, the reasonable thing is to assume a leak.

## Self-assessment

1. On the level's panel, the oracle (the true sign of the alpha) has a hit rate of 51.1% and a random forest with shuffled k-fold, 59.7%. What do you conclude?

   - A. That the random forest has found nonlinearities the oracle does not see
   - B. That the oracle is miscalculated
   - C. That with 58,000 rows any hit rate is significant
   - D. That the validation has a leak: no model can beat the true expectation out of sample, so 8 points above the ceiling are information about the answer

   *Answer D: That the validation has a leak: no model can beat the true expectation out of sample, so 8 points above the ceiling are information about the answer.* The oracle uses the true alpha, the best possible forecast of the predictable part. Beating it out of sample is impossible; with purging the same forest stays at 50.1%, below the ceiling, as it should.

2. With the one-week label (`ret_sig_semana`, no overlap) and the same features, shuffled k-fold gives 50.0%. What does it show?

   - A. That the one-week label has no information
   - B. That the leak needs overlapping labels: with independent labels, recognizing the same stock a week earlier says nothing about the answer
   - C. That the random forest does not work with weekly data
   - D. That shuffling has no effect at all in finance

   *Answer B: That the leak needs overlapping labels: with independent labels, recognizing the same stock a week earlier says nothing about the answer.* Feature persistence is the same in both cases. What changes is that the 21-session label matches that of the previous week 78% of the time and the one-week label, 50%.

3. Shuffled k-fold by rows is replaced with one shuffled by weeks (all the stocks of a week go together into the same fold). The hit rate barely falls, from 59.7% to 59.2%. Why?

   - A. Because the leak comes from the same stock in the neighboring weeks, which still fall into training: grouping by date removes the cross-sectional leak, not the temporal one
   - B. Because the leak came from the neighboring stocks in the same week, and grouping them does not touch it
   - C. Because GroupKFold does not shuffle
   - D. Because there are few weeks

   *Answer A: Because the leak comes from the same stock in the neighboring weeks, which still fall into training: grouping by date removes the cross-sectional leak, not the temporal one.* The label is relative to the week's median, so the common market component is already gone. What the forest recognizes is each stock's fingerprint in the surrounding weeks, and those are still on the other side of the boundary.

4. With shuffled k-fold, the best minimum leaf size is 1 (60.4%); with purged k-fold, all sizes give between 50.1% and 50.3%. Which reading is correct?

   - A. Leaves of 1 should be used, because it gives the best result in some validation
   - B. The two validations agree
   - C. The shuffled validation rewards memorizing; the honest one says that, with no signal the forest can exploit, the hyperparameter does not matter. Choosing with the first leads to the most overfit model
   - D. Purged k-fold is no use for choosing hyperparameters

   *Answer C: The shuffled validation rewards memorizing; the honest one says that, with no signal the forest can exploit, the hyperparameter does not matter. Choosing with the first leads to the most overfit model.* One-observation leaves make it possible to isolate each stock in each stretch of time, which is exactly what the leak exploits. Outside shuffled k-fold that capacity is worth nothing.

5. Which of these leaks is detected by NEITHER blocked k-fold NOR purged k-fold?

   - A. Label overlap between training and test
   - B. The memory of persistent features in shuffled k-fold
   - C. A feature dated before it is known (like `revision_analistas`, which uses the first two days of the following week)
   - D. The concentration of the leak at the fold boundaries

   *Answer C: A feature dated before it is known (like `revision_analistas`, which uses the first two days of the following week).* If a feature already carries information from after its date, each row drags it into any fold. It is only discovered by auditing when each piece of data is known (lesson 5.2.2); validation cannot see it.

## Key points

- On the level's panel (80 stocks, 725 weeks, 21-session label), a random forest gives 59.7% with shuffled k-fold, 50.0% with blocks and 50.1% with purging. The oracle's hit rate is 51.1%.
- A hit rate above the oracle is the signature of a leak. In financial ML the ceiling is low, and knowing it, even roughly, protects more than any metric.
- The leak needs two things at once: overlapping labels (the 21-session one matches that of the previous week 78% of the time) and persistent features (weekly autocorrelation of 0.97 to 0.99 in momentum, size or value).
- With four persistent columns with no effect at all, shuffled k-fold already gives 53.5%. With the one-week label, 50.0%.
- Grouping by date fixes nothing (59.2%): on a panel the leak is temporal, from the same stock in the neighboring weeks.
- The shuffled validation picks the smallest leaves, the model that memorizes most. With purging, no leaf size stands out: it is the right answer when there is no signal to learn.

## Resources

### Research

- [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min

### Documentation

- [scikit-learn: cross-validation (i.i.d. assumption, grouped data and time series)](https://scikit-learn.org/stable/modules/cross_validation.html) · scikit-learn.org
- [scikit-learn: GroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html) · scikit-learn.org
- [scikit-learn: RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) · scikit-learn.org

### Books

- [López de Prado, M. (2018): Advances in Financial Machine Learning. Wiley (chapter 7: cross-validation in finance)](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com

### Data

- [Level 5 data: weekly panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet
- [Level 5 data: labels with their end date (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_etiquetas.parquet) · Download · panel5_etiquetas.parquet

Previous lesson: [5.3.5 · Event-based sampling: the symmetric CUSUM filter](https://pythonparatrading.com/en/python-trading-course/level-5/event-sampling-cusum-filter/)

Next lesson: [5.4.2 · Purged k-fold with embargo and CPCV for ML models](https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/)