---
title: "5.4.2 Purged k-fold with embargo and CPCV for models"
description: "Purging and embargo with each label's end date, checked row by row, and CPCV to compare a random forest with a logistic regression."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv.png"
---
# 5.4.2 · Purged k-fold with embargo and CPCV for ML models
> Lesson 2.5.3 programmed purging, embargo and combinatorial validation to choose a strategy's parameter. With a machine learning model on a panel three things change: each date has many rows, the purge has to come from the end dates of each label and each split trains a new model whose predictions have to be reassembled into paths. This lesson programs it, checks it against the data and uses it to compare a random forest with a linear baseline.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.4 · Validation for financial series
- Duration: 70 min
- Prerequisites: [5.4.1 · Why random k-fold fails on financial data](https://pythonparatrading.com/en/python-trading-course/level-5/why-random-k-fold-fails-finance/), [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/), [5.3.4 · Label uniqueness, overlap weights and the sequential bootstrap](https://pythonparatrading.com/en/python-trading-course/level-5/label-uniqueness-weights-sequential-bootstrap/)
## By the end of this lesson
- Program a purge and an embargo that work with the start and end dates of each label (`fecha` and `fin_sig_21d`) on a panel, with the interface of scikit-learn splitters.
- Check, with an independent function, that no training row overlaps the test and that the embargo leaves the requested gap, and measure how much data each embargo costs.
- Build a CPCV for models: train in each split, store the predictions by group and reassemble paths with classification and ranking metrics (hit rate and IC).
- Compare a random forest with a logistic regression path by path, against the oracle's ceiling, and see that CPCV does not detect a leaky feature either.
## What changes with respect to 2.5.3
Lesson 2.5.3 worked with a single series: one row per session, label intervals by position (`t0`, `t1`) and a CPCV
that chose the best of eight lookbacks. ML on a panel complicates all three pieces:
| Piece | Strategy on a series (2.5.3) | Model on a panel (this lesson) |
|---|---|---|
| Rows per date | one | 80 (one per stock) |
| Label interval | positions `t + 1` to `t + H` | dates `fecha` and `fin_sig_21d` from the file |
| What is trained in each split | nothing: a parameter is chosen | a new model |
| What is kept from the test | returns of the chosen variant | predictions (probabilities) per row |
| Metric of each path | Sharpe | hit rate and weekly rank IC |
The panel is the one of lesson 5.4.1 (80 stocks, 725 weeks, label “above the week's median at 21 sessions”) and the
model, the same random forest. Synthetic data with a known answer: they are for measuring the method.
```python
import zlib
from itertools import combinations
from math import comb
import numpy as np
import pandas as pd
from scipy.stats import norm
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (TimeSeriesSplit,
cross_val_score)
feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
d = feat.merge(etiq, on=["fecha", "ticker"])
completa = (d.groupby("fecha")[["ret_1s", "ret_sig_21d"]]
.transform(lambda s: s.notna().all()).all(axis=1))
d = d[completa].copy()
d["med_21d"] = d.groupby("fecha")["ret_sig_21d"].transform("median")
tickers = sorted(d["ticker"].unique())
rng = np.random.default_rng([2029, zlib.crc32(b"n5_panel_cv")])
elegidas = [tickers[i] for i in
np.sort(rng.choice(400, 80, replace=False))]
d = (d[d["ticker"].isin(elegidas)]
.sort_values(["fecha", "ticker"]).reset_index(drop=True))
FEATS = [c for c in feat.columns if c not in
("fecha", "ticker", "sector", "revision_analistas")]
X = np.nan_to_num(d[FEATS].to_numpy(dtype=float))
X_fuga = np.nan_to_num(
d[FEATS + ["revision_analistas"]].to_numpy(dtype=float))
y = (d["ret_sig_21d"] > d["med_21d"]).astype(int).to_numpy()
t0 = d["fecha"].to_numpy() # the row becomes known
t1 = d["fin_sig_21d"].to_numpy() # its label ends
fechas = np.unique(t0) # the 725 feature dates
sesiones = pd.DatetimeIndex(pd.read_parquet(
"panel5_precios.parquet", columns=["fecha"])["fecha"].unique())
sesiones = sesiones.sort_values()
n_ses = sesiones.searchsorted(t1) - sesiones.searchsorted(t0)
n_fechas = (np.searchsorted(fechas, t1, side="right")
- np.searchsorted(fechas, t0))
print("sessions per label:", np.unique(n_ses))
print("feature dates inside each label:")
print(pd.Series(n_fechas).value_counts().sort_index().to_string())
seis = np.unique(t0[n_fechas == 6])
print("labels covering 6 dates:", seis.astype("datetime64[D]"))
```
```output
sessions per label: [21]
feature dates inside each label:
1 80
2 80
3 80
4 80
5 57600
6 80
labels covering 6 dates: ['2024-12-20']
```
All labels measure exactly 21 sessions. In feature dates, almost all of them (57,600) cover their own and the next
four: the README's “4 weeks” rule is correct for them. But 80 rows, those of December 20, 2024 (with Christmas and New
Year in the following weeks), cover 6; and the last rows of the sample cover fewer because there are no more feature
dates after them. A purge written in weeks would fail silently in the first case. Written with `fin_sig_21d`, it
cannot fail.
## Purge and embargo with dates
The rule is that of 2.5.3, in dates. For each test **segment** (a set of contiguous feature dates), its interval runs
from the first `fecha` to the last `fin_sig_21d` of its rows. Out of training:
- **Purge**: every row whose interval `[fecha, fin_sig_21d]` overlaps that of the segment.
- **Embargo**: in addition, the rows that start within the `embargo` sessions after the end of the segment. Their
label no longer overlaps, but their long-window features (`ret_1m`, `ret_3m`, `mom_12_1`) were computed with returns
that are part of the test labels.
$$
\text{out}_i \iff t_{0,i} \le \text{end}_b + e \;\land\; t_{1,i} \ge \text{start}_b \quad\text{for some segment } b
$$
with $e$ the embargo in sessions. A test set can have several segments (in the CPCV, two separate groups), so the
function separates them first.
```python
def tramos(prueba):
"""(start, end) of each segment of contiguous dates in `prueba`."""
f = np.unique(t0[prueba])
pos = np.searchsorted(fechas, f)
for bloque in np.split(f, np.flatnonzero(np.diff(pos) > 1) + 1):
en = (t0[prueba] >= bloque[0]) & (t0[prueba] <= bloque[-1])
yield t0[prueba][en].min(), t1[prueba][en].max()
def purgar(prueba, embargo=0):
"""Training positions: out go the test, the rows that overlap
a segment and those in the embargo after each segment."""
fuera = np.zeros(len(t0), bool)
fuera[prueba] = True
for ini, fin in tramos(prueba):
j = min(sesiones.searchsorted(fin) + embargo,
len(sesiones) - 1)
fuera |= (t0 <= sesiones[j]) & (t1 >= ini)
return np.flatnonzero(~fuera)
class PurgedKFold:
"""k-fold by date blocks with purge and embargo. It has
split() and get_n_splits(): it works as cv= in scikit-learn."""
def __init__(self, n_splits=5, embargo=0):
self.n_splits, self.embargo = n_splits, embargo
def get_n_splits(self, X=None, y=None, groups=None):
return self.n_splits
def split(self, X=None, y=None, groups=None):
for b in np.array_split(fechas, self.n_splits):
prueba = np.flatnonzero(np.isin(t0, b))
yield purgar(prueba, self.embargo), prueba
```
The class does not store the data: it reads `t0`, `t1`, `fechas` and `sesiones` from the environment, just enough for
the lesson. In a project, those four arrays would go in the constructor (as in the `PurgedKFold` of López de Prado's
chapter 7, which receives the series of label ends).
## Checking the purge against the data
A function that purges and another that checks, written in a different way, keep watch on each other. The check does
not use `tramos`: for each test **date** it takes the latest label end of its rows and looks for training rows that
overlap that interval. It also measures the embargo gap: how many sessions there are between the last label end of
the test and the start of the first later training row.
```python
def comprobar(entreno, prueba):
"""(training rows overlapping the test, sessions between the
last label end of the test and the next training row)."""
fin_por_fecha = pd.Series(t1[prueba]).groupby(t0[prueba]).max()
solapes = np.zeros(len(entreno), bool)
for f, fin in fin_por_fecha.items():
solapes |= (t0[entreno] <= fin) & (t1[entreno] >= f)
fin_prueba = t1[prueba].max()
despues = t0[entreno][t0[entreno] > fin_prueba]
hueco = (sesiones.searchsorted(despues.min())
- sesiones.searchsorted(fin_prueba)
if len(despues) else None)
return int(solapes.sum()), hueco
# the README's rule: 4 weeks on each side of the block
semana = np.searchsorted(fechas, t0)
regla_4s = []
for b in np.array_split(np.arange(len(fechas)), 5):
lejos = (semana < b[0] - 4) | (semana > b[-1] + 4)
regla_4s.append((np.flatnonzero(lejos),
np.flatnonzero(np.isin(semana, b))))
filas = []
for nombre, div in [
("4-week rule", regla_4s),
("purge by dates", list(PurgedKFold(5).split())),
("purge + embargo 21", list(PurgedKFold(5, 21).split()))]:
for i, (ent, pru) in enumerate(div):
sol, hueco = comprobar(ent, pru)
filas.append((nombre, i, len(ent), sol, hueco))
pd.DataFrame(filas, columns=["esquema", "pliegue", "entreno",
"solapes", "gap (ses.)"]
).set_index(["esquema", "pliegue"])
```
```output
entreno solapes gap (ses.)
esquema pliegue
4-week rule 0 46080 0 4.0
1 45760 0 4.0
2 45760 0 3.0
3 45760 0 3.0
4 46080 0 NaN
purge by dates 0 46080 0 4.0
1 45760 0 4.0
2 45760 0 3.0
3 45760 0 3.0
4 46080 0 NaN
purge + embargo 21 0 45760 0 23.0
1 45440 0 23.0
2 45440 0 22.0
3 45440 0 22.0
4 46080 0 NaN
```
The purge by dates leaves **zero** overlaps in the five folds, and the 21-session embargo shows up in the last column
as the gap it should be. The README's 4-week rule also gives zero here, because no boundary of the five blocks falls
in the holiday week: it is correct by luck, not by construction. Note also fold 4, the last one: it has no training
after it, so the gap does not exist.
*Fig. — Date-based purge at the boundary of the third test block (4 August 2017): the four weekly rows whose 21-session label reaches the test set leave the training set.* Eight horizontal bars, one per weekly date from 23 June to 11 August 2017, running from the date to the end of its label. The 23 and 30 June rows end on 25 July and 1 August, before the block, and stay in training. The 7 to 28 July rows end between 7 and 28 August, inside the block, and are purged. The 4 and 11 August rows (highlighted) are test rows.
The figure shows, for one stock, the start of the third test block (fold 2, which starts on August 4, 2017) with the
purge by dates: the four weeks before the block leave training because their label reaches the first test date, and
the four after it, because they start before the last test label ends. What is set aside is what overlaps, no more
and no less.
## How much the embargo costs
The embargo is not free: each session of embargo is one week of 80 rows fewer to train on at each later boundary. The
longest feature window is that of momentum, 252 sessions. Covering it in full would be the safest and the most
expensive option:
```python
sin_purga = sum(len(t0) - len(p) for _, p in PurgedKFold(5).split())
coste = {}
for e in (0, 5, 21, 63, 252):
n = sum(len(ent) for ent, _ in PurgedKFold(5, e).split())
coste[e] = {"training rows": n,
"perdidas": f"{1 - n / sin_purga:.1%}"}
pd.DataFrame(coste).T.rename_axis("embargo (sessions)")
```
```output
training rows perdidas
embargo (sessions)
0 229440 1.1%
5 229120 1.2%
21 228160 1.7%
63 225280 2.9%
252 212800 8.3%
```
*Fig. — Share of training rows lost in a 5-block purged k-fold by embargo length, on the level 5 panel.* Horizontal bars. Purge only: 1.1%. 5-session embargo: 1.2%. 21: 1.7%. 63 (the one used in the lesson, highlighted): 2.9%. 252, the momentum window: 8.3%.
The purge alone costs 1.1% of training. A 21-session embargo (one month) raises the loss to 1.7%; one of 63 (three
months, the window of `ret_3m`), to 2.9%; covering the year of momentum, to 8.3%. With more folds there are more
boundaries and the cost multiplies: in the CPCV below, each split has up to two test segments with their embargo.
López de Prado's recommendation, a small embargo on the order of 1% of the sample, is a compromise: with 725 weeks,
about 7 weeks. Here 63 sessions will be used, which cover all windows except the 252-session ones.
## What purge and embargo change in the hit rate
With block folds, the gross leak of shuffled k-fold is gone (lesson 5.4.1). What remains is the leak at the
**boundaries**: the first and last weeks of each test block have overlapping neighbors in training if there is no
purge. That is why, in addition to the hit rate of the whole fold, the hit rate of the boundary rows (the first 4 and
the last 4 weeks of each block) is measured.
```python
def bosque():
return RandomForestClassifier(
n_estimators=60, min_samples_leaf=5, max_features="sqrt",
max_samples=0.3, n_jobs=4, random_state=0)
bloques = [(np.flatnonzero(~np.isin(t0, b)),
np.flatnonzero(np.isin(t0, b)))
for b in np.array_split(fechas, 5)]
res = {}
for nombre, div in [("blocks, no purge", bloques),
("purge", list(PurgedKFold(5).split())),
("purge + embargo 63",
list(PurgedKFold(5, 63).split()))]:
total, borde = [], []
for ent, pru in div:
ok = bosque().fit(X[ent], y[ent]).predict(X[pru]) == y[pru]
s = semana[pru]
en_borde = (s < s.min() + 4) | (s > s.max() - 4)
total.append(ok.mean())
borde.append(ok[en_borde].mean())
res[nombre] = {"acierto": np.mean(total),
"hit rate at boundaries": np.mean(borde)}
pd.DataFrame(res).T.map(lambda a: f"{a:.1%}")
```
```output
acierto hit rate at boundaries
blocks, no purge 50.0% 50.8%
purge 50.1% 49.2%
purge + embargo 63 50.1% 49.3%
```
Over each fold as a whole, the three versions give the same result: 50.0-50.1%. At the boundaries, without purge the
forest's hit rate is 50.8% and with purge 49.2%: it is the trace of the leak that remains at the edges, which the
average dilutes because the boundaries are only 8 of the 145 weeks of each block. The difference is on the order of
the noise of these figures (about 12,800 boundary rows, heavily overlapping with each other), so it cannot be read as
a precise measurement; what is certain is the direction, and that it grows with the number of folds, because there
are more boundaries.
## A custom splitter inside scikit-learn
Since `PurgedKFold` has `split` and `get_n_splits`, any scikit-learn function that accepts `cv=` uses it as is:
`cross_val_score`, `cross_validate`, `GridSearchCV`. It serves to introduce the **baseline**: a logistic regression on
the features as normal ranks within each week (the same treatment as lesson 5.2.4), with heavy regularization. It is
fast and linear: if the forest does not beat it, the forest adds nothing.
```python
rango = d.groupby("fecha")[FEATS].rank()
n_sem = d.groupby("fecha")[FEATS].transform("count")
G = np.nan_to_num(norm.ppf((rango - 0.5) / n_sem))
def logistica():
return LogisticRegression(C=0.01, max_iter=500)
cv = PurgedKFold(5, embargo=63)
for nombre, modelo, Xm in [("logistic (ranks)", logistica(), G),
("random forest", bosque(), X)]:
a = cross_val_score(modelo, Xm, y, cv=cv, scoring="accuracy")
print(f"{nombre:<20}hit rate {a.mean():.2%} "
f"(folds {np.round(a, 3).tolist()})")
```
```output
logistic (ranks) hit rate 50.55% (folds [0.51, 0.503, 0.503, 0.505, 0.507])
random forest hit rate 50.07% (folds [0.502, 0.505, 0.503, 0.491, 0.503])
```
The logistic regression gets somewhat more right than the forest, with the same validation. Both figures are close to
50%, as they should be with a ceiling of 51.1% (lesson 5.4.1), and the difference, a few tenths, is on the order of
the noise. To know whether it is consistent, more than one number is needed: that is what CPCV provides.
## CPCV for a machine learning model
The CPCV of 2.5.3 chose a parameter with the training data and applied the chosen variant to the test groups. With a
model, the procedure is:
1. Split the dates into N contiguous groups.
2. For each of the $\binom{N}{k}$ combinations of k test groups: purge and embargo around each group (the `purgar`
function already separates the segments), **train a new model** on the rest and store its predictions for each
test group separately.
3. Reassemble $\binom{N-1}{k-1}$ paths: for each group, the prediction from its j-th appearance in the test.
4. Measure each path with the problem's metrics: here the hit rate and the **IC** (rank correlation between the
probability and the 21-session return, week by week, and its mean).
```mermaid
flowchart TB
F[Dates in N groups] --> C[C N k splits]
C --> P[Purge and embargo]
P --> E[Train a model]
E --> G[Predict groups]
G --> R[Reassemble paths]
R --> M[IC per path]
```
With N = 6 and k = 2 there are 15 splits, 15 models and 5 paths; each model trains on about 4/6 of the dates, minus
the purge and embargo around the two test groups.
```python
N, K = 6, 2
grupos = np.array_split(fechas, N)
divisiones = list(combinations(range(N), K))
fila_grupo = [np.flatnonzero(np.isin(t0, g)) for g in grupos]
def cpcv(modelo, Xm, embargo=63):
"""{(split, group): probabilities of the group's rows}."""
pred = {}
for i, prueba_g in enumerate(divisiones):
prueba = np.concatenate([fila_grupo[g] for g in prueba_g])
ent = purgar(prueba, embargo)
m = modelo().fit(Xm[ent], y[ent])
for g in prueba_g:
pred[(i, g)] = m.predict_proba(Xm[fila_grupo[g]])[:, 1]
return pred
def ic_semanal(p, filas):
"""Mean of the weekly rank correlation between p and the
21-session return."""
df = pd.DataFrame({"s": semana[filas], "p": p,
"r": d["ret_sig_21d"].to_numpy()[filas]})
rk = df.groupby("s")[["p", "r"]].rank()
return rk.groupby(df["s"]).corr().xs("p", level=1)["r"].mean()
def caminos(pred):
filas = []
for j in range(comb(N - 1, K - 1)):
p = np.concatenate([
pred[([i for i, dv in enumerate(divisiones)
if g in dv][j], g)] for g in range(N)])
todas = np.concatenate(fila_grupo)
filas.append({"acierto": ((p > 0.5) == y[todas]).mean(),
"IC": ic_semanal(p, todas)})
return pd.DataFrame(filas).rename_axis("camino")
tabla = pd.concat({
"logistic": caminos(cpcv(logistica, G)),
"forest": caminos(cpcv(bosque, X)),
"forest + leak": caminos(cpcv(bosque, X_fuga))}, axis=1)
tabla.round(4)
```
```output
logistic forest forest + leak
acierto IC acierto IC acierto IC
camino
0 0.5041 0.0123 0.5009 -0.0011 0.5126 0.0442
1 0.5023 0.0105 0.5000 0.0008 0.5145 0.0451
2 0.5069 0.0190 0.5026 0.0040 0.5167 0.0503
3 0.5048 0.0148 0.5025 0.0137 0.5162 0.0499
4 0.5037 0.0125 0.5005 -0.0008 0.5123 0.0432
```
To read the table the reference is missing: the oracle's IC (the true alpha against the same 21-session return) on
the same 80 stocks. Since all paths cover all dates, it is the same for all five.
```python
verdad = pd.read_parquet("panel5_verdad.parquet",
columns=["fecha", "ticker", "alfa_verdadero"])
alfa = d[["fecha", "ticker"]].merge(verdad)["alfa_verdadero"]
ic_oraculo = ic_semanal(alfa.to_numpy(), np.arange(len(d)))
print(f"oracle IC: {ic_oraculo:.4f}")
dif = tabla["forest"]["IC"] - tabla["logistic"]["IC"]
print("forest minus logistic, per path:",
dif.round(4).tolist())
print(f"paths won by the logistic regression: {(dif < 0).sum()}/5")
```
```output
oracle IC: 0.0270
forest minus logistic, per path: [-0.0135, -0.0096, -0.015, -0.001, -0.0134]
paths won by the logistic regression: 5/5
```
The header figure puts the fifteen numbers on one scale. Three readings:
- **The baseline wins.** The logistic regression on ranks has an IC of 0.011 to 0.019 per path and the forest, of
−0.001 to 0.014. The comparison is paired (same splits, same test rows) and the logistic regression comes out ahead
on all five paths. With this signal-to-noise ratio, the forest's flexibility is spent on noise; lesson 5.5.2 comes
back to how to regularize it.
- **Nobody goes past the ceiling.** The oracle gives the maximum achievable IC; the two honest models stay below it,
as they should. The logistic regression captures about half of it.
- **The leak wins on every path.** With `revision_analistas`, the forest has an IC of 0.043 to 0.050 and a hit rate of
51.2-51.7%, above the oracle on all five. CPCV, with purge and embargo, cannot see it: the information from the
future is inside the feature (lesson 5.2.2).
## Hyperparameters inside CPCV
If the forest's hyperparameters were chosen by looking at the CPCV paths, the paths would stop being out of sample:
they would be the set used to choose. There are two correct ways out:
- **Nested validation**: inside each CPCV split, a `PurgedKFold` on the training data chooses the hyperparameters
(for example, with `GridSearchCV(cv=PurgedKFold(…))`), and the chosen model predicts the test groups. It costs
(number of splits) × (combinations) × (inner folds) trainings.
- **Counting the trials**: if several configurations are tried and the best is chosen by the paths, all of them are
recorded and the result is discounted with the Deflated Sharpe or the PBO (lessons 2.5.4 and 5.4.4).
And the uniqueness weights of lesson 5.3.4 go into the same loop: `fit(X[ent], y[ent], sample_weight=w[ent])`, with
the weights computed **only** with the training labels.
## Exercises
1. Check `TimeSeriesSplit` with `comprobar`: five folds without a gap and with `gap=4 * 80`. How many training rows
overlap the test in each case?
**Solution.**
```python
for gap in (0, 4 * 80):
tss = TimeSeriesSplit(5, gap=gap)
sol = [comprobar(a, b)[0] for a, b in tss.split(X)]
print(f"gap={gap:>3}: overlaps per fold {sol}")
```
```output
gap= 0: overlaps per fold [390, 376, 362, 348, 334]
gap=320: overlaps per fold [70, 56, 42, 28, 14]
```
Without a gap, each fold leaves in training between 334 and 390 rows whose labels reach the test: a little more than
four weeks of 80 stocks. With `gap=320` (four weeks in rows) the overlaps go down, but they do not disappear: between
14 and 70 remain. The reason is that `TimeSeriesSplit` cuts by rows: each test block has 58,000 // 6 = 9,666 rows,
which is not a multiple of 80, so the cut falls in the middle of a date and part of the stocks of the boundary week
stay in training with a label that reaches the test. With an unbalanced panel (stocks that enter and leave) it would
be worse. A gap in rows is not a gap in dates; the purge by `fin_sig_21d` is.
2. Compute, without training anything, how many models and paths there are with N = 8 and k = 2, and what average
fraction of the rows is left for training after purging and embargoing 63 sessions around the two test groups.
**Solution.**
```python
N8, K8 = 8, 2
grupos8 = np.array_split(fechas, N8)
fracs = []
for dv in combinations(range(N8), K8):
prueba = np.flatnonzero(np.isin(t0, np.concatenate(
[grupos8[g] for g in dv])))
fracs.append(len(purgar(prueba, 63)) / len(t0))
print(f"models {comb(N8, K8)}, paths {comb(N8 - 1, K8 - 1)}, "
f"mean training {np.mean(fracs):.1%} "
f"(no purge {(N8 - K8) / N8:.1%})")
```
```output
models 28, paths 7, mean training 70.7% (no purge 75.0%)
```
With N = 8, 28 models are needed for 7 paths, almost twice as many trainings as with N = 6 for two more paths. Each
model trains on about three quarters of the rows, somewhat less because of the purge and embargo around each test
group (two segments, or one if the groups are contiguous). With expensive models, like the networks of modules 5.6
and 5.7, small N and k are the practical option.
3. A colleague chooses the number of trees and the leaf size of the forest with the five CPCV paths (tries twelve
combinations and keeps the one with the best mean IC) and presents that IC as an out-of-sample result. What is
wrong and how would you fix it?
**Solution.**
The paths were used to choose: the IC of the best of twelve configurations is biased upward, just like the Sharpe of
the best of N backtests (lesson 2.5.1). Two fixes. The clean one is nesting: inside each of the 15 splits, choose the
configuration with an inner `PurgedKFold` on that split's training data, and predict the test groups with the chosen
one; the paths are out of sample again. The cheap one is to record the twelve configurations with their paths and
discount the selection: with the return series of the twelve, the Deflated Sharpe (lesson 5.4.4) or the PBO by CSCV
(lesson 2.5.4). In both cases, the result is compared with the logistic regression, which needs no tuning.
## Self-assessment
1. Why is the purge computed with `fin_sig_21d` and not with a fixed rule of “4 weeks on each side”?
- A. Because it is faster
- B. Because the horizon is 21 sessions, not weeks: with holidays a label can span the dates of 6 weekly rows instead of 5, and the fixed rule would leave that overlap in training
- C. Because scikit-learn requires it
- D. Because the fixed rule always removes too many rows
*Answer B: Because the horizon is 21 sessions, not weeks: with holidays a label can span the dates of 6 weekly rows instead of 5, and the fixed rule would leave that overlap in training.* On the panel, 57,600 of the 58,000 labels cover 5 feature dates and 80 cover 6. The end date is the truth; any rule in weeks is an approximation that can fail right at a boundary.
2. What does the embargo protect that the purge does not already protect?
- A. The training rows right after the test whose features were computed with test returns (for example `ret_1m` or `mom_12_1`), even though their label no longer overlaps
- B. Label overlap
- C. Test rows with an ambiguous label
- D. The `revision_analistas` leak
*Answer A: The training rows right after the test whose features were computed with test returns (for example `ret_1m` or `mom_12_1`), even though their label no longer overlaps.* The purge looks at the labels. Long-window features look backward and, after a test block, include returns that are part of its labels. The embargo sets those rows aside; covering the longest window (252 sessions) would cost a lot of data.
3. In the CPCV with N = 6 and k = 2, the logistic regression on ranks has a mean IC per path of 0.011 to 0.019 and the random forest, of −0.001 to 0.014. What do you conclude?
- A. That the random forest is better because it has more parameters
- B. That the five paths are five independent samples and the difference is conclusive as it stands
- C. That the linear baseline wins on all five paths: with this signal-to-noise ratio the forest does not exploit the signal, and the path-by-path comparison shows it with the same test data
- D. That neither beats chance
*Answer C: That the linear baseline wins on all five paths: with this signal-to-noise ratio the forest does not exploit the signal, and the path-by-path comparison shows it with the same test data.* The paths share data, so they are not independent evidence; but the comparison is paired (same rows, same splits) and the logistic regression comes out ahead on all of them. It is the baseline any complex model has to beat.
4. With `revision_analistas` the forest improves on every CPCV path (IC ≈ 0.045 against ≈ 0.004). What does this say about the feature?
- A. That it is the best feature in the panel
- B. That the purge does not work
- C. That the embargo has to be raised
- D. Nothing about whether it is legitimate: CPCV purges label overlap, but a feature that already contains information from after its date passes every validation. You have to audit when it is known
*Answer D: Nothing about whether it is legitimate: CPCV purges label overlap, but a feature that already contains information from after its date passes every validation. You have to audit when it is known.* The leak is inside each row: it uses the first two days of the following week. Neither the purge nor the embargo can see it. It is detected with the point-in-time audit (lesson 5.2.2).
5. How many paths and how many models have to be trained in a CPCV with N = 8 groups and k = 2?
- A. 8 paths and 8 models
- B. 28 paths and 7 models
- C. 7 paths and 28 models
- D. 2 paths and 8 models
*Answer C: 7 paths and 28 models.* There are C(8, 2) = 28 splits, each with its own model, and C(7, 1) = 7 paths. With expensive models, the number of trainings grows much faster than the number of paths.
## Key points
- On a panel the purge is defined by dates: out of training goes every row whose label `[fecha, fin_sig_21d]` overlaps a test segment. A rule in weeks is only an approximation.
- An independent checking function (0 overlaps, embargo gap measured in sessions) is worth more than trust in the code that purges.
- The embargo costs data: with 5 blocks, the purge alone sets aside 1.1% of training, a 63-session embargo 2.9% and one that covers the 252-session momentum, 8.3%. It is chosen knowing which features have long windows.
- Without shuffling, purge and embargo change this forest's mean hit rate little (50.0% against 50.1%): the leak that remained was at the boundaries.
- CPCV for models trains one per split, stores each group's predictions and reassembles paths. With N = 6 and k = 2: 15 models and 5 paths.
- The logistic regression on ranks beats the random forest on all five paths and the forest with `revision_analistas` wins on all of them: CPCV compares models, but does not detect a misdated feature.
## Resources
### Research
- [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min
### Documentation
- [scikit-learn: cross-validation and custom iterators (cv accepts an object with split and get_n_splits)](https://scikit-learn.org/stable/modules/cross_validation.html) · scikit-learn.org
- [scikit-learn: cross_val_score](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) · scikit-learn.org
- [scikit-learn: TimeSeriesSplit (gap parameter)](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) · scikit-learn.org
### Books
- [López de Prado, M. (2018): Advances in Financial Machine Learning. Wiley (chapters 7 and 12: purged k-fold, embargo and CPCV)](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com
### Data
- [Level 5 data: labels with their end date (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_etiquetas.parquet) · Download · panel5_etiquetas.parquet
- [Level 5 data: true alpha of the panel (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_verdad.parquet) · Download · panel5_verdad.parquet
Previous lesson: [5.4.1 · Why random k-fold fails on financial data](https://pythonparatrading.com/en/python-trading-course/level-5/why-random-k-fold-fails-finance/)
Next lesson: [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/)