Why random k-fold fails on financial data.

A hit rate that cannot exist

Lesson 2.5.3 measured the leak of shuffled k-fold on a series of pure noise: an AUC of 0.93 where there was nothing to predict. Here the problem is that of a real machine learning model: a panel of 400 stocks with 22 weekly features, a small planted signal and a classification question like those of 5.3.1, “will this stock end up above its week's median over the next 21 sessions?”. The README of the level's data publishes the result of a random forest with three validations; this lesson reproduces it and takes it apart.

The set is the README's: 80 fixed stocks (chosen with the generator's seed) over the 725 weeks that have features and a complete 21-session label, 21 features without revision_analistas (the planted leak, left for the end) and a binary label relative to the median of the 400 stocks of that week. Being relative, the common market move disappears from the answer: what is asked is to rank stocks, not to guess the index.

import zlib
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold

feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
d = feat.merge(etiq, on=["fecha", "ticker"])

# complete weeks: features and 21-session label for all 400
completa = (d.groupby("fecha")[["ret_1s", "ret_sig_21d"]]
            .transform(lambda s: s.notna().all()).all(axis=1))
d = d[completa].copy()
for col, med in (("ret_sig_21d", "med_21d"),
                 ("ret_sig_semana", "med_sem")):
    d[med] = d.groupby("fecha")[col].transform("median")

# the README's 80 stocks (the generator's seed)
tickers = sorted(d["ticker"].unique())
rng = np.random.default_rng([2029, zlib.crc32(b"n5_panel_cv")])
elegidas = [tickers[i] for i in
            np.sort(rng.choice(400, 80, replace=False))]
d = (d[d["ticker"].isin(elegidas)]
     .sort_values(["fecha", "ticker"]).reset_index(drop=True))

FEATS = [c for c in feat.columns if c not in
         ("fecha", "ticker", "sector", "revision_analistas")]
X = np.nan_to_num(d[FEATS].to_numpy(dtype=float))
y = (d["ret_sig_21d"] > d["med_21d"]).astype(int).to_numpy()
semana = d["fecha"].rank(method="dense").astype(int).to_numpy() - 1
print(f"{len(d)} rows = {d['ticker'].nunique()} stocks x "
      f"{semana.max() + 1} weeks; {len(FEATS)} features; "
      f"share of ones {y.mean():.3f}")

58000 rows = 80 stocks x 725 weeks; 21 features; share of ones 0.498

The model is the README's: a forest of 60 trees with leaves of at least 5 observations, each tree trained on 30% of the rows. It fixes random_state and the number of threads, so the result is the same on every run. The three validations, with 5 folds each:

• shuffled k-fold: KFold(5, shuffle=True), the one in any tutorial.
• blocked k-fold: KFold(5) without shuffling; since the rows are sorted by date, each fold is 145 consecutive weeks.
• purged k-fold: the same blocks, removing from training the 4 weeks on each side of the test block (the README's rule; lesson 5.4.2 programs it properly, with the end dates of each label).

def bosque(hoja=5):
    return RandomForestClassifier(
        n_estimators=60, min_samples_leaf=hoja, max_features="sqrt",
        max_samples=0.3, n_jobs=4, random_state=0)

def acierto(divisiones, Xv=X, yv=y, hoja=5):
    """Mean out-of-sample hit rate of the folds."""
    res = []
    for ent, pru in divisiones:
        m = bosque(hoja).fit(Xv[ent], yv[ent])
        res.append((m.predict(Xv[pru]) == yv[pru]).mean())
    return float(np.mean(res))

barajado = list(KFold(5, shuffle=True, random_state=0).split(X))
bloques = list(KFold(5).split(X))
purgado = []
for b in np.array_split(np.arange(semana.max() + 1), 5):
    lejos = (semana < b[0] - 4) | (semana > b[-1] + 4)
    purgado.append((np.flatnonzero(lejos),
                    np.flatnonzero(np.isin(semana, b))))

res = {"shuffled k-fold": acierto(barajado),
       "blocked k-fold": acierto(bloques),
       "purged k-fold": acierto(purgado)}
for nombre, a in res.items():
    print(f"{nombre:<20}{a:.1%}")

shuffled k-fold     59.7%
blocked k-fold      50.0%
purged k-fold       50.1%

Shuffled, the forest gets almost six out of ten right; with blocks, half, like a coin. The question is which of the two figures is the truth, and on this panel there is no need to trust either: the ceiling can be computed.

The ceiling: what the oracle would get right

panel5_verdad.parquet contains alfa_verdadero, the extra expected return of each stock and week due to the planted effects. It is the best possible forecast of the predictable part of the return: a perfect model would recover all of it and could do nothing more, because the rest is noise. If each stock is classified above or below the median alpha of its week, the hit rate against the label is the maximum any model can reach out of sample (with the same label precision: the 21-session label also picks up alphas from later weeks, which this week's oracle does not see, so it is an approximate ceiling).

verdad = pd.read_parquet("panel5_verdad.parquet",
                         columns=["fecha", "ticker", "alfa_verdadero"])
verdad["med_alfa"] = (verdad.groupby("fecha")["alfa_verdadero"]
                      .transform("median"))
o = d[["fecha", "ticker"]].merge(verdad, on=["fecha", "ticker"])
acierta = ((o["alfa_verdadero"] > o["med_alfa"]).to_numpy() == y)
reciente = (d["fecha"] >= "2019-01-01").to_numpy()
print(f"oracle: {acierta.mean():.1%} "
      f"(2019-2025: {acierta[reciente].mean():.1%})")
error_iid = np.sqrt(0.25 / len(y))
print(f"standard error if the rows were independent: "
      f"{error_iid:.2%}")

oracle: 51.1% (2019-2025: 50.5%)
standard error if the rows were independent: 0.21%

The oracle's hit rate is 51.1%. It is the figure that puts everything in its place: the 59.7% of shuffled k-fold is eight points above what the existing information allows, with a standard error (if the rows were independent, which they are not) of two tenths of a point. It is not an excellent model: it is a broken validation. The 50.0% and 50.1% of the blocks, below the ceiling, are what the forest really achieves, which is almost nothing.

The header figure summarizes the four figures, with the ceiling as a reference.

Anatomy of the overlap: five weeks per label

The label of a row is the return from the close of fecha to that of 21 sessions later, fin_sig_21d. The features are weekly. Therefore, a Friday's label spans that week and the following four, and the following week's label shares four of its five weeks with it:

una = d[d["ticker"] == elegidas[0]].set_index("fecha")
tabla = una.loc["2019-03-01":"2019-03-29",
                ["fin_sig_21d", "ret_sig_21d", "med_21d"]].copy()
tabla["etiqueta"] = (tabla["ret_sig_21d"]
                     > tabla.pop("med_21d")).astype(int)
print(elegidas[0])
tabla.round({"ret_sig_21d": 4})

BAKU
           fin_sig_21d  ret_sig_21d  etiqueta
fecha
2019-03-01  2019-04-01      -0.0860         0
2019-03-08  2019-04-08      -0.0042         1
2019-03-15  2019-04-15      -0.0771         0
2019-03-22  2019-04-23      -0.0905         0
2019-03-29  2019-04-30      -0.0156         1

The horizon is counted in sessions, not days: the March 22 label ends on April 23 because the NYSE was closed on April 19, 2019 (Good Friday). The figure draws those five labels as intervals. If the March 15 one falls in the test set and those of the two previous and following weeks in the training set, the forest has four rows of the same stock to train on whose answer is computed mostly with the same returns. In this example two of the four have the same label as the one of the 15th; across the whole panel, the proportion is much higher for adjacent weeks. How similar those answers are is measured directly: the agreement of each row's label with that of the same stock k weeks earlier.

y_sem = (d["ret_sig_semana"] > d["med_sem"]).astype(int).to_numpy()
etiquetas = pd.DataFrame({"ticker": d["ticker"],
                          "21 sessions": y, "1 week": y_sem})
acuerdo = {}
for k in range(1, 7):
    antes = etiquetas.groupby("ticker")[["21 sessions",
                                         "1 week"]].shift(k)
    hay = antes.notna().all(axis=1)
    acuerdo[k] = (antes[hay] == etiquetas.loc[hay, ["21 sessions",
                                                    "1 week"]]
                  ).mean()
pd.DataFrame(acuerdo).T.rename_axis("k").round(3)

   21 sessions  1 week
k
1        0.777   0.496
2        0.680   0.503
3        0.601   0.501
4        0.528   0.502
5        0.502   0.501
6        0.504   0.498

With the 21-session label, the row of the previous week has the same answer 77.7% of the time; two weeks earlier, 68%; the agreement drops to one half in the fifth week, when the two labels no longer share any return. With the one-week label, which does not overlap, the agreement is 50% from the first week. It is the same stock and the same features: the only thing that changes is whether the answer is shared.

The second ingredient: features that barely change

For the model to exploit that agreement it has to recognize that the test row is the neighbor of a training row. It manages to if the features of one week look a lot like those of the previous week. The weekly autocorrelation of each feature, within each stock, measures it:

persist = {c: d.groupby("ticker")[c]
           .apply(lambda s: s.autocorr(1)).mean() for c in FEATS}
persist = pd.Series(persist, name="weekly autocorr.")
print(persist.sort_values(ascending=False).round(2).to_string())

valor_bp              0.99
log_capitalizacion    0.99
roe                   0.99
ey                    0.99
ruido_3               0.97
mom_12_1              0.97
vol_63d               0.97
dist_max_52s          0.96
sue                   0.94
ret_3m                0.92
ruido_2               0.89
vol_21d               0.88
ruido_5               0.77
ret_1m                0.77
rsi_14                0.63
dias_desde_informe    0.63
sentimiento           0.59
volumen_rel           0.46
ruido_4               0.33
ret_1s                0.03
ruido_1               0.00

Almost all the features are slow: value, size, the accounting ratios and 12-month momentum have a weekly autocorrelation of 0.94 to 0.99, and ruido_3, an AR(1) noise with no effect at all, 0.97. Together they form a fingerprint that identifies each stock in each stretch of time. Only ret_1s (0.03) and ruido_1 (0.00) are fully renewed every week; ruido_4 and volumen_rel, halfway. For a forest with small leaves, finding the neighbor is as easy as recognizing a face is for a human.

flowchart TB
  A[Overlapping label] --> C[Same answer]
  B[Slow features] --> D[Recognizable neighbor]
  C --> E[Shuffled k-fold]
  D --> E
  E --> F[Inflated hit rate]

The two ingredients, separately

If the explanation is correct, removing either of the two ingredients makes the leak disappear, and the second alone is enough to manufacture a hit rate with columns that have no effect at all. Three experiments, always with shuffled k-fold:

• the one-week label (no overlap) with the same 21 features;
• the 21-session label with four persistent features with no effect: ruido_2, ruido_3, ruido_5 and log_capitalizacion;
• the 21-session label with three fast features: ret_1s, ruido_1 and volumen_rel.

def columnas(cols):
    return np.nan_to_num(d[cols].to_numpy(dtype=float))

lentas = ["ruido_2", "ruido_3", "ruido_5", "log_capitalizacion"]
rapidas = ["ret_1s", "ruido_1", "volumen_rel"]
exp = {
    "21 sessions, 21 features": res["shuffled k-fold"],
    "1 week, 21 features": acierto(barajado, yv=y_sem),
    "21 sessions, 4 slow, no effect":
        acierto(barajado, Xv=columnas(lentas)),
    "21 sessions, 3 fast": acierto(barajado, Xv=columnas(rapidas)),
}
for nombre, a in exp.items():
    print(f"{nombre:<34}{a:.1%}")

21 sessions, 21 features          59.7%
1 week, 21 features               50.0%
21 sessions, 4 slow, no effect    53.5%
21 sessions, 3 fast               49.8%

With the one-week label, shuffled k-fold becomes honest: 50.0%, the same result as with blocks. With fast features, too. And with four columns that have no effect at all on returns (three noises and size, which on this panel earns no premium), the forest reaches 53.5%: above the oracle, with no information whatsoever about the future. The leak does not need a signal; it needs a fingerprint and a shared answer.

Grouping by date is not enough

A panel has two dimensions, and the usual intuition is to protect the cross-sectional one: the 80 stocks of the same week should not be split between training and test, because they share that week's market and sector moves. That is what GroupKFold does with the week as the group: each whole week falls into a single fold. With shuffle=True, the weeks are assigned to folds at random.

gkf = GroupKFold(5, shuffle=True, random_state=0)
por_semana = list(gkf.split(X, y, groups=semana))
comunes = [np.intersect1d(semana[a], semana[b]).size
           for a, b in por_semana]
print(f"weeks on both sides, per fold: {comunes}")
print(f"GroupKFold by week, shuffled: {acierto(por_semana):.1%}")

weeks on both sides, per fold: [0, 0, 0, 0, 0]
GroupKFold by week, shuffled: 59.2%

No week is on both sides, and the hit rate barely moves: 59.2%. The leak was not cross-sectional. The label is relative to the week's median, so the common component was already out of the answer; what the forest exploits is the same stock in the surrounding weeks, which still fall into training when the weeks are shuffled. On a financial panel the group that matters is time, and not as a group label but as an order: the folds have to be contiguous blocks, with a margin around them.

The wrong validation picks the wrong model

The leak does not only inflate a figure. If the validation is used to choose hyperparameters (lesson 5.5.4), it picks whatever exploits the leak best. The hyperparameter that controls how much a forest can memorize is the minimum leaf size: with one-observation leaves, each tree can isolate a stock in a specific stretch of weeks. Four sizes, with the two validations:

hojas = {}
for hoja in (1, 5, 50, 500):
    if hoja == 5:  # already computed above
        hojas[hoja] = (res["shuffled k-fold"], res["purged k-fold"])
    else:
        hojas[hoja] = (acierto(barajado, hoja=hoja),
                       acierto(purgado, hoja=hoja))
pd.DataFrame(hojas, index=["shuffled", "purged"]).T.rename_axis(
    "min leaf").map(lambda a: f"{a:.1%}")

         shuffled purged
min leaf
1           60.4%  50.3%
5           59.7%  50.1%
50          55.7%  50.2%
500         52.3%  50.2%

Shuffled k-fold has a clear, monotonic preference: the smaller the leaf, the higher the hit rate, from 52.3% with leaves of 500 to 60.4% with leaves of 1. A hyperparameter search with that validation would pick the forest that memorizes most and present it with 60%. Purged k-fold gives between 50.1% and 50.3% with all four sizes: none stands out, because the forest does not find in these features a signal it can exploit out of sample. It is the right answer, however disappointing, and the one to take seriously: lesson 5.5.1 shows that a linear regression with the features as ranks does pick up part of the signal (IC of 0.0185 in the README), and 5.4.2 checks it with purged validation.

What to do in practice

1. No shuffling of rows in a problem with labels that span several periods or with persistent features, which in finance means almost all of them. Folds contiguous in time.
2. Purge and embargo around each test block with the real start and end dates of each label, not with an approximate rule (lesson 5.4.2).
3. Walk-forward for the history that would have been lived in real time, with retraining (lesson 5.4.3).
4. Compare with a ceiling and with a baseline: on synthetic data, the oracle; on real data, the order of magnitude in the literature and a simple linear model. A result far above both is a leak until proven otherwise.
5. Audit when each piece of data is known before validating anything. Purging protects against label overlap, not against a misdated feature.

Exercises

1. With the 21-session label, shuffled k-fold gives 53.5% with four slow features with no effect. Is a single one enough? Repeat it with only log_capitalizacion and with only ruido_3, and explain the difference with their persistence.

for col in ("log_capitalizacion", "ruido_3"):
    a = acierto(barajado, Xv=columnas([col]))
    print(f"{col:<20}autocorr. {persist[col]:.2f}  "
          f"shuffled {a:.1%}")

log_capitalizacion  autocorr. 0.99  shuffled 51.1%
ruido_3             autocorr. 0.97  shuffled 50.4%

2. The 58,000 rows are not 58,000 independent observations. Estimate the effective number of independent labels (each stock has a new label roughly every five weeks) and the corresponding standard error of the hit rate. Does the conclusion about the 59.7% change?

n_ef = len(y) / 5  # one independent label every ~5 weeks
for nombre, n in (("rows", len(y)), ("effective", n_ef)):
    print(f"{nombre:<10} n = {n:>8,.0f}  standard error "
          f"{np.sqrt(0.25 / n):.2%}")
print(f"59.7% against the 51.1% ceiling: "
      f"{(0.597 - 0.511) / np.sqrt(0.25 / n_ef):.0f} standard errors")

rows       n =   58,000  standard error 0.21%
effective  n =   11,600  standard error 0.46%
59.7% against the 51.1% ceiling: 19 standard errors

3. A colleague presents a weekly stock classification model with a 57% hit rate validated with TimeSeriesSplit without gap and one-month labels. What three questions would you ask before believing them?