Walk-forward with retraining: windows and frequency.

Two decisions that validation does not make for you

Lesson 2.5.2 ran a walk-forward to choose a parameter of a strategy and 5.4.2, a purged validation to compare models. A walk-forward with a machine learning model adds two design decisions:

• The window: with all the available history (expanding), with the last N years (rolling) or with all of it but giving more weight to recent data (exponential weights, with a half-life).
• The frequency: retrain every year, every quarter, every month or every week.

flowchart TB
  D[Decision date] --> E[Closed labels]
  E --> V[Choose window]
  V --> M[Retrain if due]
  M --> P[Predict the week]
  P --> A[Advance one week]
  A --> D

The figure compares what the model sees before each test year with an expanding window and with a three-year rolling window, both retrained every January:

Intuition pushes toward short windows and frequent retraining: markets change. The level 5 panel is a good place to put it to the test, because the effects really do change and you know by how much. According to the README, each true coefficient is base × regime × m, with m a weekly AR(1) with persistence 0.9 (the effects weaken and sometimes reverse) and momentum multiplied by −2.5 in the rebounds after a crisis (the momentum crash). panel5_coef_verdad.csv contains the coefficients of each week and its regime.

The panel as arrays

To retrain hundreds of times it is convenient to have the panel as arrays of weeks × stocks × features: 730 × 400 × 21. The transformation is the README one (and that of lesson 5.2.4): the cross-sectional normal rank $g(x) = \Phi^{-1}\big((\text{rank} - 0.5)/n\big)$ over the stocks that have data, and 0 where it is missing.

import numpy as np
import pandas as pd
from scipy.stats import norm, rankdata
from sklearn.linear_model import Ridge

feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
feat = feat.sort_values(["fecha", "ticker"])
etiq = etiq.sort_values(["fecha", "ticker"])
FEATS = [c for c in feat.columns if c not in
         ("fecha", "ticker", "sector", "revision_analistas")]
fechas = pd.DatetimeIndex(feat["fecha"].unique())
nW, nA, nF = len(fechas), feat["ticker"].nunique(), len(FEATS)

F = feat[FEATS].to_numpy(dtype=float).reshape(nW, nA, nF)
with np.errstate(invalid="ignore", divide="ignore"):
    rango = rankdata(F, axis=1, nan_policy="omit")
    n = np.isfinite(F).sum(axis=1, keepdims=True)
    G = np.nan_to_num(norm.ppf((rango - 0.5) / n))

R = etiq["ret_sig_semana"].to_numpy().reshape(nW, nA)
fin = etiq["fin_sig_semana"].to_numpy().reshape(nW, nA)[:, 0]
con_feat = np.isfinite(F[:, :, 0]).all(axis=1)
con_etiq = np.isfinite(R).all(axis=1)
valida = con_feat & con_etiq           # weeks for training
Yc = np.zeros_like(R)
Yc[valida] = R[valida] - R[valida].mean(axis=1, keepdims=True)

coef = pd.read_csv("panel5_coef_verdad.csv", parse_dates=["fecha"])
coef = coef.set_index("fecha").reindex(fechas)
regimen = coef["regimen"].to_numpy()
print(f"{nW} weeks x {nA} stocks x {nF} features; "
      f"valid weeks {valida.sum()}")
print(pd.Series(regimen[valida]).value_counts().to_string())

730 weeks x 400 stocks x 21 features; valid weeks 729
calma     575
estres     89
rebote     65

The time rule: train on closed labels

At decision date w you can only train on rows whose label has already ended. The rule comes from the data, not from an assumption: for each date, the last usable week is the last one whose fin_sig_semana is before it.

siguiente = (fin[:-1] == fechas[1:].to_numpy()).mean()
print(f"the label of week w ends at date w+1: "
      f"{siguiente:.1%} of the weeks")

def ultima_conocida(w):
    """Index of the last week whose label ended before
    date w (end < date: a one-week margin)."""
    return int(np.searchsorted(fin, fechas[w], side="left")) - 1

w = fechas.get_loc(pd.Timestamp("2020-03-20"))
u = ultima_conocida(w)
print(f"decision on {fechas[w].date()}: train up to the week of "
      f"{fechas[u].date()} (its label ended on "
      f"{pd.Timestamp(fin[u]).date()})")

the label of week w ends at date w+1: 100.0% of the weeks
decision on 2020-03-20: train up to the week of 2020-03-06 (its label ended on 2020-03-13)

The label of week w ends exactly at date w+1. So, at the close of date w, the label of w−1 has just closed and that of w is open. The rule fin < fecha also leaves out the label of w−1 (which ends exactly that day): it is the margin the README uses, a prudent one if orders are sent before the close. Including week w would be a leak: exercise 1 measures how much.

Retraining a thousand times in seconds

A ridge with intercept on 21 features has a closed-form solution from five quantities: the number of rows, the sums of x and y, and the matrices $X'X$ and $X'y$. All of them add up across weeks. If they are stored per week, the ridge of any window (or with per-week weights) is a sum and a 21 × 21 system:

where the subscript c denotes centered data (scikit-learn centers the data to estimate the intercept, which is not penalized).

X1 = np.concatenate([np.ones((nW, nA, 1)), G], axis=2)
peso_valida = valida.astype(float)
S_xx = np.einsum("wni,wnj->wij", X1, X1) * peso_valida[:, None, None]
S_xy = np.einsum("wni,wn->wi", X1, Yc) * peso_valida[:, None]

def ridge(pesos, alfa=10.0):
    """Ridge with intercept on the weeks with `pesos` > 0."""
    A = np.tensordot(pesos, S_xx, 1)
    b = np.tensordot(pesos, S_xy, 1)
    n_, mx, my = A[0, 0], A[0, 1:] / A[0, 0], b[0] / A[0, 0]
    C = A[1:, 1:] - n_ * np.outer(mx, mx)
    c = b[1:] - n_ * mx * my
    return np.linalg.solve(C + alfa * np.eye(nF), c)

prueba = np.flatnonzero(valida & (fechas.year < 2015))
sk = Ridge(alpha=10.0).fit(G[prueba].reshape(-1, nF),
                           Yc[prueba].reshape(-1))
pesos = np.zeros(nW)
pesos[prueba] = 1.0
print(f"max. |difference| with scikit-learn: "
      f"{np.abs(ridge(pesos) - sk.coef_).max():.1e}")

max. |difference| with scikit-learn: 8.1e-18

The difference is in the last decimal place of the float64. From here on, each retraining is a weighted sum of 730 small matrices: the whole weekly walk-forward takes less than a second.

The walk-forward and its metric

The function goes through the test weeks (2015-2025), retrains when due and predicts the score of each stock. The metric is the IC: the rank correlation, each week, between the score and the following week's return, and its mean. The standard error comes from the weekly series of ICs (the labels of one week do not overlap, so that series is almost independent).

anyo = fechas.year.to_numpy()
semanas_prueba = np.flatnonzero(con_feat & (anyo >= 2015))

def walk_forward(ventana=None, cada=None, anual=False,
                 semivida=None):
    """Scores (weeks x stocks) and coefficients per week.
    ventana: weeks of history (None = expanding); cada: retrain
    every n weeks; anual: every January; semivida: exponential weights."""
    pred = np.full((nW, nA), np.nan)
    betas = np.full((nW, nF), np.nan)
    beta = None
    for i, w in enumerate(semanas_prueba):
        toca = (beta is None
                or (anual and anyo[w] != anyo[semanas_prueba[i - 1]])
                or (cada is not None and i % cada == 0))
        if toca:
            u = ultima_conocida(w)
            p = peso_valida.copy()
            p[u + 1:] = 0.0
            if ventana is not None:
                p[:max(0, u + 1 - ventana)] = 0.0
            if semivida is not None:
                p *= 0.5 ** ((u - np.arange(nW)) / semivida)
            beta = ridge(p)
        pred[w] = G[w] @ beta
        betas[w] = beta
    return pred, betas

evaluables = semanas_prueba[con_etiq[semanas_prueba]]

def ic_semanal(pred):
    out = []
    for w in evaluables:
        out.append(np.corrcoef(rankdata(pred[w]),
                               rankdata(R[w]))[0, 1])
    return np.array(out)

def resumen(ic):
    et = ic.std(ddof=1) / np.sqrt(len(ic))
    return {"IC": ic.mean(), "std. error": et, "t": ic.mean() / et}

base, betas_base = walk_forward(anual=True)
ic_base = ic_semanal(base)
print({k: round(float(v), 4) for k, v in resumen(ic_base).items()},
      f"weeks: {len(ic_base)}")

{'IC': 0.0185, 'std. error': 0.0038, 't': 4.8789} weeks: 574

Expanding, retrained every January: an IC of 0.0185 with t = 4.9 over 574 weeks, the same result the README publishes for this model. It is the reference for the rest of the lesson.

Retraining frequency

With the expanding window, four frequencies: every January, every 13 weeks (quarterly), every 4 (monthly) and every week.

esquemas = {"expanding, annual": ic_base}
for nombre, kw in [("expanding, every 13 weeks", {"cada": 13}),
                   ("expanding, every 4 weeks", {"cada": 4}),
                   ("expanding, weekly", {"cada": 1})]:
    esquemas[nombre] = ic_semanal(walk_forward(**kw)[0])
pd.DataFrame({k: resumen(v) for k, v in esquemas.items()}).T.round(4)

                               IC  std. error       t
expanding, annual          0.0185      0.0038  4.8789
expanding, every 13 weeks  0.0176      0.0038  4.6397
expanding, every 4 weeks   0.0172      0.0038  4.5324
expanding, weekly          0.0176      0.0038  4.6463

Retraining more often improves nothing: from 0.0185 to 0.0172-0.0176, differences of a few thousandths with a standard error of almost four. It makes sense: with an expanding window, one new year on top of ten years of history barely moves the coefficients, and one new week, even less.

Expanding, rolling and weighted windows

Now the window. Rolling windows of five, three, two and one year retrained every January, and two weekly versions that should be the most adaptive: a one-year rolling window and exponential weights with half-lives of two years and of one.

for nombre, kw in [
        ("rolling 5 years, annual", {"ventana": 260, "anual": True}),
        ("rolling 3 years, annual", {"ventana": 156, "anual": True}),
        ("rolling 2 years, annual", {"ventana": 104, "anual": True}),
        ("rolling 1 year, annual", {"ventana": 52, "anual": True}),
        ("rolling 1 year, weekly", {"ventana": 52, "cada": 1}),
        ("half-life 2 years, weekly", {"semivida": 104, "cada": 1}),
        ("half-life 1 year, weekly", {"semivida": 52, "cada": 1})]:
    p, b = walk_forward(**kw)
    esquemas[nombre] = ic_semanal(p)
    if nombre == "rolling 1 year, weekly":
        betas_movil = b
tabla = pd.DataFrame({k: resumen(v) for k, v in esquemas.items()}).T
print(tabla.round(4).to_string())

                               IC  std. error       t
expanding, annual          0.0185      0.0038  4.8789
expanding, every 13 weeks  0.0176      0.0038  4.6397
expanding, every 4 weeks   0.0172      0.0038  4.5324
expanding, weekly          0.0176      0.0038  4.6463
rolling 5 years, annual    0.0143      0.0040  3.6273
rolling 3 years, annual    0.0102      0.0037  2.7680
rolling 2 years, annual    0.0085      0.0038  2.2282
rolling 1 year, annual     0.0084      0.0039  2.1652
rolling 1 year, weekly     0.0039      0.0038  1.0086
half-life 2 years, weekly  0.0120      0.0038  3.1412
half-life 1 year, weekly   0.0081      0.0038  2.1550

The figure at the top ranks the eleven schemes with their ±2 standard error interval. The pattern is clear: the shorter the window, the worse. Five years lose a quarter of the IC (0.0143); one year, more than half (0.0084), and with weekly retraining, almost all of it (0.0039, t = 1.0, indistinguishable from zero). The exponential weights fall in between, worse the shorter the half-life.

The differences between schemes are not read from the intervals of each one, which overlap: they are read from the week-by-week paired difference, which removes the common noise (the good and bad weeks are the same for every model).

def pareada(a, b):
    dif = esquemas[a] - esquemas[b]
    et = dif.std(ddof=1) / np.sqrt(len(dif))
    return {"difference": dif.mean(), "t": dif.mean() / et}

pd.DataFrame({
    "expanding weekly - rolling 1 year weekly":
        pareada("expanding, weekly", "rolling 1 year, weekly"),
    "expanding annual - expanding weekly":
        pareada("expanding, annual", "expanding, weekly"),
    "expanding annual - rolling 5 years annual":
        pareada("expanding, annual", "rolling 5 years, annual"),
}).T.round(4)

                                           difference       t
expanding weekly - rolling 1 year weekly       0.0138  3.8797
expanding annual - expanding weekly            0.0008  1.2332
expanding annual - rolling 5 years annual      0.0041  2.6339

Expanding against one-year rolling, both with weekly retraining: a difference of 0.014 with t = 3.9. It is the only firm conclusion in the table. Annual against weekly frequency: t = 1.2, nothing. Expanding against five-year rolling, both annual: 0.004 with t = 2.6, in the same direction.

By regime: the momentum crash

The hope of the rolling window was to adapt to changes. The panel has known regime changes: in estres reversal strengthens (×1.5) and in rebote momentum reverses (×−2.5). The IC by regime tells whether any window takes advantage of it:

reg_eval = regimen[evaluables]
por_regimen = {}
for nombre in ("expanding, annual", "rolling 3 years, annual",
               "rolling 1 year, weekly", "half-life 1 year, weekly"):
    ic = esquemas[nombre]
    por_regimen[nombre] = {r: ic[reg_eval == r].mean()
                           for r in ("calma", "estres", "rebote")}
t = pd.DataFrame(por_regimen).T.round(4)
t.columns = [f"{r} ({(reg_eval == r).sum()} wk.)" for r in t.columns]
t

                          calma (420 wk.)  estres (89 wk.)  rebote (65 wk.)
expanding, annual                  0.0230           0.0026           0.0110
rolling 3 years, annual            0.0156          -0.0132           0.0075
rolling 1 year, weekly             0.0073          -0.0094          -0.0003
half-life 1 year, weekly           0.0128          -0.0077          -0.0003

No short window wins in the rebounds, which is where the advantage of adapting was expected: there the expanding window remains the best (0.011, half of what it gets in calm) and the adaptive ones stay at zero. The panel's rebounds last three to six months (March-June 2020, April-July 2025…): when a one-year window starts to reflect the change in the sign of momentum, the rebound is over and the model is left with a distorted coefficient for the calm that follows. In the stress weeks, moreover, the short windows have a negative IC. The adaptation that would work is not in the window but in the model: a regime feature observable in time (and properly dated, lesson 5.2.2) with which the model can learn the interaction.

Bias versus variance: the path of a coefficient

The explanation lies in the coefficients. panel5_coef_verdad.csv contains the true momentum coefficient each week (c_mom, in weekly return per unit of g). The ridge estimates the same object with the feature mom_12_1 (with the other 20 alongside, among them ruido_4, which inherits part of the momentum, so the level need not match). What matters is how each estimate moves against the truth:

j = FEATS.index("mom_12_1")
verdad_mom = coef["c_mom"].to_numpy()[semanas_prueba] * 1e4
_, betas_exp = walk_forward(cada=1)
filas = {}
for nombre, b in (("expanding weekly", betas_exp),
                  ("rolling 1 year weekly", betas_movil)):
    est = b[semanas_prueba, j] * 1e4
    filas[nombre] = {
        "mean (bp)": est.mean(), "std. (bp)": est.std(),
        "corr. with truth": np.corrcoef(est, verdad_mom)[0, 1],
        "root mean squared error (bp)":
            np.sqrt(np.mean((est - verdad_mom) ** 2))}
filas["truth"] = {"mean (bp)": verdad_mom.mean(),
                  "std. (bp)": verdad_mom.std()}
pd.DataFrame(filas).T.round(2)

                       mean (bp)  std. (bp)  corr. with truth  root mean squared error (bp)
expanding weekly            4.88       0.93             -0.01                          6.04
rolling 1 year weekly       4.37       6.37             -0.10                          8.99
truth                       2.40       5.42               NaN                           NaN

The true coefficient fluctuates a lot from one week to the next (a standard deviation of 5.4 bp around a mean of 2.4) and changes sign in the rebounds. The one-year estimate also fluctuates, with a standard deviation of 6.4 bp, but it does not follow the truth: its correlation with it is −0.10. It chases the noise in the sample of the last twelve months, not the effect. The expanding one is almost a straight line (a standard deviation of 0.9 bp, zero correlation): it is biased, because it does not see the changes, but its root mean squared error against the truth is 6.0 bp and that of the rolling one, 9.0. Tracking an effect that changes every few weeks with one year of noisy data adds more error than it removes. It is the classic result of Pesaran and Timmermann (2007): with breaks, the optimal window depends on the size of the break relative to the noise, and when the signal is small, using all the data usually wins.

Exercises

1. Repeat the weekly expanding walk-forward and the 26-week rolling one forgetting the margin: train up to week w itself, whose label is the one being predicted. How much does the IC rise in each one, and why more in the short window?

def ultima_con_fuga(w):
    return w  # includes the label of week w itself: a leak

original = ultima_conocida
for nombre, kw in (("expanding weekly", {"cada": 1}),
                   ("rolling 26 weeks", {"ventana": 26, "cada": 1})):
    fila = []
    for fn in (original, ultima_con_fuga):
        ultima_conocida = fn
        fila.append(resumen(ic_semanal(walk_forward(**kw)[0]))["IC"])
    print(f"{nombre:<18} no leak {fila[0]:.4f}  with leak "
          f"{fila[1]:.4f}")
ultima_conocida = original

expanding weekly   no leak 0.0176  with leak 0.0272
rolling 26 weeks   no leak 0.0051  with leak 0.0603

2. Try exponential weights with half-lives of 13, 26, 52, 104, 208 and 416 weeks (weekly retraining). Which one wins? Can you present its IC as an out-of-sample result?

barrido = {h: resumen(ic_semanal(walk_forward(semivida=h,
                                              cada=1)[0]))["IC"]
           for h in (13, 26, 52, 104, 208, 416)}
pd.Series(barrido, name="IC").rename_axis("half-life").round(4)

half-life
13     0.0048
26     0.0057
52     0.0081
104    0.0120
208    0.0150
416    0.0165
Name: IC, dtype: float64

3. The level's deriva5_* data (lesson 5.10.2) have a permanent sign change in the effect of a feature in July 2019. Without doing any calculations, would you expect the expanding or the rolling window to win there? What would you need in order to choose without looking at the result?