In-sample and out-of-sample: anchored and rolling walk-forward.

Choosing and measuring with different data

Almost every strategy has parameters: the horizon of a signal, a threshold, the length of a moving average. Choosing them is unavoidable, and choosing them with data is reasonable. The problem appears when they are measured with the same data they were chosen with: the result is no longer an estimate of what the strategy will do, it is the maximum of all the variants tried, which is always optimistic (lesson 2.5.1, the selection room).

The classic solution is to split the data into two parts:

• In-sample (IS): the data used to choose. Here you can try as many variants as you like.
• Out-of-sample (OOS): the data used to measure the chosen variant. It is looked at once, when the choice has already been made.

flowchart LR
  D[History] --> IS[In-sample: choose]
  IS --> P[Fixed parameter]
  P --> OOS[Out-of-sample: measure]

The rule looks easy to follow and is often broken without noticing: you look at the out-of-sample result, you do not like it, you change something “reasonable” (a filter, another threshold) and you look again. At that moment the out-of-sample data has become part of the in-sample data. Lesson 2.5.1 measures it: filtering a thousand noise strategies by both halves leaves seven false positives.

There is a second, more technical problem. A single split is a single draw: the result depends on which regime falls on each side of the cut. Walk-forward validation solves both things at once: it repeats the split along the history, with a mechanical rule that prevents looking ahead of time.

One parameter, eight variants

The test strategy is the one from lesson 2.5.1: every market with the same volatility target and the sign of the return over the last L sessions. The parameter is L, with eight values from two weeks to a year and a half. The variantes matrix has one column per value: the daily return of the portfolio with that L, already executed with a one-day lag.

import numpy as np
import pandas as pd

fut = pd.read_parquet("futuros2_continuos.parquet")
precio = fut.pivot(index="fecha", columns="mercado",
                   values="cierre_ajustado")
r = precio.pct_change(fill_method=None)
vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252)

def sharpe(x):
    return x.mean() / x.std() * np.sqrt(252)

LOOKBACKS = [10, 21, 42, 63, 126, 189, 252, 378]
variantes = {}
for n in LOOKBACKS:
    senal = np.sign(precio.pct_change(n, fill_method=None))
    peso = (senal * 0.40 / vol / precio.shape[1]).shift(1)
    variantes[n] = (peso * r).sum(axis=1, min_count=1)
variantes = pd.DataFrame(variantes).loc["2002":]
print(variantes.shape)
sharpe(variantes).round(2).to_frame("Sharpe 2002-2025").T

(6039, 8)
                   10    21    42    63    126   189   252   378
Sharpe 2002-2025  0.03  0.34  0.61  0.62  0.78  0.91  1.06  0.85

Looking at the whole history, the best lookback is 252, the one planted by the generator: the Sharpe rises from 0.03 with 10 sessions to 1.06 with 252 and falls back to 0.85 with 378. But this table is exactly what cannot be done in real time: in 2002 nobody knew the 2002-2025 Sharpe. The table also shows something that matters for what follows: all eight variants have positive Sharpes, because they are all momentum on the same markets (the columns are highly correlated), and the three longest ones (189, 252 and 378) are less than 0.25 apart. Choosing between them is choosing between look-alikes, and that makes the choice unstable.

A single split, to begin with: choose with the years up to a cut and measure on the following ones. To see how much it depends on the cut, four different cuts:

filas = {}
for corte in (2006, 2009, 2013, 2017):
    dentro = variantes.loc[:str(corte)]
    fuera = variantes.loc[str(corte + 1):]
    mejor = sharpe(dentro).idxmax()
    filas[f"2002-{corte} / {corte + 1}-2025"] = {
        "chosen L": mejor, "IS": sharpe(dentro)[mejor],
        "OOS": sharpe(fuera)[mejor], "252 OOS": sharpe(fuera)[252]}
pd.DataFrame(filas).T.astype({"chosen L": int}).round(2)

                       chosen L    IS   OOS  252 OOS
2002-2006 / 2007-2025       189  1.33  0.81     1.09
2002-2009 / 2010-2025       189  0.85  0.95     1.17
2002-2013 / 2014-2025       252  1.01  1.10     1.10
2002-2017 / 2018-2025       252  0.99  1.19     1.19

With a cut in 2013 or in 2017, the split picks 252 and out-of-sample gives slightly more than in-sample (1.10 versus 1.01 and 1.19 versus 0.99): it looks like a clean validation. With a cut in 2006 or in 2009 it picks the neighbor, 189, and out-of-sample gives 0.81 and 0.95, below what 252 would have given in the same years (1.09 and 1.17). With the 2006 cut the in-sample Sharpe, 1.33, promises much more than what arrives out-of-sample. The same strategy, with the same method, gives different conclusions depending on where the cut falls, because each cut is a single draw. And the resulting number does not say whether the chosen parameter would have been stable: there is only one choice.

Walk-forward: optimize, test, move forward

Walk-forward validation turns the split into a loop. At each step:

1. Take an in-sample window that ends the day before the test window.
2. Choose the variant with the best criterion (here, the Sharpe) in that window.
3. Apply the chosen variant, untouched, to the following out-of-sample window.
4. Move forward one test window and repeat.

The out-of-sample windows do not overlap and are chained together: their concatenation is the equity curve of someone who had followed the procedure in real time, reoptimizing every year. There are two ways of defining the in-sample window, and the figure at the top draws them with the years of this example (5 years in-sample, 1 out-of-sample):

• Anchored (expanding): it always starts on the first day of the data and grows at each step.
• Rolling: it has a fixed length and moves forward with the test window; the oldest data is forgotten.

stateDiagram-v2
  [*] --> Optimize
  Optimize --> Test: fixed parameter
  Test --> Advance: store OOS
  Advance --> Optimize: data left
  Advance --> [*]: end

The following function works for any variant matrix (rows, days; columns, variants), so it serves equally well for the futures and for the noise strategies further down:

def walk_forward(var, modo, anos_dentro=5, desde=2007, hasta=2025):
    """Each year, picks the best column of `var` in the in-sample
    window and applies it to the following year."""
    filas, fuera = [], []
    for ano in range(desde, hasta + 1):
        if modo == "anclado":
            inicio = var.index[0].year
        else:
            inicio = ano - anos_dentro
        dentro = var.loc[str(inicio):str(ano - 1)]
        s = sharpe(dentro)
        elegido = s.idxmax()
        prueba = var.loc[str(ano), elegido]
        filas.append({"ano": ano, "L": elegido, "sh_dentro": s.max(),
                      "sh_fuera": sharpe(prueba)})
        fuera.append(prueba)
    return pd.DataFrame(filas).set_index("ano"), pd.concat(fuera)

anc, oos_anc = walk_forward(variantes, "anclado")
mov, oos_mov = walk_forward(variantes, "movil")
pd.concat({"anchored": anc, "rolling": mov}, axis=1).round(2)

     anchored                    rolling
            L sh_dentro sh_fuera       L sh_dentro sh_fuera
ano
2007      189      1.33    -0.51     189      1.33    -0.51
2008      189      1.04     0.82      42      0.96    -0.33
2009      189      0.99    -0.15     252      1.09    -0.09
2010      189      0.85     0.72     252      0.62     1.65
2011      252      0.92     0.77     252      0.77     0.77
2012      252      0.91     0.57     252      0.88     0.57
2013      252      0.87     2.51     252      0.90     2.51
2014      252      1.01     2.61     252      1.05     2.61
2015      252      1.12     0.76     252      1.59     0.76
2016      252      1.09     0.07     252      1.37     0.07
2017      252      1.03     0.21     378      1.28    -1.03
2018      252      0.99     1.26     252      1.26     1.26
2019      252      1.00     2.18     252      0.99     2.18
2020      252      1.06    -0.06     252      0.91    -0.06
2021      252      1.00     1.89      63      1.06     1.33
2022      252      1.04     0.73      63      1.37     1.85
2023      252      1.03     2.25      63      1.58     0.93
2024      252      1.09     1.71      21      1.65    -0.29
2025      252      1.12    -0.30     189      1.50    -0.78

Each row is a decision taken with the data available before it. Two things stand out. The first is stability: the anchored version picks 189 from 2007 to 2010 and 252 from 2011 on, with a single change in 19 years; the rolling one changes seven times among six values, from 21 to 378 sessions, because with five years of data the difference between similar variants is noise. Between 2021 and 2024 it goes as far as choosing 63 and 21 sessions, and in 2022 the 63 works out (1.85 versus 0.73 for the anchored one): one good year does not make the procedure good. The second is that the year-by-year out-of-sample Sharpes range from −1.03 to 2.61: one year is little for measuring a Sharpe (its annual standard error is 1), and that is why what gets evaluated is the chained series, not each window.

How much is lost when leaving the sample

The measure that summarizes a walk-forward is the Sharpe decay: the mean of the in-sample windows (what the researcher saw when choosing) versus that of the chained out-of-sample series (what they would have earned). Their ratio is sometimes called walk-forward efficiency, a term Pardo (2008) uses in his book on strategy evaluation (here, with the Sharpe):

An efficiency close to 1 says the procedure is not fooling itself; one close to 0 or negative, that the in-sample Sharpe was almost all selection. As a comparison, the fixed 252 variant, with nothing optimized, over the same years:

resumen = {}
for nombre, tabla, serie in (("anchored", anc, oos_anc),
                             ("rolling", mov, oos_mov)):
    sd, sf = tabla["sh_dentro"].mean(), sharpe(serie)
    resumen[nombre] = {"IS": sd, "OOS": sf,
                       "efficiency": sf / sd,
                       "changes": int((tabla["L"].diff() != 0)
                                      .iloc[1:].sum())}
fijo = sharpe(variantes.loc["2007":, 252])
resumen["252 fixed"] = {"IS": np.nan, "OOS": fijo,
                        "efficiency": np.nan, "changes": 0}
pd.DataFrame(resumen).T.round(2)

             IS   OOS  efficiency  changes
anchored   1.02  0.95        0.93      1.0
rolling    1.17  0.70        0.60      7.0
252 fixed   NaN  1.09         NaN      0.0

The anchored version keeps 93% of the in-sample Sharpe; the rolling one, 60%. In this data the anchored version wins because the effect is constant (252 sessions for 26 years): accumulating history only helps to measure it better. With an effect that really changed over time, the rolling version could adapt and the anchored one would fall behind; the price of that ability to adapt is the instability of the previous table. Neither of them beats the fixed 252 parameter (1.09), which is the known answer: optimizing adds nothing when the variants are similar and one of them is already the right one, and it subtracts the noise of the choice.

The windows are parameters too

The walk-forward has its own decisions: the length of the in-sample window, that of the test window, the grid of values and the selection criterion. Each one changes the result, and choosing them by looking at the out-of-sample result is another form of selection. In the rolling version, with test windows from 2012 to 2025 so that all three lengths use the same years:

filas = {}
for anos in (3, 5, 10):
    t, s = walk_forward(variantes, "movil", anos_dentro=anos,
                        desde=2012)
    filas[f"{anos} years"] = {
        "IS": t["sh_dentro"].mean(), "OOS": sharpe(s),
        "distinct L values": t["L"].nunique()}
pd.DataFrame(filas).T.round(2)

            IS   OOS  distinct L values
3 years   1.39  0.88                6.0
5 years   1.24  0.87                5.0
10 years  1.09  1.12                2.0

With ten years the out-of-sample series comes out better (1.12) and the choice barely moves (two values of L); with three and with five, worse and almost the same (0.88 and 0.87), with six and five distinct values. It fits what is known about this data, a stable effect that is measured better with more history, but the difference between 0.87 and 1.12 over 14 years of testing is within the standard error of a Sharpe over that span ($1/\sqrt{14} \approx 0.27$). And even if the ten-year window really were the best, if it were published because it is the one that does best out-of-sample, the result would no longer be out-of-sample. The sound practice is to fix the windows before looking (by the horizon of the signal, by the amount of data needed to measure a Sharpe with some precision) and to record every combination tried in the research journal (lesson 2.2.4), as one more trial for the Deflated Sharpe (lesson 2.5.4).

Noise in a walk-forward

To see the signature of overfitting with no edge to hide it, the same procedure on the 1,000 strategies of ruido2_estrategias.parquet (all with no edge except five). The strategies only cover 2022-2025, so the windows are shorter: one year in-sample (252 sessions), one quarter out-of-sample (63), moving forward quarter by quarter.

est = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")

filas, fuera = [], []
for i in range(252, len(est), 63):
    dentro = est.iloc[i - 252:i]
    s = sharpe(dentro)
    elegida = s.idxmax()
    prueba = est.iloc[i:i + 63][elegida]
    fuera.append(prueba)
    filas.append({"desde": est.index[i].date(), "elegida": elegida,
                  "sh_dentro": s.max(), "sh_fuera": sharpe(prueba),
                  "real": verdad.loc[elegida, "sharpe_real"]})
ruido = pd.DataFrame(filas)
print(ruido.round(2).to_string(index=False))
print(f"In-sample, mean: {ruido['sh_dentro'].mean():.2f}; "
      f"out-of-sample, chained: {sharpe(pd.concat(fuera)):.2f}")

     desde elegida  sh_dentro  sh_fuera  real
2023-01-04   e0868       3.29     -1.95   0.0
2023-04-05   e0601       2.63      0.75   0.0
2023-07-07   e0462       3.23      2.27   0.0
2023-10-05   e0542       3.62      3.25   0.0
2024-01-05   e0133       3.96     -3.00   0.8
2024-04-08   e0542       4.03      0.56   0.0
2024-07-09   e0598       3.42     -0.94   0.0
2024-10-07   e0091       3.24      0.97   0.0
2025-01-07   e0032       3.43     -1.79   0.0
2025-04-09   e0897       3.64     -0.78   0.0
2025-07-11   e0023       3.79      0.80   0.0
2025-10-09   e0023       3.21     -3.74   0.0
In-sample, mean: 3.46; out-of-sample, chained: -0.53

Every quarter, the best of a thousand has an in-sample Sharpe between 2.6 and 4.0; out-of-sample, the chained series loses money. Only once does the procedure pick one of the five strategies with an edge (e0133, in January 2024), and that quarter it loses. The choice changes almost every quarter: 10 different strategies in 12 windows. It is the full signature of overfitting: a very high Sharpe in-sample, zero or negative out-of-sample, and parameters that do not repeat. In the futures, with a real effect, the decay exists but is partial and the choice tends to stabilize; in the noise, the decay is total.

The comparison also shows the limit of walk-forward validation: with 1,000 candidates and one-quarter windows, not even the five strategies with a true edge (Sharpe from 0.5 to 1.5) stand out from the noise. Walk-forward validation detects overfitting, but it does not give the good strategy a statistical power the data does not have. To make better use of little data there is purged cross-validation (lesson 2.5.3) and, to correct for the number of trials, the Deflated Sharpe and the probability of backtest overfitting (lesson 2.5.4).

Common mistakes

• Optimizing with data that overlaps the test. The in-sample window ends the day before the test window. The warm-up of the indicators (the 378 sessions the longest lookback needs) can take data from before the in-sample window; never from after it.
• Reoptimizing after looking. Changing the grid, the criterion or the windows because you do not like the out-of-sample result turns the walk-forward into one more variant (previous section).
• Evaluating the windows separately. A one-year Sharpe has a standard error of 1. What gets evaluated is the chained series and, at most, the proportion of positive windows.
• Confusing the out-of-sample curve with the strategy. The chained series measures the procedure (choosing the best L every year), not a specific L. If a fixed L is going to be used in production, you have to say which one and why.
• Forgetting costs and portfolio turnover. A procedure that changes parameter often also changes positions; costs are deducted before choosing and in the test (lesson 2.4.7).

Exercises

1. Repeat the anchored walk-forward with two-year test windows (reoptimizing every two years). Does the choice of the parameter change? And the Sharpe decay?

filas, fuera = [], []
for ano in range(2007, 2026, 2):
    dentro = variantes.loc[:str(ano - 1)]
    elegido = sharpe(dentro).idxmax()
    prueba = variantes.loc[str(ano):str(ano + 1), elegido]
    filas.append((ano, elegido, sharpe(dentro).max()))
    fuera.append(prueba)
t = pd.DataFrame(filas, columns=["desde", "L", "sh_dentro"])
print(t.round(2).to_string(index=False))
print(f"In-sample {t['sh_dentro'].mean():.2f}, "
      f"out-of-sample {sharpe(pd.concat(fuera)):.2f}")

 desde   L  sh_dentro
  2007 189       1.33
  2009 189       0.99
  2011 252       0.92
  2013 252       0.87
  2015 252       1.12
  2017 252       1.03
  2019 252       1.00
  2021 252       1.00
  2023 252       1.03
  2025 252       1.12
In-sample 1.04, out-of-sample 0.95

2. In the walk-forward on the noise, how many times does the chosen strategy have a positive out-of-sample Sharpe? Compare it with what you would expect by chance.

positivos = int((ruido["sh_fuera"] > 0).sum())
print(f"{positivos} of {len(ruido)} quarters with a positive Sharpe")
from scipy.stats import binom
print(f"P(≥ {positivos} of {len(ruido)} with p = 0.5): "
      f"{binom.sf(positivos - 1, len(ruido), 0.5):.2f}")

6 of 12 quarters with a positive Sharpe
P(≥ 6 of 12 with p = 0.5): 0.61

3. A colleague shows you a walk-forward with an out-of-sample Sharpe of 1.4 and an efficiency of 0.9, and tells you they tried it “with a few windows until it came out stable.” What do you ask them, and what would you do with the result?