--- title: "2.5.2 In-sample, out-of-sample and walk-forward" description: "Optimizing a parameter without fooling yourself: in- and out-of-sample split, anchored and rolling walk-forward, parameter stability and Sharpe decay." url: "https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward.png" --- # 2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward > A parameter chosen with the same data it is evaluated on always looks good. Walk-forward validation repeats the split between optimizing and testing along the whole history, and measures what really matters: how much is lost when moving from in-sample to out-of-sample. - Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/) - Module: 2.5 · Biases and validation - Duration: 60 min - Prerequisites: [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/), [1.4.6 · Continuous futures series: roll rules, difference adjustment and ratio adjustment](https://pythonparatrading.com/en/python-trading-course/level-1/continuous-futures-series-and-roll-adjustment/), [0.5.3 · Selection, filtering, groupby, merge, pivot, resample and moving windows](https://pythonparatrading.com/en/python-trading-course/level-0/pandas-groupby-merge-resample-rolling/) ## By the end of this lesson - Split the data into in-sample (to choose) and out-of-sample (to measure), and explain why a single split is a single draw. - Program a generic walk-forward, anchored and rolling, over a matrix of strategy variants. - Measure the Sharpe decay from in-sample to out-of-sample and the stability of the parameters chosen in each window. - Recognize in noise the typical pattern of overfitting: a high Sharpe in-sample, zero or negative out-of-sample, and parameters that jump from one window to the next. ## Choosing and measuring with different data Almost every strategy has parameters: the horizon of a signal, a threshold, the length of a moving average. Choosing them is unavoidable, and choosing them with data is reasonable. The problem appears when they are measured with the **same** data they were chosen with: the result is no longer an estimate of what the strategy will do, it is the maximum of all the variants tried, which is always optimistic (lesson 2.5.1, the selection room). The classic solution is to split the data into two parts: - **In-sample** (IS): the data used to choose. Here you can try as many variants as you like. - **Out-of-sample** (OOS): the data used to measure the chosen variant. It is looked at **once**, when the choice has already been made. ```mermaid flowchart LR D[History] --> IS[In-sample: choose] IS --> P[Fixed parameter] P --> OOS[Out-of-sample: measure] ``` The rule looks easy to follow and is often broken without noticing: you look at the out-of-sample result, you do not like it, you change something “reasonable” (a filter, another threshold) and you look again. At that moment the out-of-sample data has become part of the in-sample data. Lesson 2.5.1 measures it: filtering a thousand noise strategies by both halves leaves seven false positives. There is a second, more technical problem. A single split is **a single draw**: the result depends on which regime falls on each side of the cut. Walk-forward validation solves both things at once: it repeats the split along the history, with a mechanical rule that prevents looking ahead of time. ## One parameter, eight variants The test strategy is the one from lesson 2.5.1: every market with the same volatility target and the sign of the return over the last L sessions. The parameter is L, with eight values from two weeks to a year and a half. The `variantes` matrix has one column per value: the daily return of the portfolio with that L, already executed with a one-day lag. ```python import numpy as np import pandas as pd fut = pd.read_parquet("futuros2_continuos.parquet") precio = fut.pivot(index="fecha", columns="mercado", values="cierre_ajustado") r = precio.pct_change(fill_method=None) vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252) def sharpe(x): return x.mean() / x.std() * np.sqrt(252) LOOKBACKS = [10, 21, 42, 63, 126, 189, 252, 378] variantes = {} for n in LOOKBACKS: senal = np.sign(precio.pct_change(n, fill_method=None)) peso = (senal * 0.40 / vol / precio.shape[1]).shift(1) variantes[n] = (peso * r).sum(axis=1, min_count=1) variantes = pd.DataFrame(variantes).loc["2002":] print(variantes.shape) sharpe(variantes).round(2).to_frame("Sharpe 2002-2025").T ``` ```output (6039, 8) 10 21 42 63 126 189 252 378 Sharpe 2002-2025 0.03 0.34 0.61 0.62 0.78 0.91 1.06 0.85 ``` Looking at the whole history, the best lookback is 252, the one planted by the generator: the Sharpe rises from 0.03 with 10 sessions to 1.06 with 252 and falls back to 0.85 with 378. But this table is exactly what cannot be done in real time: in 2002 nobody knew the 2002-2025 Sharpe. The table also shows something that matters for what follows: all eight variants have positive Sharpes, because they are all momentum on the same markets (the columns are highly correlated), and the three longest ones (189, 252 and 378) are less than 0.25 apart. Choosing between them is choosing between look-alikes, and that makes the choice unstable. A single split, to begin with: choose with the years up to a cut and measure on the following ones. To see how much it depends on the cut, four different cuts: ```python filas = {} for corte in (2006, 2009, 2013, 2017): dentro = variantes.loc[:str(corte)] fuera = variantes.loc[str(corte + 1):] mejor = sharpe(dentro).idxmax() filas[f"2002-{corte} / {corte + 1}-2025"] = { "chosen L": mejor, "IS": sharpe(dentro)[mejor], "OOS": sharpe(fuera)[mejor], "252 OOS": sharpe(fuera)[252]} pd.DataFrame(filas).T.astype({"chosen L": int}).round(2) ``` ```output chosen L IS OOS 252 OOS 2002-2006 / 2007-2025 189 1.33 0.81 1.09 2002-2009 / 2010-2025 189 0.85 0.95 1.17 2002-2013 / 2014-2025 252 1.01 1.10 1.10 2002-2017 / 2018-2025 252 0.99 1.19 1.19 ``` With a cut in 2013 or in 2017, the split picks 252 and out-of-sample gives slightly more than in-sample (1.10 versus 1.01 and 1.19 versus 0.99): it looks like a clean validation. With a cut in 2006 or in 2009 it picks the neighbor, 189, and out-of-sample gives 0.81 and 0.95, below what 252 would have given in the same years (1.09 and 1.17). With the 2006 cut the in-sample Sharpe, 1.33, promises much more than what arrives out-of-sample. The same strategy, with the same method, gives different conclusions depending on where the cut falls, because each cut is a single draw. And the resulting number does not say whether the chosen parameter would have been stable: there is only one choice. ## Walk-forward: optimize, test, move forward Walk-forward validation turns the split into a loop. At each step: 1. Take an in-sample window that ends the day before the test window. 2. Choose the variant with the best criterion (here, the Sharpe) in that window. 3. Apply the chosen variant, untouched, to the following out-of-sample window. 4. Move forward one test window and repeat. The out-of-sample windows do not overlap and are chained together: their concatenation is the equity curve of someone who had followed the procedure in real time, reoptimizing every year. There are two ways of defining the in-sample window, and the figure at the top draws them with the years of this example (5 years in-sample, 1 out-of-sample): - **Anchored** (expanding): it always starts on the first day of the data and grows at each step. - **Rolling**: it has a fixed length and moves forward with the test window; the oldest data is forgotten. ```mermaid stateDiagram-v2 [*] --> Optimize Optimize --> Test: fixed parameter Test --> Advance: store OOS Advance --> Optimize: data left Advance --> [*]: end ``` The following function works for any variant matrix (rows, days; columns, variants), so it serves equally well for the futures and for the noise strategies further down: ```python def walk_forward(var, modo, anos_dentro=5, desde=2007, hasta=2025): """Each year, picks the best column of `var` in the in-sample window and applies it to the following year.""" filas, fuera = [], [] for ano in range(desde, hasta + 1): if modo == "anclado": inicio = var.index[0].year else: inicio = ano - anos_dentro dentro = var.loc[str(inicio):str(ano - 1)] s = sharpe(dentro) elegido = s.idxmax() prueba = var.loc[str(ano), elegido] filas.append({"ano": ano, "L": elegido, "sh_dentro": s.max(), "sh_fuera": sharpe(prueba)}) fuera.append(prueba) return pd.DataFrame(filas).set_index("ano"), pd.concat(fuera) anc, oos_anc = walk_forward(variantes, "anclado") mov, oos_mov = walk_forward(variantes, "movil") pd.concat({"anchored": anc, "rolling": mov}, axis=1).round(2) ``` ```output anchored rolling L sh_dentro sh_fuera L sh_dentro sh_fuera ano 2007 189 1.33 -0.51 189 1.33 -0.51 2008 189 1.04 0.82 42 0.96 -0.33 2009 189 0.99 -0.15 252 1.09 -0.09 2010 189 0.85 0.72 252 0.62 1.65 2011 252 0.92 0.77 252 0.77 0.77 2012 252 0.91 0.57 252 0.88 0.57 2013 252 0.87 2.51 252 0.90 2.51 2014 252 1.01 2.61 252 1.05 2.61 2015 252 1.12 0.76 252 1.59 0.76 2016 252 1.09 0.07 252 1.37 0.07 2017 252 1.03 0.21 378 1.28 -1.03 2018 252 0.99 1.26 252 1.26 1.26 2019 252 1.00 2.18 252 0.99 2.18 2020 252 1.06 -0.06 252 0.91 -0.06 2021 252 1.00 1.89 63 1.06 1.33 2022 252 1.04 0.73 63 1.37 1.85 2023 252 1.03 2.25 63 1.58 0.93 2024 252 1.09 1.71 21 1.65 -0.29 2025 252 1.12 -0.30 189 1.50 -0.78 ``` Each row is a decision taken with the data available before it. Two things stand out. The first is **stability**: the anchored version picks 189 from 2007 to 2010 and 252 from 2011 on, with a single change in 19 years; the rolling one changes seven times among six values, from 21 to 378 sessions, because with five years of data the difference between similar variants is noise. Between 2021 and 2024 it goes as far as choosing 63 and 21 sessions, and in 2022 the 63 works out (1.85 versus 0.73 for the anchored one): one good year does not make the procedure good. The second is that the year-by-year **out-of-sample** Sharpes range from −1.03 to 2.61: one year is little for measuring a Sharpe (its annual standard error is 1), and that is why what gets evaluated is the chained series, not each window. *Fig. — Lookback chosen each year of the walk-forward on the synthetic futures: anchored above, 5-year rolling below.* Two step lines with the lookback L chosen from 2007 to 2025, on a log scale, and a highlighted dashed line at 252, the true value. The anchored one picks 189 from 2007 to 2010 and 252 from 2011: one change. The rolling one jumps between 189, 42, 252, 378, 63 and 21: seven changes. ## How much is lost when leaving the sample The measure that summarizes a walk-forward is the Sharpe decay: the mean of the in-sample windows (what the researcher saw when choosing) versus that of the chained out-of-sample series (what they would have earned). Their ratio is sometimes called **walk-forward efficiency**, a term Pardo (2008) uses in his book on strategy evaluation (here, with the Sharpe): $$ \text{efficiency} = \frac{SR_{\text{OOS}}}{\overline{SR}_{\text{IS}}} $$ An efficiency close to 1 says the procedure is not fooling itself; one close to 0 or negative, that the in-sample Sharpe was almost all selection. As a comparison, the fixed 252 variant, with nothing optimized, over the same years: ```python resumen = {} for nombre, tabla, serie in (("anchored", anc, oos_anc), ("rolling", mov, oos_mov)): sd, sf = tabla["sh_dentro"].mean(), sharpe(serie) resumen[nombre] = {"IS": sd, "OOS": sf, "efficiency": sf / sd, "changes": int((tabla["L"].diff() != 0) .iloc[1:].sum())} fijo = sharpe(variantes.loc["2007":, 252]) resumen["252 fixed"] = {"IS": np.nan, "OOS": fijo, "efficiency": np.nan, "changes": 0} pd.DataFrame(resumen).T.round(2) ``` ```output IS OOS efficiency changes anchored 1.02 0.95 0.93 1.0 rolling 1.17 0.70 0.60 7.0 252 fixed NaN 1.09 NaN 0.0 ``` The anchored version keeps 93% of the in-sample Sharpe; the rolling one, 60%. In this data the anchored version wins because the effect is constant (252 sessions for 26 years): accumulating history only helps to measure it better. With an effect that really changed over time, the rolling version could adapt and the anchored one would fall behind; the price of that ability to adapt is the instability of the previous table. Neither of them beats the fixed 252 parameter (1.09), which is the known answer: optimizing adds nothing when the variants are similar and one of them is already the right one, and it subtracts the noise of the choice. *Fig. — Mean in-sample Sharpe against the chained out-of-sample Sharpe in three walk-forwards on synthetic data.* Paired horizontal bars. Futures with an anchored walk-forward: 1.02 in and 0.95 out. 5-year rolling: 1.17 in and 0.70 out. Noise, the best of 1,000 strategies each quarter: 3.46 in and −0.53 out (highlighted). ## The windows are parameters too The walk-forward has its own decisions: the length of the in-sample window, that of the test window, the grid of values and the selection criterion. Each one changes the result, and choosing them by looking at the out-of-sample result is another form of selection. In the rolling version, with test windows from 2012 to 2025 so that all three lengths use the same years: ```python filas = {} for anos in (3, 5, 10): t, s = walk_forward(variantes, "movil", anos_dentro=anos, desde=2012) filas[f"{anos} years"] = { "IS": t["sh_dentro"].mean(), "OOS": sharpe(s), "distinct L values": t["L"].nunique()} pd.DataFrame(filas).T.round(2) ``` ```output IS OOS distinct L values 3 years 1.39 0.88 6.0 5 years 1.24 0.87 5.0 10 years 1.09 1.12 2.0 ``` With ten years the out-of-sample series comes out better (1.12) and the choice barely moves (two values of L); with three and with five, worse and almost the same (0.88 and 0.87), with six and five distinct values. It fits what is known about this data, a stable effect that is measured better with more history, but the difference between 0.87 and 1.12 over 14 years of testing is within the standard error of a Sharpe over that span ($1/\sqrt{14} \approx 0.27$). And even if the ten-year window really were the best, if it were published because it is the one that does best out-of-sample, the result would no longer be out-of-sample. The sound practice is to fix the windows before looking (by the horizon of the signal, by the amount of data needed to measure a Sharpe with some precision) and to record every combination tried in the research journal (lesson 2.2.4), as one more trial for the Deflated Sharpe (lesson 2.5.4). | Walk-forward decision | How to fix it without looking at the result | |---|---| | In-sample length | Enough to measure the criterion: several cycles of the signal and years, not months | | Out-of-sample length | How often you would really reoptimize in production | | Grid of values | Few values, spaced on a logarithmic scale, decided by the economic horizon | | Selection criterion | The same one that will be used to decide live (Sharpe, Sharpe net of costs) | | Anchored or rolling | By the hypothesis: stable effect, anchored; changing effect, rolling | ## Noise in a walk-forward To see the signature of overfitting with no edge to hide it, the same procedure on the 1,000 strategies of `ruido2_estrategias.parquet` (all with no edge except five). The strategies only cover 2022-2025, so the windows are shorter: one year in-sample (252 sessions), one quarter out-of-sample (63), moving forward quarter by quarter. ```python est = pd.read_parquet("ruido2_estrategias.parquet").pivot( index="fecha", columns="estrategia", values="rendimiento") verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia") filas, fuera = [], [] for i in range(252, len(est), 63): dentro = est.iloc[i - 252:i] s = sharpe(dentro) elegida = s.idxmax() prueba = est.iloc[i:i + 63][elegida] fuera.append(prueba) filas.append({"desde": est.index[i].date(), "elegida": elegida, "sh_dentro": s.max(), "sh_fuera": sharpe(prueba), "real": verdad.loc[elegida, "sharpe_real"]}) ruido = pd.DataFrame(filas) print(ruido.round(2).to_string(index=False)) print(f"In-sample, mean: {ruido['sh_dentro'].mean():.2f}; " f"out-of-sample, chained: {sharpe(pd.concat(fuera)):.2f}") ``` ```output desde elegida sh_dentro sh_fuera real 2023-01-04 e0868 3.29 -1.95 0.0 2023-04-05 e0601 2.63 0.75 0.0 2023-07-07 e0462 3.23 2.27 0.0 2023-10-05 e0542 3.62 3.25 0.0 2024-01-05 e0133 3.96 -3.00 0.8 2024-04-08 e0542 4.03 0.56 0.0 2024-07-09 e0598 3.42 -0.94 0.0 2024-10-07 e0091 3.24 0.97 0.0 2025-01-07 e0032 3.43 -1.79 0.0 2025-04-09 e0897 3.64 -0.78 0.0 2025-07-11 e0023 3.79 0.80 0.0 2025-10-09 e0023 3.21 -3.74 0.0 In-sample, mean: 3.46; out-of-sample, chained: -0.53 ``` Every quarter, the best of a thousand has an in-sample Sharpe between 2.6 and 4.0; out-of-sample, the chained series loses money. Only once does the procedure pick one of the five strategies with an edge (e0133, in January 2024), and that quarter it loses. The choice changes almost every quarter: 10 different strategies in 12 windows. It is the full signature of overfitting: **a very high Sharpe in-sample, zero or negative out-of-sample, and parameters that do not repeat**. In the futures, with a real effect, the decay exists but is partial and the choice tends to stabilize; in the noise, the decay is total. The comparison also shows the limit of walk-forward validation: with 1,000 candidates and one-quarter windows, not even the five strategies with a true edge (Sharpe from 0.5 to 1.5) stand out from the noise. Walk-forward validation detects overfitting, but it does not give the good strategy a statistical power the data does not have. To make better use of little data there is purged cross-validation (lesson 2.5.3) and, to correct for the number of trials, the Deflated Sharpe and the probability of backtest overfitting (lesson 2.5.4). ## Common mistakes - **Optimizing with data that overlaps the test.** The in-sample window ends the day before the test window. The warm-up of the indicators (the 378 sessions the longest lookback needs) can take data from before the in-sample window; never from after it. - **Reoptimizing after looking.** Changing the grid, the criterion or the windows because you do not like the out-of-sample result turns the walk-forward into one more variant (previous section). - **Evaluating the windows separately.** A one-year Sharpe has a standard error of 1. What gets evaluated is the chained series and, at most, the proportion of positive windows. - **Confusing the out-of-sample curve with the strategy.** The chained series measures the **procedure** (choosing the best L every year), not a specific L. If a fixed L is going to be used in production, you have to say which one and why. - **Forgetting costs and portfolio turnover.** A procedure that changes parameter often also changes positions; costs are deducted before choosing and in the test (lesson 2.4.7). ## Exercises 1. Repeat the anchored walk-forward with two-year test windows (reoptimizing every two years). Does the choice of the parameter change? And the Sharpe decay? **Solution.** ```python filas, fuera = [], [] for ano in range(2007, 2026, 2): dentro = variantes.loc[:str(ano - 1)] elegido = sharpe(dentro).idxmax() prueba = variantes.loc[str(ano):str(ano + 1), elegido] filas.append((ano, elegido, sharpe(dentro).max())) fuera.append(prueba) t = pd.DataFrame(filas, columns=["desde", "L", "sh_dentro"]) print(t.round(2).to_string(index=False)) print(f"In-sample {t['sh_dentro'].mean():.2f}, " f"out-of-sample {sharpe(pd.concat(fuera)):.2f}") ``` ```output desde L sh_dentro 2007 189 1.33 2009 189 0.99 2011 252 0.92 2013 252 0.87 2015 252 1.12 2017 252 1.03 2019 252 1.00 2021 252 1.00 2023 252 1.03 2025 252 1.12 In-sample 1.04, out-of-sample 0.95 ``` With a two-year test there are fewer decisions (10 instead of 19) and the anchored window is the same in the years that coincide, so the choice does not change: 189 in 2007 and 2009 and 252 from 2011 on. Out-of-sample it gives 0.95, the same as the annual anchored version, with 1.04 in-sample. Reoptimizing more often improves nothing when the effect is stable; with a changing effect it could, at the cost of more noise in the choice. 2. In the walk-forward on the noise, how many times does the chosen strategy have a positive out-of-sample Sharpe? Compare it with what you would expect by chance. **Solution.** ```python positivos = int((ruido["sh_fuera"] > 0).sum()) print(f"{positivos} of {len(ruido)} quarters with a positive Sharpe") from scipy.stats import binom print(f"P(≥ {positivos} of {len(ruido)} with p = 0.5): " f"{binom.sf(positivos - 1, len(ruido), 0.5):.2f}") ``` ```output 6 of 12 quarters with a positive Sharpe P(≥ 6 of 12 with p = 0.5): 0.61 ``` It is 6 of 12, exactly what you would expect from tossing a coin (a probability of 0.61 of getting 6 or more). A procedure with an edge should give more positive quarters than a coin; this one does not. With 12 windows, though, the test has very little power: even a strategy with a Sharpe of 1 would have many negative quarters. 3. A colleague shows you a walk-forward with an out-of-sample Sharpe of 1.4 and an efficiency of 0.9, and tells you they tried it “with a few windows until it came out stable.” What do you ask them, and what would you do with the result? **Solution.** I would ask how many combinations of windows, grids and criteria they tried, and what results the others gave. If they chose the walk-forward configuration by looking at the out-of-sample series, that series is in-sample for the meta-decision: the 1.4 is the maximum of several tests. I would treat it as the Sharpe of the best of N trials and deflate it with the total number of combinations (lesson 2.5.4); if possible, I would set aside a final period that nobody has looked at and evaluate it there only once with the configuration already frozen. ## Self-assessment 1. What is the difference between an anchored and a rolling walk-forward? - A. The anchored one uses daily data and the rolling one, weekly data - B. The anchored one has no out-of-sample windows - C. The rolling one reoptimizes with the out-of-sample data of the same window - D. In the anchored one the optimization window always starts on the same day and grows; in the rolling one it has a fixed length and moves forward *Answer D: In the anchored one the optimization window always starts on the same day and grows; in the rolling one it has a fixed length and moves forward.* Both move the test window forward. The anchored one accumulates all the previous history (more data, slow adaptation); the rolling one forgets the oldest data (less data, fast adaptation and more noise in the choice). 2. In the 5-year rolling walk-forward on the futures, the mean in-sample Sharpe is 1.17 and that of the chained out-of-sample windows, 0.70. What does the difference measure? - A. Transaction costs - B. An error in the Sharpe calculation - C. The part of the in-sample Sharpe that came from choosing the best parameter with that same data: selection optimism - D. The effect of futures rolls *Answer C: The part of the in-sample Sharpe that came from choosing the best parameter with that same data: selection optimism.* In-sample you see the maximum of eight variants; out-of-sample, the result of a single one, chosen before seeing it. The decay is the direct measure of the procedure's overfitting. 3. A walk-forward over 1,000 strategies with no edge picks, each quarter, the best one of the previous year. What pattern should you expect? - A. Similar in- and out-of-sample Sharpes, around 0 - B. A very high Sharpe in-sample (the maximum of a thousand noises) and around 0 out-of-sample, with a different strategy almost every quarter - C. A negative Sharpe in-sample and a positive one out-of-sample - D. The same strategy in every window *Answer B: A very high Sharpe in-sample (the maximum of a thousand noises) and around 0 out-of-sample, with a different strategy almost every quarter.* In `ruido2_estrategias` the walk-forward gives a mean Sharpe of 3.46 in-sample and −0.53 out-of-sample, and picks 10 different strategies in 12 quarters. That is the signature of noise. 4. After a disappointing walk-forward, windows of 3, 5 and 10 years are tried and the 3-year one is published because it gives the best out-of-sample Sharpe. What is the problem? - A. The window length was chosen by looking at the out-of-sample results: the walk-forward has become one more variant of the selection - B. None: the result is still out-of-sample - C. That 3 years is too short for any strategy - D. That the anchored walk-forward is always better *Answer A: The window length was chosen by looking at the out-of-sample results: the walk-forward has become one more variant of the selection.* The meta-decisions of the procedure (window lengths, parameter grid, selection criterion) are parameters too. Choosing them by looking at the final result reintroduces the selection bias of lesson 2.5.1. 5. Why is a single split (for example, 2002-2013 in-sample and 2014-2025 out-of-sample) not enough? - A. Because it is too long - B. Because it is a single draw: the result depends on which regime falls on each side, and it says nothing about the stability of the chosen parameter - C. Because the out-of-sample period must always be one year - D. Because a single split uses information from the future *Answer B: Because it is a single draw: the result depends on which regime falls on each side, and it says nothing about the stability of the chosen parameter.* A single split gives one number. Walk-forward validation gives a series of choices and results that shows how the parameter changes and how much is lost in each period. ## Key points - In-sample you choose; out-of-sample you measure, only once. Any correction after looking at the out-of-sample data turns it into in-sample. - Walk-forward validation repeats the split: it optimizes in one window, tests in the next and moves forward. The anchored version accumulates history; the rolling one forgets it. - On the synthetic futures, the anchored version goes from a mean Sharpe of 1.02 in-sample to 0.95 out-of-sample, with a single lookback change; the 5-year rolling one, from 1.17 to 0.70, with seven changes in 19 years. - On 1,000 noise strategies, the walk-forward gives 3.46 in-sample and −0.53 out-of-sample: a large decay and unstable parameters are the signature of overfitting. - Window lengths, the grid and the criterion are parameters too: they are fixed before looking and counted as trials. ## Resources ### Research - [Intraday Seasonality IV - Field Search](https://pythonparatrading.com/en/seasonality-intraday-optimization/) · Research-0027 · 7 min ### Documentation - [scikit-learn: TimeSeriesSplit (expanding or fixed-length windows)](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) · scikit-learn.org ### Books - [Pardo, R. (2008): The Evaluation and Optimization of Trading Strategies, 2nd ed. Wiley](https://www.wiley.com/en-us/The+Evaluation+and+Optimization+of+Trading+Strategies%2C+2nd+Edition-p-9780470128015) · wiley.com - [Bailey, Borwein, López de Prado and Zhu (2014): Pseudo-Mathematics and Financial Charlatanism. Notices of the AMS 61(5), 458-471](https://doi.org/10.1090/noti1105) · doi.org ### Data - [Level 2 data: continuous futures, 24 markets, 2000-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/futuros2_continuos.parquet) · Download · futuros2_continuos.parquet - [Level 2 data: 1,000 noise strategies and five with an edge (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/ruido2_estrategias.parquet) · Download · ruido2_estrategias.parquet Previous lesson: [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/) Next lesson: [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/)