--- title: "2.5.1 Backtest biases: look-ahead, survivorship and more" description: "A museum of biases on synthetic data: same-bar execution, full-sample normalization, survivorship, the best of a thousand and narrative." url: "https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/backtest-biases.png" --- # 2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative > A backtest does not lie by accident: it always lies in the same direction, in favor of the strategy. This lesson sets up a museum of biases on synthetic data where the answer is known, triggers each one on purpose and measures how much it inflates the result. - Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/) - Module: 2.5 · Biases and validation - Duration: 60 min - Prerequisites: [1.4.4 · Universes without survivorship bias: delistings, mergers and symbol changes](https://pythonparatrading.com/en/python-trading-course/level-1/survivorship-bias-and-point-in-time-data/), [1.7.4 · Multi-timeframe analysis without look-ahead](https://pythonparatrading.com/en/python-trading-course/level-1/multi-timeframe-analysis-without-look-ahead/), [0.7.2 · Probability and distributions, correlation versus causation and simple regression](https://pythonparatrading.com/en/python-trading-course/level-0/probability-distributions-and-regression/) ## By the end of this lesson - Recognize the five biases that inflate a backtest the most (look-ahead, survivorship, selection, data snooping and narrative) and at which step of the process each one enters. - Trigger and measure look-ahead bias in two ways: executing on the signal bar and normalizing with full-sample statistics. - Quantify survivorship bias by comparing a point-in-time index with today's members projected backward. - Measure how much the best of a thousand candidates degrades out of sample and why using the validation sample as a filter turns it into training data. - Apply automatic tests (lagging the signal, truncating the history) and a checklist against narrative bias. ## A museum of biases A backtest is a simulation of what would have happened if someone had followed a set of rules in the past. The errors of such a simulation are not random: almost all of them consist of giving that someone something they did not have (information from the future, a list of the companies that survived, the choice among a thousand variants already tested), and that is why **they all push in the same direction**, in favor of the strategy. A backtest with a Sharpe of 3 is not good news to celebrate; it is an anomaly to explain. This lesson is a museum: each room triggers a bias on purpose on the level 2 data and measures how much it inflates the result. The header figure sums up the four rooms with code; the fifth, narrative, is not measured with code and closes the lesson. ```mermaid flowchart LR S[Backtest biases] --> I[Information] S --> U[Universe] S --> E[Choice] S --> N[Interpretation] I --> A1[Same bar] I --> A2[Full sample] U --> SV[Survivorship] E --> SE[Selection] E --> DS[Data snooping] N --> NA[Narrative] ``` | Bias | What the past is given | Where it enters | Symptom | Test | |---|---|---|---|---| | Look-ahead (same bar) | The price used to decide | Execution | Huge Sharpe on fast signals | Lag one more session | | Look-ahead (full sample) | Means, maxima or scales from the future | Data preparation | The signal changes when data is added | Truncate the history | | Survivorship | The list of who is still alive today | Universe | No bankruptcies or delistings | Point-in-time universe | | Selection | The best of many variants | Choice | Big drop out of sample | Count the attempts | | Data snooping | The validation sample, looked at many times | Iteration | “Validated” and then flat | Untouched final sample | | Narrative | A tailor-made explanation | Interpretation | Perfect story, no refutation | Hypothesis written beforehand | ## Look-ahead I: executing on the signal bar Look-ahead bias is using at date t a piece of data that is only known after t. The most common form is the simplest: computing the signal with today's close and applying it to today's return. In the real world, the close is known when the session is already over; the order that close triggers is executed, at the earliest, in the next session (or in the closing auction if the signal is computed a few minutes before, with a price that is not exactly the close). ```mermaid flowchart LR C[Close of t] --> S[Signal and order] S --> X[Execution at t+1] X --> R[Earns r of t+1] ``` The data is the 24 level 2 futures: continuous series with backward proportional adjustment (lesson 1.4.6), sessions from 2000 to 2025. The test portfolio gives each market the same target volatility, estimated with a 60-session exponential average, and the signal is the sign of the return over the last 252 sessions: time-series momentum, the effect that the generator plants in all markets (lesson 2.3.1). ```python import numpy as np import pandas as pd fut = pd.read_parquet("futuros2_continuos.parquet") precio = fut.pivot(index="fecha", columns="mercado", values="cierre_ajustado") r = precio.pct_change(fill_method=None) vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252) def sharpe(x): x = x.dropna() return x.mean() / x.std() * np.sqrt(252) def cartera(senal, retraso=1): """Each market with 40% / 24 target volatility.""" peso = senal * 0.40 / vol / precio.shape[1] return (peso.shift(retraso) * r).sum(axis=1, min_count=1) print(precio.shape, precio.index[0].date(), precio.index[-1].date()) ``` ```output (6539, 24) 2000-01-03 2025-12-31 ``` `retraso` is the number of sessions between the signal date and the return it earns. With `retraso=1`, the weight computed at the close of t multiplies the return of t+1: that is the honest execution. With `retraso=0`, it multiplies the return of t, the same one that has already entered the signal and the volatility. ```python tsmom = np.sign(precio.pct_change(252, fill_method=None)) bien = cartera(tsmom, retraso=1).loc["2001":] mal = cartera(tsmom, retraso=0).loc["2001":] print(f"Execution at t+1: Sharpe {sharpe(bien):.2f}") print(f"Execution on the same bar: Sharpe {sharpe(mal):.2f}") ``` ```output Execution at t+1: Sharpe 1.00 Execution on the same bar: Sharpe 3.16 ``` The 252-session momentum has a real edge in this data, and the honest execution captures it. Executing on the same bar multiplies the Sharpe by more than three. The signal only changes sign when the one-year return crosses zero, so on most days the lag does not matter; the deception is concentrated on the days when today's return is the one that causes the crossing, and in the volatility scaling, which with `retraso=0` already “knows” whether today was a turbulent day. The faster the signal, the larger the part of it that today's return explains: ```python filas = {} for n in (20, 50, 100, 200): cruce = np.sign(precio / precio.rolling(n).mean() - 1) filas[f"price vs. mean {n}"] = { f"lag {k}": sharpe(cartera(cruce, k).loc["2001":]) for k in (0, 1, 2)} pd.DataFrame(filas).T.round(2) ``` ```output lag 0 lag 1 lag 2 price vs. mean 20 17.04 0.26 0.37 price vs. mean 50 10.95 0.55 0.50 price vs. mean 100 7.83 0.74 0.66 price vs. mean 200 5.67 0.84 0.92 ``` With a 20-session mean, executing on the same bar gives a double-digit Sharpe: no real system comes close to that, and it is the first clue that something is wrong. The table also shows the **lag test**: between `lag 1` and `lag 2` the Sharpe changes little, because an honest signal does not depend on a single day; between `lag 0` and `lag 1` there is a cliff. If the backtest you are reviewing looks like the `lag 0` column, lag the signal one session: if the result collapses, the edge was the price of a day you did not have. ## Look-ahead II: normalizing with the full sample The second form is subtler, because there is no misplaced `shift`. It happens in data preparation: a variable is standardized with the mean and standard deviation of the **whole** sample, scaled with the historical maximum, clipped at the percentiles of the whole series, or a `StandardScaler` or a PCA is fitted on all the data before splitting into training and test. Each of those statistics contains the future. The extreme example: a mean reversion rule that buys when the price is below “its mean” and sells when it is above. With the full-sample mean, the rule knows in 2001 where the price will be on average until 2025. With an expanding mean (only the history up to each day), it does not: ```python lp = np.log(precio) z_total = (lp - lp.mean()) / lp.std() z_exp = (lp - lp.expanding(252).mean()) / lp.expanding(252).std() for nombre, z in (("full sample", z_total), ("expanding", z_exp)): regla = -z.clip(-2, 2) # buy cheap, sell expensive print(f"{nombre:>16}: Sharpe {sharpe(cartera(regla)['2001':]):.2f}") ``` ```output full sample: Sharpe 0.67 expanding: Sharpe -0.65 ``` Execution is honest in both cases (`retraso=1`), and even so the full-sample version wins and the expanding one loses. In this data the spot prices of commodities, currencies and bonds do revert toward an anchor, but the futures curve already prices it in: at the peaks it trades in backwardation and at the troughs in contango, so the continuous series, which is what gets traded, does not keep that reversion (only 10% of the reversion carry has a premium, according to the README). What is left is momentum, and a contrarian rule loses. With the full-sample mean, the rule knows the level to which the price “will return” because that level was computed with the prices it returned to. It is an impossible result disguised as statistics. This bias cannot be seen by reading the backtest code, because it comes earlier. It shows up with a mechanical test: **truncating the history**. An honest signal at date t depends only on data up to t, so computing it with the whole series or with the series truncated at t has to give exactly the same values up to t. ```python def prueba_corte(fn, datos, corte="2010-12-31"): """Maximum difference up to `corte` between the signal computed with the full history and with the history truncated at `corte`.""" completa = fn(datos).loc[:corte] cortada = fn(datos.loc[:corte]) return float((completa - cortada).abs().max().max()) def f_total(lp): return (lp - lp.mean()) / lp.std() def f_exp(lp): return (lp - lp.expanding(252).mean()) / lp.expanding(252).std() def f_tsmom(p): return np.sign(p.pct_change(252, fill_method=None)) for fn, datos in ((f_total, lp), (f_exp, lp), (f_tsmom, precio)): print(f"{fn.__name__:>8}: {prueba_corte(fn, datos):.6f}") ``` ```output f_total: 2.975494 f_exp: 0.000000 f_tsmom: 0.000000 ``` The full-sample z-score changes by up to several standard deviations when fifteen years of data are added; the expanding one and the momentum do not change at all. The test costs three lines and it is worth running it on every function that produces signals, at several truncation dates, as one more `pytest` test (lesson 0.4.5). ## Survivorship: today's members projected backward Survivorship bias is choosing the universe with information from the end: “the stocks in the index”, when the list that gets downloaded is today's. The companies that went bankrupt, the ones that were acquired and the ones that left the index after collapsing are not on that list, so a portfolio built with it cannot hold them in 2008. Lesson 1.4.4 covered it with level 1 data; the level 2 universe has a 100-member index with point-in-time history and ten documented delistings, four of them bankruptcies with a delisting return of −100%. ```python aj = pd.read_parquet("universo2_ajustados.parquet").pivot( index="fecha", columns="ticker", values="cierre_ajustado") miembros = pd.read_csv("universo2_miembros.csv", parse_dates=["alta", "baja"]) eventos = pd.read_csv("universo2_eventos.csv", parse_dates=["fecha"]) ru = aj.pct_change(fill_method=None) dentro = pd.DataFrame(False, index=ru.index, columns=ru.columns) fin_datos = ru.index[-1] + pd.Timedelta(days=1) for m in miembros.itertuples(): hasta = m.baja if pd.notna(m.baja) else fin_datos dentro.loc[(ru.index >= m.alta) & (ru.index < hasta), m.ticker] = True # the delisting return enters the session after the last one salidas = eventos.dropna(subset=["rentabilidad_deslistado"]) for e in salidas.itertuples(): i = ru.index.get_loc(aj[e.ticker].last_valid_index()) + 1 if i < len(ru): ru.iloc[i, ru.columns.get_loc(e.ticker)] = ( e.rentabilidad_deslistado) dentro.iloc[i, ru.columns.get_loc(e.ticker)] = True print(miembros["ticker"].nunique(), "companies passed through the index") print(salidas.groupby("evento").size().to_string()) ``` ```output 115 companies passed through the index evento adquisicion_efectivo 4 deslistado_quiebra 4 exclusion_voluntaria 1 fusion_acciones 1 ``` Two implementation details count as much as the list. The first, the **delisting return**: when a stock stops trading, the last return is not that of the last close, but the one from that close to the final payment (−100% in a bankruptcy; in a cash acquisition, the spread that remained up to the offer price, between 0.03% and 0.22% in this data). Without it, the bankruptcy disappears from the portfolio with its last price of 0.85 USD as if it had been sold at that price. The second, the **date**: membership is checked at each session, with the `alta` (entry) and `baja` (exit) of each interval. A portfolio that rebalances monthly with `universo2_caracteristicas.parquet` has the same problem solved another way: each delisted stock has a last row at the month-end of its exit, with no price and with a `rentabilidad_mes` that already includes the delisting return. Those rows are not used to form signals (they are filtered out by non-empty `cierre`), but they have to enter the portfolio return. ```python def anual(x): return (1 + x.fillna(0)).prod() ** (252 / len(x)) - 1 pit = ru.where(dentro).mean(axis=1).iloc[1:] hoy = miembros.loc[miembros["baja"].isna(), "ticker"] superv = ru[hoy].mean(axis=1).iloc[1:] for nombre, x in (("point-in-time", pit), ("today's members", superv)): print(f"{nombre:>15}: {anual(x):.2%} annual, Sharpe {sharpe(x):.2f}") fuera = set(miembros["ticker"]) - set(hoy) quiebras = salidas.loc[salidas["evento"] == "deslistado_quiebra", "ticker"] print(len(fuera), "companies not on today's list; bankruptcies:", sorted(fuera & set(quiebras))) ``` ```output point-in-time: 5.75% annual, Sharpe 0.38 today's members: 6.95% annual, Sharpe 0.44 15 companies not on today's list; bankruptcies: ['QXKV', 'QXQT', 'QXYX'] ``` *Fig. — Equal-weighted index of the synthetic universe, 2006-2025: point-in-time with delistings against today's 100 members projected backwards.* Two monthly equity curves starting at 1 in January 2006. Today's members (highlighted) end at 3.83, 6.95% a year; the point-in-time one, with bankruptcies and their delisting returns, at 3.05, 5.75% a year. Both fall in 2008 and 2020; the gap grows slowly and widens most from 2019. That is 1.2 points of return a year, which compound over twenty years: 1 USD from 2006 ends at 3.83 USD with today's members and at 3.05 USD with the point-in-time index. The difference comes from the 15 companies that passed through the index and are not on today's list: three of the four bankruptcies, with their −100% (the fourth, QXZT, never made it into the index), the ones that left the index after collapsing and the acquired ones. And the bias does not only affect an index portfolio: any strategy that buys losers (reversion, value) looks better with survivors, because the losers that kept falling until they disappeared are not in the sample. Momentum, conversely, can look worse than it is for the same reason. ## Selection and data snooping: the best of a thousand Selection bias appears when choosing after the fact what to show: the best variant of a rule, the market where it worked, the period in which it turned out well. **Data snooping** is its iterative version: reusing the same data to test idea after idea until one works. Both have the same arithmetic. If N candidates with no edge are tested, each with a measured Sharpe that is noise, the best of them is not “average” noise: it is the maximum of N noises. `ruido2_estrategias.parquet` holds the daily returns of 1,000 fictitious strategies over the sessions from 2022 to 2025, in 50 families of 20 correlated variants. All have zero edge except five, whose true Sharpe is in `ruido2_verdad.csv`. The test: choose with 2022-2023 and look at 2024-2025. ```python est = pd.read_parquet("ruido2_estrategias.parquet").pivot( index="fecha", columns="estrategia", values="rendimiento") verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia") def sharpe_cols(df): return df.mean() / df.std() * np.sqrt(252) sh_is = sharpe_cols(est.loc[:"2023"]) sh_oos = sharpe_cols(est.loc["2024":]) top = sh_is.nlargest(10).index tabla = pd.DataFrame({"IS 2022-23": sh_is[top], "OOS 2024-25": sh_oos[top], "real": verdad.loc[top, "sharpe_real"]}) print(tabla.round(2)) print(f"Top 10 mean: IS {sh_is[top].mean():.2f}, " f"OOS {sh_oos[top].mean():.2f}") ``` ```output IS 2022-23 OOS 2024-25 real estrategia e0133 2.98 -0.04 0.8 e0542 2.91 -0.02 0.0 e0462 2.40 0.51 0.0 e0781 2.36 -1.47 0.0 e0518 2.28 0.85 1.5 e0565 2.08 -0.67 0.0 e0060 1.98 -0.26 0.0 e0124 1.91 0.92 0.0 e0950 1.89 -0.57 0.0 e0835 1.88 0.09 0.0 Top 10 mean: IS 2.27, OOS -0.07 ``` *Fig. — The 10 best of 1,000 synthetic strategies in 2022-2023 and their Sharpe in 2024-2025.* Slope chart: ten lines from the 2022-2023 Sharpe, between 1.88 and 2.98, to the 2024-2025 one, between −1.47 and 0.92. The mean (highlighted) falls from 2.27 to −0.07. The two strategies with a real edge, e0133 and e0518, go from 2.98 to −0.04 and from 2.28 to 0.85. The top 10 of the first half has a mean Sharpe above 2, and in the second its mean is negative. Two of the ten have a real edge (e0133 and e0518, with true Sharpe 0.8 and 1.5); the other eight are noise that had two good years. The sample of each half, 501 sessions, gives a standard error of the annual Sharpe of $\sqrt{252/501} \approx 0.71$: with a thousand candidates, the maximum of the noise is around three standard errors above zero. Extreme value theory gives an approximation for the expected maximum of N independent Sharpes with no edge, with $\gamma \approx 0.5772$ (the Euler-Mascheroni constant) and $\Phi^{-1}$ the inverse of the normal: $$ \mathbb{E}\left[\max_n \widehat{SR}_n\right] \approx \sigma_{SR}\left[(1-\gamma)\,\Phi^{-1}\!\left(1-\tfrac{1}{N}\right)+\gamma\,\Phi^{-1}\!\left(1-\tfrac{1}{N e}\right)\right] $$ ```python from scipy.stats import norm def max_esperado(n, error_tipico): g = np.euler_gamma z = ((1 - g) * norm.ppf(1 - 1 / n) + g * norm.ppf(1 - 1 / (n * np.e))) return z * error_tipico et = np.sqrt(252 / 501) for n in (1, 10, 100, 1000): m = max_esperado(n, et) if n > 1 else 0.0 print(f"N = {n:>4}: expected maximum {m:.2f}") ruido = verdad.index[verdad["sharpe_real"] == 0] print(f"Noise maximum in 2022-23: {sh_is[ruido].max():.2f}") ``` ```output N = 1: expected maximum 0.00 N = 10: expected maximum 1.12 N = 100: expected maximum 1.79 N = 1000: expected maximum 2.31 Noise maximum in 2022-23: 2.91 ``` With a single candidate, the expected Sharpe of a strategy with no edge is 0; with a thousand, around 2.3. The observed maximum of the noise, 2.91, is even higher: it is a single realization, and the maximum of a thousand noises has a lot of dispersion. The formula assumes independent candidates; here they are not (20 variants per family, correlation 0.5), which reduces the **effective** number of attempts and the expected maximum. Lesson 2.5.4 turns this calculation into a test, the Deflated Sharpe Ratio, and lesson 2.1.8 covers multiple testing corrections. Snooping is more treacherous, because it looks prudent. A researcher who already knows that the best of a thousand is misleading decides to require the strategy to be good in both halves: ```python for umbral in (0.5, 1.0): pasan = sh_is.index[(sh_is > umbral) & (sh_oos > umbral)] reales = int((verdad.loc[pasan, "sharpe_real"] > 0).sum()) print(f"Sharpe > {umbral} in both halves: {len(pasan)} " f"strategies, {reales} with a real edge") ``` ```output Sharpe > 0.5 in both halves: 56 strategies, 3 with a real edge Sharpe > 1.0 in both halves: 7 strategies, 0 with a real edge ``` With a threshold of 0.5, 56 strategies pass, and only three have an edge. With a threshold of 1, seven pass, and none do: the five that do have one are left out in one of the two halves, because with 501 sessions even a true Sharpe of 1.5 can measure below 1. The second half stopped being a validation sample the moment it was used to choose. An out-of-sample set is only independent if it is looked at **once**, at the end, with the strategy already decided. Every time it is looked at and something is corrected, it becomes part of the training. Lessons 2.5.2 and 2.5.3 organize that separation (walk-forward and purged cross-validation) and lesson 2.2.4 explains how to log each attempt so that N can be counted. White (2000) proposed one of the first formal tests of data snooping, the *Reality Check*, which compares the best rule with the distribution of the maximum of all those tested. ## Narrative bias The last bias is not in the code. It is the tendency to build, after seeing a result, an explanation that makes it look inevitable, and to take it as proof. Once the backtest of a variant turns out well, there is always a story: “funds rebalance at month-end”, “retail investors overreact to news”, “carry pays for crisis risk”. The story may be true. The problem is the order: it was chosen because it fit the data, and with 1,000 different results there would have been 1,000 different stories, all convincing. The symptoms are recognizable: the explanation comes after the result; nobody wrote down beforehand what result would refute it; the chosen parameters have a tailor-made economic justification (“63 sessions because it is a quarter”); the bad periods are explained as exceptions (“2022 was an odd year”) and the good ones as the rule; and the story predicts nothing that has not already been looked at. In the academic literature it has a name, HARKing (*hypothesizing after the results are known*), and it is one of the reasons why Harvey, Liu and Zhu (2016) call for more demanding thresholds for new factors. The defense is procedural. This list is filled in **before** looking at the test data, and saved with a date in the research journal (lesson 2.2.4): | Question | Minimum acceptable answer | |---|---| | What is the hypothesis? | One sentence with the effect and its sign, written before the backtest | | Who pays for the edge and why would they keep paying? | A mechanism: risk premium, constraint on others, behavioral error (lesson 2.2.1) | | What result would refute it? | A concrete threshold: “Sharpe < 0.3 in 2015-2025”, “IC with the opposite sign in two sectors” | | What does it predict besides the main result? | Something not yet looked at: another market, another period, a relationship with a variable | | How many variants have been tested? | The number, including the ones that failed (for the Deflated Sharpe, lesson 2.5.4) | | Which data has not been touched? | A reserved period or universe, looked at only once | | What does the placebo say? | The same method applied to a variable with no effect (in level 2, `puntuacion_esg`) | A useful test: also write the opposite story. If for the opposite result (“the rule loses because…”) an equally convincing explanation is found just as easily, the explanation is not providing information. ## How much each bias inflates The header figure puts the four rooms with code on the same scale, the annual Sharpe: same-bar execution takes momentum from 1.00 to 3.16; full-sample normalization turns a losing rule (−0.65) into a winning one (0.67); survivors raise the index Sharpe from 0.38 to 0.44; and selecting the best of a thousand gives a top 10 of 2.3 that is worth −0.07 out of sample. The magnitudes come from this synthetic data, but the order is typical: look-ahead produces the most absurd results (and that is why it is the easiest to detect), while survivorship and selection give plausible results, which are the most dangerous. | Bias | Without bias | With bias | Automatic defense | |---|---|---|---| | Same bar (252 momentum) | 1.00 | 3.16 | Lag test | | Full sample (reversion) | −0.65 | 0.67 | Truncation test | | Survivorship (index) | 0.38 | 0.44 | Point-in-time universe and delistings | | Selection (top 10 of 1,000) | −0.07 out | 2.3 in | Count N and deflate (2.5.4) | Lesson 2.5.8 gathers these tests into an “honest backtest” checklist before moving on to paper trading. ## Exercises 1. Apply the lag test to the “price vs. 50-session mean” rule with lags from 1 to 5 sessions. Does it behave like an honest signal? **Solution.** ```python cruce50 = np.sign(precio / precio.rolling(50).mean() - 1) pd.Series({k: sharpe(cartera(cruce50, k).loc["2001":]) for k in range(0, 6)}, name="Sharpe").round(2) ``` ```output 0 10.95 1 0.55 2 0.50 3 0.54 4 0.58 5 0.53 Name: Sharpe, dtype: float64 ``` From 1 to 5 sessions the Sharpe ranges between 0.50 and 0.58, with no trend or jumps: that is what to expect from a trend signal, whose edge is spread over many sessions. The only jump is between 0 and 1, which is the same-bar bias. A sharp drop between 1 and 2 would be suspicious for another reason: an edge that lives in a single session is usually a data alignment error, or an edge that costs eat up (lesson 2.4.7). 2. Repeat the point-in-time index without the delisting returns (that is, without the −100% of the bankruptcies). What part of the survivorship bias does that detail explain? **Solution.** ```python ru_sin = aj.pct_change(fill_method=None) dentro_sin = dentro.copy() for e in salidas.itertuples(): i = ru.index.get_loc(aj[e.ticker].last_valid_index()) + 1 if i < len(ru): dentro_sin.iloc[i, ru.columns.get_loc(e.ticker)] = False pit_sin = ru_sin.where(dentro_sin).mean(axis=1).iloc[1:] print(f"Point-in-time with delistings: {anual(pit):.2%}") print(f"Point-in-time without delistings: {anual(pit_sin):.2%}") print(f"Today's members: {anual(superv):.2%}") ``` ```output Point-in-time with delistings: 5.75% Point-in-time without delistings: 5.96% Today's members: 6.95% ``` Without the delisting return, the four bankruptcies leave the portfolio at their last price, between 0.66 and 1.12 USD, instead of losing everything. The index gains some return, but stays far from the survivors: most of the bias is not in the last day of the bankruptcies, but in the 320 sessions of prior decline and in the companies that left the index after collapsing, which a universe of survivors never includes. 3. A colleague shows you a weekly reversal strategy on stocks with a Sharpe of 1.8 between 2010 and 2020 and explains that it works “because retail investors overreact to weekend news”. Write three questions from the checklist that you would ask before looking at the code and which answer would make you suspicious. **Solution.** First: “how many variants did you test (horizons, thresholds, universes) before this one?”. I would be suspicious of “I don't know” or of a large number without deflating (lesson 2.5.4). Second: “which universe did you use and what date is the list from?”. If it is the current list of an index, the reversal is inflated by survivorship: the losers that kept falling are not there. Third: “what result would have refuted the retail investor story?”. If the explanation came after the backtest and predicts nothing that has not been looked at (for example, that the effect is larger in stocks with more retail trading or after weekends with news), it is narrative. I would also ask for the lag test: weekly reversal is very sensitive to executing at the same close that computes the signal. ## Self-assessment 1. A 252-session momentum portfolio gives a Sharpe of 1.00 when the position is taken the day after the signal and 3.16 when it is applied to the return of the same session. What explains the difference? - A. That momentum works better in the short term - B. That transaction costs are lower with lag 0 - C. That the proportional adjustment of continuous series favors lag 0 - D. That with lag 0 the position earns the return of the day that has already entered the signal and the volatility: information that did not exist when deciding *Answer D: That with lag 0 the position earns the return of the day that has already entered the signal and the volatility: information that did not exist when deciding.* The signal and the volatility scaling are computed with today's close. Applying them to today's return is trading at the close with a price that is only known after the close. The jump from 1.00 to 3.16 is all look-ahead. 2. What does the “truncation test” (computing the signal with the history up to a date and comparing it with the one computed with the full history) detect? - A. Whether the signal has too many parameters - B. Whether any value of the signal at a date changes when later data is added, that is, whether it uses the future - C. Whether the signal is stationary - D. Whether there are gaps in the data *Answer B: Whether any value of the signal at a date changes when later data is added, that is, whether it uses the future.* An honest signal at date t depends only on data up to t, so it does not change when later data is added. The z-score with the full-sample mean changes at every date: it fails the test. 3. In the synthetic universe, the point-in-time equal-weighted index returns 5.75% a year and that of today's 100 members projected backward, 6.95%. Where does the difference come from? - A. From today's members being larger companies - B. From dividends - C. From today's list excluding the companies that went bankrupt or left the index after falling: the past is filled with winners known after the fact - D. From a rounding error *Answer C: From today's list excluding the companies that went bankrupt or left the index after falling: the past is filled with winners known after the fact.* The list of current members is information from 2025. Using it in 2006 guarantees that none of the four bankruptcies (delisting return −100%) nor the ones dropped for becoming too small enter the portfolio. 4. Out of 1,000 strategies with no edge except five, seven exceed a Sharpe of 1 in both 2022-2023 and 2024-2025. How many of those seven have a real edge? - A. None: with a thousand candidates, passing two filters by chance is to be expected, and using the second half as a filter makes it part of the selection - B. All seven: passing two periods is a validation - C. Exactly five - D. It cannot be known *Answer A: None: with a thousand candidates, passing two filters by chance is to be expected, and using the second half as a filter makes it part of the selection.* In this data the truth is known (`ruido2_verdad.csv`): all seven have a true Sharpe of 0. A validation period is only independent if it is looked at once, after having chosen. 5. What is narrative bias? - A. Building, after seeing the result, a convincing explanation that makes it look inevitable, and taking it as proof - B. Using news data in a strategy - C. Preferring trend strategies - D. Publishing only the strategies that work *Answer A: Building, after seeing the result, a convincing explanation that makes it look inevitable, and taking it as proof.* The story comes after the data and always fits, because it was chosen to fit. The defense is to write down the hypothesis, the mechanism and what would refute it before looking (lessons 2.2.1 and 2.2.4). ## Key points - All backtest biases push in the same direction: in favor of the strategy. A good result calls for more tests, not fewer. - Look-ahead: executing on the signal bar raises the Sharpe of futures momentum from 1.00 to 3.16; normalizing with the full-sample mean turns a losing rule (−0.65) into a winning one (0.67). - Two automatic tests catch almost all look-ahead: lagging the signal one more session and recomputing it with the history truncated at a date. - Survivorship: today's members projected backward return 6.95% a year versus 5.75% for the point-in-time index. - Selection and snooping: the top 10 of a thousand strategies (mean Sharpe 2.3 in 2022-2023) gives −0.07 in 2024-2025, and filtering on both halves leaves seven false positives. - Narrative is not measured with code: it is prevented by writing down hypothesis, mechanism and refutation criterion before looking at the data. ## Resources ### Research - [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min ### Books - [White, H. (2000): A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126](https://doi.org/10.1111/1468-0262.00152) · doi.org - [Bailey, Borwein, López de Prado and Zhu (2014): Pseudo-Mathematics and Financial Charlatanism. Notices of the AMS 61(5), 458-471](https://doi.org/10.1090/noti1105) · doi.org - [Harvey, Liu and Zhu (2016): … and the Cross-Section of Expected Returns. Review of Financial Studies 29(1), 5-68](https://doi.org/10.1093/rfs/hhv059) · doi.org ### Data - [Level 2 data: continuous futures, 24 markets, 2000-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/futuros2_continuos.parquet) · Download · futuros2_continuos.parquet - [Level 2 data: point-in-time index membership (CSV)](https://pythonparatrading.com/curso/datos/nivel-2/universo2_miembros.csv) · Download · universo2_miembros.csv - [Level 2 data: 1,000 noise strategies and five with an edge (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/ruido2_estrategias.parquet) · Download · ruido2_estrategias.parquet Previous lesson: [2.4.8 · Fill modeling: limit orders, queue, partial fills, gaps and stops](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-order-fill-modeling/) Next lesson: [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/)