Backtest biases: look-ahead, survivorship, selection and narrative.

A museum of biases

A backtest is a simulation of what would have happened if someone had followed a set of rules in the past. The errors of such a simulation are not random: almost all of them consist of giving that someone something they did not have (information from the future, a list of the companies that survived, the choice among a thousand variants already tested), and that is why they all push in the same direction, in favor of the strategy. A backtest with a Sharpe of 3 is not good news to celebrate; it is an anomaly to explain.

This lesson is a museum: each room triggers a bias on purpose on the level 2 data and measures how much it inflates the result. The header figure sums up the four rooms with code; the fifth, narrative, is not measured with code and closes the lesson.

flowchart LR
  S[Backtest biases] --> I[Information]
  S --> U[Universe]
  S --> E[Choice]
  S --> N[Interpretation]
  I --> A1[Same bar]
  I --> A2[Full sample]
  U --> SV[Survivorship]
  E --> SE[Selection]
  E --> DS[Data snooping]
  N --> NA[Narrative]

Look-ahead I: executing on the signal bar

Look-ahead bias is using at date t a piece of data that is only known after t. The most common form is the simplest: computing the signal with today's close and applying it to today's return. In the real world, the close is known when the session is already over; the order that close triggers is executed, at the earliest, in the next session (or in the closing auction if the signal is computed a few minutes before, with a price that is not exactly the close).

flowchart LR
  C[Close of t] --> S[Signal and order]
  S --> X[Execution at t+1]
  X --> R[Earns r of t+1]

The data is the 24 level 2 futures: continuous series with backward proportional adjustment (lesson 1.4.6), sessions from 2000 to 2025. The test portfolio gives each market the same target volatility, estimated with a 60-session exponential average, and the signal is the sign of the return over the last 252 sessions: time-series momentum, the effect that the generator plants in all markets (lesson 2.3.1).

import numpy as np
import pandas as pd

fut = pd.read_parquet("futuros2_continuos.parquet")
precio = fut.pivot(index="fecha", columns="mercado",
                   values="cierre_ajustado")
r = precio.pct_change(fill_method=None)
vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252)

def sharpe(x):
    x = x.dropna()
    return x.mean() / x.std() * np.sqrt(252)

def cartera(senal, retraso=1):
    """Each market with 40% / 24 target volatility."""
    peso = senal * 0.40 / vol / precio.shape[1]
    return (peso.shift(retraso) * r).sum(axis=1, min_count=1)

print(precio.shape, precio.index[0].date(), precio.index[-1].date())

(6539, 24) 2000-01-03 2025-12-31

retraso is the number of sessions between the signal date and the return it earns. With retraso=1, the weight computed at the close of t multiplies the return of t+1: that is the honest execution. With retraso=0, it multiplies the return of t, the same one that has already entered the signal and the volatility.

tsmom = np.sign(precio.pct_change(252, fill_method=None))
bien = cartera(tsmom, retraso=1).loc["2001":]
mal = cartera(tsmom, retraso=0).loc["2001":]
print(f"Execution at t+1:          Sharpe {sharpe(bien):.2f}")
print(f"Execution on the same bar: Sharpe {sharpe(mal):.2f}")

Execution at t+1:          Sharpe 1.00
Execution on the same bar: Sharpe 3.16

The 252-session momentum has a real edge in this data, and the honest execution captures it. Executing on the same bar multiplies the Sharpe by more than three. The signal only changes sign when the one-year return crosses zero, so on most days the lag does not matter; the deception is concentrated on the days when today's return is the one that causes the crossing, and in the volatility scaling, which with retraso=0 already “knows” whether today was a turbulent day. The faster the signal, the larger the part of it that today's return explains:

filas = {}
for n in (20, 50, 100, 200):
    cruce = np.sign(precio / precio.rolling(n).mean() - 1)
    filas[f"price vs. mean {n}"] = {
        f"lag {k}": sharpe(cartera(cruce, k).loc["2001":])
        for k in (0, 1, 2)}
pd.DataFrame(filas).T.round(2)

                    lag 0  lag 1  lag 2
price vs. mean 20   17.04   0.26   0.37
price vs. mean 50   10.95   0.55   0.50
price vs. mean 100   7.83   0.74   0.66
price vs. mean 200   5.67   0.84   0.92

With a 20-session mean, executing on the same bar gives a double-digit Sharpe: no real system comes close to that, and it is the first clue that something is wrong. The table also shows the lag test: between lag 1 and lag 2 the Sharpe changes little, because an honest signal does not depend on a single day; between lag 0 and lag 1 there is a cliff. If the backtest you are reviewing looks like the lag 0 column, lag the signal one session: if the result collapses, the edge was the price of a day you did not have.

Look-ahead II: normalizing with the full sample

The second form is subtler, because there is no misplaced shift. It happens in data preparation: a variable is standardized with the mean and standard deviation of the whole sample, scaled with the historical maximum, clipped at the percentiles of the whole series, or a StandardScaler or a PCA is fitted on all the data before splitting into training and test. Each of those statistics contains the future.

The extreme example: a mean reversion rule that buys when the price is below “its mean” and sells when it is above. With the full-sample mean, the rule knows in 2001 where the price will be on average until 2025. With an expanding mean (only the history up to each day), it does not:

lp = np.log(precio)
z_total = (lp - lp.mean()) / lp.std()
z_exp = (lp - lp.expanding(252).mean()) / lp.expanding(252).std()

for nombre, z in (("full sample", z_total),
                  ("expanding", z_exp)):
    regla = -z.clip(-2, 2)  # buy cheap, sell expensive
    print(f"{nombre:>16}: Sharpe {sharpe(cartera(regla)['2001':]):.2f}")

     full sample: Sharpe 0.67
       expanding: Sharpe -0.65

Execution is honest in both cases (retraso=1), and even so the full-sample version wins and the expanding one loses. In this data the spot prices of commodities, currencies and bonds do revert toward an anchor, but the futures curve already prices it in: at the peaks it trades in backwardation and at the troughs in contango, so the continuous series, which is what gets traded, does not keep that reversion (only 10% of the reversion carry has a premium, according to the README). What is left is momentum, and a contrarian rule loses. With the full-sample mean, the rule knows the level to which the price “will return” because that level was computed with the prices it returned to. It is an impossible result disguised as statistics.

This bias cannot be seen by reading the backtest code, because it comes earlier. It shows up with a mechanical test: truncating the history. An honest signal at date t depends only on data up to t, so computing it with the whole series or with the series truncated at t has to give exactly the same values up to t.

def prueba_corte(fn, datos, corte="2010-12-31"):
    """Maximum difference up to `corte` between the signal computed
    with the full history and with the history truncated at `corte`."""
    completa = fn(datos).loc[:corte]
    cortada = fn(datos.loc[:corte])
    return float((completa - cortada).abs().max().max())

def f_total(lp):
    return (lp - lp.mean()) / lp.std()

def f_exp(lp):
    return (lp - lp.expanding(252).mean()) / lp.expanding(252).std()

def f_tsmom(p):
    return np.sign(p.pct_change(252, fill_method=None))

for fn, datos in ((f_total, lp), (f_exp, lp), (f_tsmom, precio)):
    print(f"{fn.__name__:>8}: {prueba_corte(fn, datos):.6f}")

 f_total: 2.975494
   f_exp: 0.000000
 f_tsmom: 0.000000

The full-sample z-score changes by up to several standard deviations when fifteen years of data are added; the expanding one and the momentum do not change at all. The test costs three lines and it is worth running it on every function that produces signals, at several truncation dates, as one more pytest test (lesson 0.4.5).

Survivorship: today's members projected backward

Survivorship bias is choosing the universe with information from the end: “the stocks in the index”, when the list that gets downloaded is today's. The companies that went bankrupt, the ones that were acquired and the ones that left the index after collapsing are not on that list, so a portfolio built with it cannot hold them in 2008. Lesson 1.4.4 covered it with level 1 data; the level 2 universe has a 100-member index with point-in-time history and ten documented delistings, four of them bankruptcies with a delisting return of −100%.

aj = pd.read_parquet("universo2_ajustados.parquet").pivot(
    index="fecha", columns="ticker", values="cierre_ajustado")
miembros = pd.read_csv("universo2_miembros.csv",
                       parse_dates=["alta", "baja"])
eventos = pd.read_csv("universo2_eventos.csv", parse_dates=["fecha"])

ru = aj.pct_change(fill_method=None)
dentro = pd.DataFrame(False, index=ru.index, columns=ru.columns)
fin_datos = ru.index[-1] + pd.Timedelta(days=1)
for m in miembros.itertuples():
    hasta = m.baja if pd.notna(m.baja) else fin_datos
    dentro.loc[(ru.index >= m.alta) & (ru.index < hasta),
               m.ticker] = True

# the delisting return enters the session after the last one
salidas = eventos.dropna(subset=["rentabilidad_deslistado"])
for e in salidas.itertuples():
    i = ru.index.get_loc(aj[e.ticker].last_valid_index()) + 1
    if i < len(ru):
        ru.iloc[i, ru.columns.get_loc(e.ticker)] = (
            e.rentabilidad_deslistado)
        dentro.iloc[i, ru.columns.get_loc(e.ticker)] = True

print(miembros["ticker"].nunique(), "companies passed through the index")
print(salidas.groupby("evento").size().to_string())

115 companies passed through the index
evento
adquisicion_efectivo    4
deslistado_quiebra      4
exclusion_voluntaria    1
fusion_acciones         1

Two implementation details count as much as the list. The first, the delisting return: when a stock stops trading, the last return is not that of the last close, but the one from that close to the final payment (−100% in a bankruptcy; in a cash acquisition, the spread that remained up to the offer price, between 0.03% and 0.22% in this data). Without it, the bankruptcy disappears from the portfolio with its last price of 0.85 USD as if it had been sold at that price. The second, the date: membership is checked at each session, with the alta (entry) and baja (exit) of each interval. A portfolio that rebalances monthly with universo2_caracteristicas.parquet has the same problem solved another way: each delisted stock has a last row at the month-end of its exit, with no price and with a rentabilidad_mes that already includes the delisting return. Those rows are not used to form signals (they are filtered out by non-empty cierre), but they have to enter the portfolio return.

def anual(x):
    return (1 + x.fillna(0)).prod() ** (252 / len(x)) - 1

pit = ru.where(dentro).mean(axis=1).iloc[1:]
hoy = miembros.loc[miembros["baja"].isna(), "ticker"]
superv = ru[hoy].mean(axis=1).iloc[1:]

for nombre, x in (("point-in-time", pit), ("today's members", superv)):
    print(f"{nombre:>15}: {anual(x):.2%} annual, Sharpe {sharpe(x):.2f}")
fuera = set(miembros["ticker"]) - set(hoy)
quiebras = salidas.loc[salidas["evento"] == "deslistado_quiebra",
                       "ticker"]
print(len(fuera), "companies not on today's list; bankruptcies:",
      sorted(fuera & set(quiebras)))

  point-in-time: 5.75% annual, Sharpe 0.38
today's members: 6.95% annual, Sharpe 0.44
15 companies not on today's list; bankruptcies: ['QXKV', 'QXQT', 'QXYX']

That is 1.2 points of return a year, which compound over twenty years: 1 USD from 2006 ends at 3.83 USD with today's members and at 3.05 USD with the point-in-time index. The difference comes from the 15 companies that passed through the index and are not on today's list: three of the four bankruptcies, with their −100% (the fourth, QXZT, never made it into the index), the ones that left the index after collapsing and the acquired ones. And the bias does not only affect an index portfolio: any strategy that buys losers (reversion, value) looks better with survivors, because the losers that kept falling until they disappeared are not in the sample. Momentum, conversely, can look worse than it is for the same reason.

Selection and data snooping: the best of a thousand

Selection bias appears when choosing after the fact what to show: the best variant of a rule, the market where it worked, the period in which it turned out well. Data snooping is its iterative version: reusing the same data to test idea after idea until one works. Both have the same arithmetic. If N candidates with no edge are tested, each with a measured Sharpe that is noise, the best of them is not “average” noise: it is the maximum of N noises.

ruido2_estrategias.parquet holds the daily returns of 1,000 fictitious strategies over the sessions from 2022 to 2025, in 50 families of 20 correlated variants. All have zero edge except five, whose true Sharpe is in ruido2_verdad.csv. The test: choose with 2022-2023 and look at 2024-2025.

est = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")

def sharpe_cols(df):
    return df.mean() / df.std() * np.sqrt(252)

sh_is = sharpe_cols(est.loc[:"2023"])
sh_oos = sharpe_cols(est.loc["2024":])
top = sh_is.nlargest(10).index
tabla = pd.DataFrame({"IS 2022-23": sh_is[top],
                      "OOS 2024-25": sh_oos[top],
                      "real": verdad.loc[top, "sharpe_real"]})
print(tabla.round(2))
print(f"Top 10 mean: IS {sh_is[top].mean():.2f}, "
      f"OOS {sh_oos[top].mean():.2f}")

            IS 2022-23  OOS 2024-25  real
estrategia
e0133             2.98        -0.04   0.8
e0542             2.91        -0.02   0.0
e0462             2.40         0.51   0.0
e0781             2.36        -1.47   0.0
e0518             2.28         0.85   1.5
e0565             2.08        -0.67   0.0
e0060             1.98        -0.26   0.0
e0124             1.91         0.92   0.0
e0950             1.89        -0.57   0.0
e0835             1.88         0.09   0.0
Top 10 mean: IS 2.27, OOS -0.07

The top 10 of the first half has a mean Sharpe above 2, and in the second its mean is negative. Two of the ten have a real edge (e0133 and e0518, with true Sharpe 0.8 and 1.5); the other eight are noise that had two good years. The sample of each half, 501 sessions, gives a standard error of the annual Sharpe of $\sqrt{252/501} \approx 0.71$: with a thousand candidates, the maximum of the noise is around three standard errors above zero. Extreme value theory gives an approximation for the expected maximum of N independent Sharpes with no edge, with $\gamma \approx 0.5772$ (the Euler-Mascheroni constant) and $\Phi^{-1}$ the inverse of the normal:

from scipy.stats import norm

def max_esperado(n, error_tipico):
    g = np.euler_gamma
    z = ((1 - g) * norm.ppf(1 - 1 / n)
         + g * norm.ppf(1 - 1 / (n * np.e)))
    return z * error_tipico

et = np.sqrt(252 / 501)
for n in (1, 10, 100, 1000):
    m = max_esperado(n, et) if n > 1 else 0.0
    print(f"N = {n:>4}: expected maximum {m:.2f}")
ruido = verdad.index[verdad["sharpe_real"] == 0]
print(f"Noise maximum in 2022-23: {sh_is[ruido].max():.2f}")

N =    1: expected maximum 0.00
N =   10: expected maximum 1.12
N =  100: expected maximum 1.79
N = 1000: expected maximum 2.31
Noise maximum in 2022-23: 2.91

With a single candidate, the expected Sharpe of a strategy with no edge is 0; with a thousand, around 2.3. The observed maximum of the noise, 2.91, is even higher: it is a single realization, and the maximum of a thousand noises has a lot of dispersion. The formula assumes independent candidates; here they are not (20 variants per family, correlation 0.5), which reduces the effective number of attempts and the expected maximum. Lesson 2.5.4 turns this calculation into a test, the Deflated Sharpe Ratio, and lesson 2.1.8 covers multiple testing corrections.

Snooping is more treacherous, because it looks prudent. A researcher who already knows that the best of a thousand is misleading decides to require the strategy to be good in both halves:

for umbral in (0.5, 1.0):
    pasan = sh_is.index[(sh_is > umbral) & (sh_oos > umbral)]
    reales = int((verdad.loc[pasan, "sharpe_real"] > 0).sum())
    print(f"Sharpe > {umbral} in both halves: {len(pasan)} "
          f"strategies, {reales} with a real edge")

Sharpe > 0.5 in both halves: 56 strategies, 3 with a real edge
Sharpe > 1.0 in both halves: 7 strategies, 0 with a real edge

With a threshold of 0.5, 56 strategies pass, and only three have an edge. With a threshold of 1, seven pass, and none do: the five that do have one are left out in one of the two halves, because with 501 sessions even a true Sharpe of 1.5 can measure below 1. The second half stopped being a validation sample the moment it was used to choose. An out-of-sample set is only independent if it is looked at once, at the end, with the strategy already decided. Every time it is looked at and something is corrected, it becomes part of the training. Lessons 2.5.2 and 2.5.3 organize that separation (walk-forward and purged cross-validation) and lesson 2.2.4 explains how to log each attempt so that N can be counted. White (2000) proposed one of the first formal tests of data snooping, the Reality Check, which compares the best rule with the distribution of the maximum of all those tested.

Narrative bias

The last bias is not in the code. It is the tendency to build, after seeing a result, an explanation that makes it look inevitable, and to take it as proof. Once the backtest of a variant turns out well, there is always a story: “funds rebalance at month-end”, “retail investors overreact to news”, “carry pays for crisis risk”. The story may be true. The problem is the order: it was chosen because it fit the data, and with 1,000 different results there would have been 1,000 different stories, all convincing.

The symptoms are recognizable: the explanation comes after the result; nobody wrote down beforehand what result would refute it; the chosen parameters have a tailor-made economic justification (“63 sessions because it is a quarter”); the bad periods are explained as exceptions (“2022 was an odd year”) and the good ones as the rule; and the story predicts nothing that has not already been looked at. In the academic literature it has a name, HARKing (hypothesizing after the results are known), and it is one of the reasons why Harvey, Liu and Zhu (2016) call for more demanding thresholds for new factors.

The defense is procedural. This list is filled in before looking at the test data, and saved with a date in the research journal (lesson 2.2.4):

A useful test: also write the opposite story. If for the opposite result (“the rule loses because…”) an equally convincing explanation is found just as easily, the explanation is not providing information.

How much each bias inflates

The header figure puts the four rooms with code on the same scale, the annual Sharpe: same-bar execution takes momentum from 1.00 to 3.16; full-sample normalization turns a losing rule (−0.65) into a winning one (0.67); survivors raise the index Sharpe from 0.38 to 0.44; and selecting the best of a thousand gives a top 10 of 2.3 that is worth −0.07 out of sample. The magnitudes come from this synthetic data, but the order is typical: look-ahead produces the most absurd results (and that is why it is the easiest to detect), while survivorship and selection give plausible results, which are the most dangerous.

Lesson 2.5.8 gathers these tests into an “honest backtest” checklist before moving on to paper trading.

Exercises

1. Apply the lag test to the “price vs. 50-session mean” rule with lags from 1 to 5 sessions. Does it behave like an honest signal?

cruce50 = np.sign(precio / precio.rolling(50).mean() - 1)
pd.Series({k: sharpe(cartera(cruce50, k).loc["2001":])
           for k in range(0, 6)}, name="Sharpe").round(2)

0    10.95
1     0.55
2     0.50
3     0.54
4     0.58
5     0.53
Name: Sharpe, dtype: float64

2. Repeat the point-in-time index without the delisting returns (that is, without the −100% of the bankruptcies). What part of the survivorship bias does that detail explain?

ru_sin = aj.pct_change(fill_method=None)
dentro_sin = dentro.copy()
for e in salidas.itertuples():
    i = ru.index.get_loc(aj[e.ticker].last_valid_index()) + 1
    if i < len(ru):
        dentro_sin.iloc[i, ru.columns.get_loc(e.ticker)] = False
pit_sin = ru_sin.where(dentro_sin).mean(axis=1).iloc[1:]
print(f"Point-in-time with delistings:    {anual(pit):.2%}")
print(f"Point-in-time without delistings: {anual(pit_sin):.2%}")
print(f"Today's members:                  {anual(superv):.2%}")

Point-in-time with delistings:    5.75%
Point-in-time without delistings: 5.96%
Today's members:                  6.95%

3. A colleague shows you a weekly reversal strategy on stocks with a Sharpe of 1.8 between 2010 and 2020 and explains that it works “because retail investors overreact to weekend news”. Write three questions from the checklist that you would ask before looking at the code and which answer would make you suspicious.