--- title: "2.5.8 Honest backtest checklist" description: "Checklist and executable auditor before paper trading: execution, costs, out of sample, DSR, PBO, plateau, capacity and survivorship." url: "https://pythonparatrading.com/en/python-trading-course/level-2/honest-backtest-checklist/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/honest-backtest-checklist.png" --- # 2.5.8 · “Honest backtest” checklist before moving to paper trading > A backtest is not approved by its Sharpe, but by the tests it has passed. This lesson gathers the module's tests into a list, codes an auditor that runs eight of them on a result object and applies it to two strategies: one with a planted effect and an optimized placebo that, looked at one test at a time, seems almost as good. - Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/) - Module: 2.5 · Biases and validation - Duration: 70 min - Prerequisites: [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/), [2.5.6 · Parameter sensitivity: plateaus versus peaks, robustness across assets and periods](https://pythonparatrading.com/en/python-trading-course/level-2/parameter-sensitivity-and-robustness/), [2.3.5 · Factors (value, quality, low volatility, size) and event strategies](https://pythonparatrading.com/en/python-trading-course/level-2/factor-investing-and-event-strategies/) ## By the end of this lesson - Use an honest backtest checklist and know which lesson of the course covers each item. - Represent a backtest as an object with everything needed to audit it: returns, weights, the grid of all the tests, the in-sample and out-of-sample split and the universe. - Code eight automatic checks: next-bar execution, costs, out of sample, Deflated Sharpe, PBO, plateau, capacity and survivorship. - Interpret a verdict: why a false positive passes several tests and which test gives it away, and what is left for manual review and paper trading. ## A gate before the money The lessons of this module have presented the errors of a backtest one by one: look-ahead and survivorship (2.5.1), in-sample selection (2.5.2 and 2.5.3), number of tests (2.5.4), the chance of ordering (2.5.5), parameter peaks (2.5.6) and scenarios the sample does not have (2.5.7). This lesson turns them into a **gate**: a list that every backtest must pass before a strategy touches money, first simulated and then real. Harvey and Liu (2015)[^1] put it in a similar way: a backtest result has to be adjusted for everything that was tried before arriving at it. ```mermaid flowchart TB B[Candidate backtest] --> A{Auditor} A -- fails --> I[Research more] A -- passes --> M{Manual review} M -- fails --> I M -- passes --> P[Paper trading] P --> C{Does it match?} C -- no --> I C -- yes --> R[Live, small] ``` The gate has two parts. One can be automated: they are tests computed with the data and the code of the backtest, and they are best run the same way every time, without deciding case by case. The other requires judgment: the hypothesis, the quality of the data, the code review. Both are on the list. ## The list | # | Check | Question | How | Guideline threshold | Auditor | Lesson | |---|---|---|---|---|---|---| | 1 | Prior hypothesis | Where does the edge come from and who loses on the other side? | Written before seeing results | One verifiable sentence | Manual | 2.2.1 | | 2 | Experiment journal | How many things were really tried? | Record of every test, including the discarded ones | Total number of tests | Manual | 2.2.4 | | 3 | Point-in-time universe | Does the universe include the companies that disappeared? | Delisted stocks and their final return | None left out | Yes | 1.4.4 and 2.5.1 | | 4 | Next-bar execution | Does the signal use only past data? | Data truncation and one more period of lag | Truncation identical; Sharpe with lag ≥ 50% | Yes | 2.5.1 | | 5 | Costs | Does it survive costs worse than planned? | Costs ×2 | Sharpe ≥ 0.5 | Yes | 2.4.7 | | 6 | Fills | Are limit orders, stops and gaps realistic? | Fill model | No impossible fills | Manual | 2.4.8 | | 7 | Out of sample | Does what was chosen hold on data not used to choose it? | Split or walk-forward | Out ≥ 50% of in | Yes | 2.5.2 | | 8 | Number of tests | Does it beat the maximum expected by chance? | Deflated Sharpe | DSR ≥ 0.95 | Yes | 2.5.4 | | 9 | Selection | Does choosing the best help, or does it do harm? | PBO (CSCV) and probability of loss | Loss ≤ 10% | Yes | 2.5.3 and 2.5.4 | | 10 | Plateau | Do neighboring parameters give something similar? | Neighbors of the chosen one | ≥ 70% of the chosen one | Yes | 2.5.6 | | 11 | Robustness | Does it work on other assets and periods? | By market, sector and segment | Majority positive | Manual | 2.5.6 | | 12 | Capacity | Does the planned capital fit? | Orders against daily volume | p95 ≤ 10% of volume | Yes | 2.2.3 | | 13 | Risk | What drawdown and what size? | Block bootstrap, fan chart | Acceptable risk of ruin | Manual | 2.5.5 | | 14 | Stress | Where does it lose? | Synthetic scenarios | Tolerable loss | Manual | 2.5.7 | | 15 | Reproducibility | Can someone else get the same figures? | Versioned code, fingerprinted data, seeds | Same result | Manual | 1.4.8 | The thresholds are guidelines: each team sets its own, **before** seeing the results and for all strategies alike. A threshold adjusted afterward so that a strategy passes is another form of overfitting. ## Two backtests to audit Both strategies are monthly long-short on the 120 fictitious stocks of level 2: they buy the top bucket of a score and sell the bottom one, equally weighted, with the weights set at each month's close and the return of the following month. First, the monthly returns. The detail that matters is the month in which a company stops trading: you have to include its last price **and** the delisting return (−100% in a bankruptcy), or the backtest ignores the worst loss of each bankruptcy (lesson 1.4.4): ```python import itertools import numpy as np import pandas as pd from dataclasses import dataclass, field from scipy import stats aj = pd.read_parquet("universo2_ajustados.parquet") px = aj.pivot(index="fecha", columns="ticker", values="cierre_ajustado") fin_mes = px.index.to_series().groupby(px.index.to_period("M")).max() mensual = px.loc[fin_mes.to_numpy()] R = mensual.pct_change(fill_method=None) # the month of a delisting: up to the last price and then the payout eventos = pd.read_csv("universo2_eventos.csv", parse_dates=["fecha"]) deslist = eventos.dropna(subset=["rentabilidad_deslistado"]) for _, ev in deslist.iterrows(): tk = ev["ticker"] ultima = px[tk].last_valid_index() mes = R.index[R.index >= ultima][0] antes = R.index[R.index < ultima][-1] R.loc[mes, tk] = (px.loc[ultima, tk] / mensual.loc[antes, tk] * (1 + ev["rentabilidad_deslistado"]) - 1) print(R.shape, f"{len(deslist)} delisted stocks with their final return") R.loc["2008-10", ["QXQT"]] ``` ```output (240, 120) 10 delisted stocks with their final return ticker QXQT fecha 2008-10-31 -1.0 ``` The characteristics are those of `universo2_caracteristicas.parquet`, which the README defines as **point-in-time**: what was known at each month's close. Value is the log of book equity over market capitalization (only with positive equity): ```python car = pd.read_parquet("universo2_caracteristicas.parquet").set_index( ["fecha", "ticker"]) def panel(col): return car[col].unstack().reindex(R.index) pn, cap = panel("patrimonio_neto"), panel("capitalizacion") datos = { "R": R, "value": np.log(pn.where(pn > 0) / cap), "esg": panel("puntuacion_esg").astype(float), "horquilla": panel("horquilla_pb"), "adv_usd": panel("volumen_medio_21d") * panel("cierre"), "cotiza": panel("cierre").notna(), } datos["value"].iloc[-1].describe().round(2) ``` ```output count 110.00 mean -0.54 std 0.68 min -2.22 25% -0.99 50% -0.60 75% 0.00 max 0.81 Name: 2025-12-31 00:00:00, dtype: float64 ``` The engine: rank-based normal scores, momentum that skips the last month, quantile long-short and a cost of half the spread per unit of turnover, with the `horquilla_pb` of each stock and month (lesson 2.4.7). The `retardo` parameter allows executing one month later, and the auditor will use it. ```python def z_rango(df): """Rank-based normal score on each date.""" rk = df.rank(axis=1) n = df.notna().sum(axis=1) u = (rk - 0.5).div(n, axis=0).to_numpy() return pd.DataFrame(stats.norm.ppf(u), index=df.index, columns=df.columns) def momentum(R, L): """Return of months m-L+1 to m-1 (skips the last one).""" with np.errstate(divide="ignore"): # log1p(-1) = -inf: bankruptcy lr = np.log1p(R) return np.expm1(lr.shift(1).rolling(L - 1).sum()) def largo_corto(puntos, puede, q): """Long the top q and short the bottom q, equally weighted.""" rk = puntos.where(puede).rank(axis=1, pct=True) lar, cor = (rk > 1 - q).astype(float), (rk <= q).astype(float) return (lar.div(lar.sum(axis=1), axis=0) - cor.div(cor.sum(axis=1), axis=0)).fillna(0) INICIO = "2007-08" # the longest window (18 months) already exists def backtest(d, puntos, q, universo, coste=1.0, retardo=1): """Weights at the close of month t; return of month t + retardo.""" puede = d["cotiza"] & d["cotiza"].columns.isin(list(universo)) w = largo_corto(puntos, puede, q) bruto = (w.shift(retardo) * d["R"]).sum(axis=1) giro = (w - w.shift(1)).abs() pb = d["horquilla"].fillna(20) / 2 / 1e4 # half spread gasto = (giro * pb).sum(axis=1).shift(retardo) * coste return (bruto - gasto).loc[INICIO:], w.loc[INICIO:] def sharpe(x): x = np.asarray(x, dtype=float) return x.mean() / x.std(ddof=1) * np.sqrt(12) ``` **Strategy A** combines momentum and value. The researcher tries six momentum windows and five quantiles, 30 combinations, and chooses the best in the first half of the data (from 2007-08 to 2016-07): ```python def senal_vm(d, L): return z_rango(momentum(d["R"], L)) + z_rango(d["value"]) TODOS = set(R.columns) # point-in-time universe rejilla_a = pd.DataFrame({ (L, q): backtest(datos, senal_vm(datos, L), q, TODOS)[0] for L in (3, 6, 9, 12, 15, 18) for q in (0.1, 0.2, 0.3, 0.4, 0.5)}) CORTE = "2016-07" # in sample: 2007-08 to 2016-07 sr_in = rejilla_a.loc[:CORTE].apply(sharpe) elegido_a = sr_in.idxmax() print(f"Chosen in sample: L = {elegido_a[0]} months, " f"q = {elegido_a[1]} (Sharpe {sr_in.max():.2f})") sr_in.unstack().round(2) ``` ```output Chosen in sample: L = 6 months, q = 0.4 (Sharpe 0.97) 0.1 0.2 0.3 0.4 0.5 3 0.42 0.69 0.90 0.74 0.85 6 0.69 0.86 0.82 0.97 0.90 9 0.58 0.63 0.50 0.55 0.65 12 0.86 0.72 0.74 0.80 0.87 15 0.47 0.67 0.61 0.64 0.65 18 0.36 0.61 0.73 0.73 0.54 ``` It chooses 6 months and 40%, with an in-sample Sharpe of 0.97, on a grid in which all 30 combinations are positive: a plateau. **Strategy B** is what comes out of a less careful process. The researcher tries the smoothed ESG score with six windows, with both signs (buying the best-scored or the worst-scored) and five quantiles, 60 combinations, and does it on the companies that **trade today**, because that is the list at hand: ```python def senal_esg(d, k, signo): return signo * d["esg"].rolling(k).mean() sect = pd.read_csv("universo2_sectores.csv") SUPERVIVIENTES = set(sect.loc[sect["ultima_sesion"].isna(), "ticker"]) rejilla_b = pd.DataFrame({ (k, s, q): backtest(datos, senal_esg(datos, k, s), q, SUPERVIVIENTES)[0] for k in (1, 2, 3, 6, 9, 12) for s in (1, -1) for q in (0.1, 0.2, 0.3, 0.4, 0.5)}) sr_in_b = rejilla_b.loc[:CORTE].apply(sharpe) elegido_b = sr_in_b.idxmax() print(f"{len(SUPERVIVIENTES)} survivors out of {len(TODOS)}") print(f"Chosen: {elegido_b[0]}-month average, sign {elegido_b[1]}, " f"q = {elegido_b[2]} (Sharpe {sr_in_b.max():.2f})") sr_in_b.unstack([1, 2]).round(2).iloc[:, :5] ``` ```output 110 survivors out of 120 Chosen: 9-month average, sign 1, q = 0.2 (Sharpe 0.53) 1 0.1 0.2 0.3 0.4 0.5 1 0.27 0.32 0.17 0.16 0.17 2 0.35 0.38 0.11 0.16 0.08 3 0.35 0.50 0.20 0.10 0.21 6 0.22 0.43 0.17 0.01 0.22 9 0.41 0.53 0.24 0.16 0.09 12 0.30 0.26 0.15 0.11 -0.00 ``` An in-sample Sharpe of 0.53, with a sign and a window chosen after looking at the results. Told this way, B looks like a weaker strategy than A, but not an absurd one. The auditor has to say whether it is. ## The object being audited Auditing a backtest takes more than its equity curve. The `ResultadoBacktest` object stores everything: the net returns, the weights, a function to run it again with other costs or lags, the signal function (for truncation), the **full grid** of everything that was tried (for the DSR, the PBO and the plateau), the chosen combination, the in-sample and out-of-sample split, the number of tests, the universe and the planned capital. If the researcher cannot fill in a field (does not know how many tests were run, did not keep the grid), that is already an audit finding. ```python @dataclass class ResultadoBacktest: nombre: str rend: pd.Series # monthly, net of costs pesos: pd.DataFrame # at each month's close ejecutar: object # (datos, coste, retardo) -> rend senal: object # datos -> scores rejilla: pd.DataFrame # ALL the combinations tried elegido: tuple corte: str # last in-sample month n_pruebas: int # including those not written down universo: set fijos: tuple = () # grid axes that have no order capital: float = 50e6 # planned capital (USD) def resultado(nombre, senal, q, universo, rejilla, elegido, n, fijos=()): rend, w = backtest(datos, senal(datos), q, universo) return ResultadoBacktest( nombre, rend, w, lambda d, c=1.0, t=1: backtest(d, senal(d), q, universo, c, t)[0], senal, rejilla, elegido, CORTE, n, universo, fijos) L, q = elegido_a A = resultado("value + momentum", lambda d: senal_vm(d, L), q, TODOS, rejilla_a, elegido_a, rejilla_a.shape[1]) k, s, qb = elegido_b B = resultado("placebo ESG", lambda d: senal_esg(d, k, s), qb, SUPERVIVIENTES, rejilla_b, elegido_b, rejilla_b.shape[1], fijos=(1,)) # the sign has no neighbors for res in (A, B): print(f"{res.nombre}: {len(res.rend)} months, Sharpe " f"{sharpe(res.rend):.2f}, {res.n_pruebas} tests") ``` ```output value + momentum: 221 months, Sharpe 1.12, 30 tests placebo ESG: 221 months, Sharpe 0.54, 60 tests ``` `fijos=(1,)` tells the auditor that the second axis of B's grid (the sign) has no order: buying the best-scored is not a “neighbor” of buying the worst-scored. ## The eight automatic tests Each test returns a value, its threshold and whether it passes. The first three run the backtest again. **Next-bar execution.** Two checks. **Truncation**: all data after a date are deleted and the signal is recomputed; if the score for that date changes, the signal uses the future. And **lag**: with one more month of delay in execution, an edge with a monthly horizon should lose something, not disappear. **Costs**: the backtest with double the costs. **Out of sample**: the Sharpe of the chosen combination after the split against the one before. ```python def prueba(valor, umbral, pasa, texto): return {"valor": valor, "umbral": umbral, "pasa": bool(pasa), "detalle": texto} def p_ejecucion(res, fechas=("2010-06", "2015-12", "2020-03")): """Truncation: the signal for d does not change if what comes after d is deleted. Lag: with one more month of delay, the edge does not vanish.""" completa = res.senal(datos) iguales = all( np.allclose(res.senal({c: v.loc[:f] for c, v in datos.items()}) .loc[f].to_numpy(), completa.loc[f].to_numpy(), equal_nan=True) for f in fechas) s1, s2 = sharpe(res.rend), sharpe(res.ejecutar(datos, 1.0, 2)) return prueba(s2 / s1, ">= 0.5 and truncation identical", iguales and s2 >= 0.5 * s1, f"truncation {'identical' if iguales else 'DIFFERENT'}; " f"Sharpe t+1 {s1:.2f}, t+2 {s2:.2f}") def p_costes(res): s = sharpe(res.ejecutar(datos, 2.0, 1)) return prueba(s, ">= 0.5", s >= 0.5, f"Sharpe with costs x2 {s:.2f}") def p_fuera(res): r = res.rejilla[res.elegido] s_in, s_out = sharpe(r.loc[:res.corte]), sharpe(r.loc[res.corte:] .iloc[1:]) return prueba(s_out / s_in, ">= 0.5", s_out >= 0.5 * s_in > 0, f"in sample {s_in:.2f}, out of sample {s_out:.2f}") for res in (A, B): for f in (p_ejecucion, p_costes, p_fuera): print(f"{res.nombre:>16} | {f.__name__:<11} | " f"{f(res)['detalle']}") ``` ```output value + momentum | p_ejecucion | truncation identical; Sharpe t+1 1.12, t+2 0.94 value + momentum | p_costes | Sharpe with costs x2 1.08 value + momentum | p_fuera | in sample 0.97, out of sample 1.27 placebo ESG | p_ejecucion | truncation identical; Sharpe t+1 0.54, t+2 0.36 placebo ESG | p_costes | Sharpe with costs x2 0.53 placebo ESG | p_fuera | in sample 0.53, out of sample 0.56 ``` Both strategies pass all three. A loses 0.18 of Sharpe with a one-month delay (from 1.12 to 0.94), almost nothing with double the costs (1.08) and improves out of sample (from 0.97 to 1.27). B also passes: 0.36 with the delay, 0.53 with double the costs and 0.56 out of sample against 0.53 in sample. This deserves a pause. The placebo has no edge by construction, and even so its out-of-sample Sharpe is as good as its in-sample one. The reason: in these data, by chance, the ESG score is related to returns throughout the whole sample, and a stable chance relationship does not break when the sample is split in two. The out-of-sample check detects overfitting to an era; it does not detect a spurious effect that lasts the whole available history or the selection among many tests. **Deflated Sharpe** (lesson 2.5.4): the probability that the true Sharpe exceeds the maximum that could be expected by chance among $N$ tests, $$ SR_0 = \sqrt{V[\widehat{SR}_n]}\left[(1-\gamma)\,\Phi^{-1}\!\left(1-\tfrac{1}{N}\right) + \gamma\,\Phi^{-1}\!\left(1-\tfrac{1}{N e}\right)\right] $$ with $V[\widehat{SR}_n]$ the variance of the Sharpes of all the tests and $\gamma$ the Euler-Mascheroni constant. **PBO** by combinatorially symmetric cross-validation (CSCV): the sample is split into 10 blocks, the 252 ways of choosing 5 as “in sample” are taken, and in each one the out-of-sample rank of the best in-sample combination is checked. The auditor also computes the **probability of loss**: in what fraction of the partitions the chosen one loses out of sample. ```python EULER = np.euler_gamma # Euler-Mascheroni constant def p_dsr(res): """Deflated Sharpe Ratio (Bailey and López de Prado, 2014).""" r = res.rend.to_numpy() T, sr = len(r), r.mean() / r.std(ddof=1) # per month g3, g4 = stats.skew(r), stats.kurtosis(r, fisher=False) pruebas = res.rejilla.apply(lambda c: c.mean() / c.std(ddof=1)) N = res.n_pruebas sr0 = np.sqrt(pruebas.var(ddof=1)) * ( (1 - EULER) * stats.norm.ppf(1 - 1 / N) + EULER * stats.norm.ppf(1 - 1 / (N * np.e))) z = (sr - sr0) * np.sqrt(T - 1) / np.sqrt( 1 - g3 * sr + (g4 - 1) / 4 * sr ** 2) dsr = stats.norm.cdf(z) return prueba(dsr, ">= 0.95", dsr >= 0.95, f"Sharpe {sr * np.sqrt(12):.2f} against a maximum " f"expected by chance of {sr0 * np.sqrt(12):.2f}") def p_pbo(res, S=10): """CSCV: half of the blocks in sample, the other half out of sample.""" M = res.rejilla.to_numpy() T, N = M.shape bloques = np.array_split(np.arange(T), S) logits, pierde = [], [] for dentro in itertools.combinations(range(S), S // 2): i = np.concatenate([bloques[j] for j in dentro]) o = np.concatenate([bloques[j] for j in range(S) if j not in dentro]) sr_i = M[i].mean(0) / M[i].std(0, ddof=1) sr_o = M[o].mean(0) / M[o].std(0, ddof=1) n = sr_i.argmax() w = stats.rankdata(sr_o)[n] / (N + 1) logits.append(np.log(w / (1 - w))) pierde.append(sr_o[n] <= 0) pbo, p_perd = np.mean(np.array(logits) <= 0), np.mean(pierde) return prueba(p_perd, "loss <= 10%", p_perd <= 0.10, f"PBO {pbo:.2f}; the chosen one loses out of sample in " f"{p_perd:.0%} of the {len(logits)} partitions") for res in (A, B): for f in (p_dsr, p_pbo): print(f"{res.nombre:>16} | {f.__name__:<5} | " f"{f(res)['detalle']}") ``` ```output value + momentum | p_dsr | Sharpe 1.12 against a maximum expected by chance of 0.30 value + momentum | p_pbo | PBO 0.55; the chosen one loses out of sample in 0% of the 252 partitions placebo ESG | p_dsr | Sharpe 0.54 against a maximum expected by chance of 0.85 placebo ESG | p_pbo | PBO 0.14; the chosen one loses out of sample in 17% of the 252 partitions ``` *Fig. — Deflated Sharpe: measured Sharpe of each strategy (dot) against the maximum expected by chance given its number of trials (tick).* Two rows. Value + momentum: measured Sharpe 1.12 against a chance maximum of 0.30 with 30 trials. ESG placebo, highlighted: measured Sharpe 0.54 against 0.85 with 60 trials, which include both signs of the signal; its measured value falls below what chance would give. Here they part ways. A's Sharpe, 1.12, is well above the 0.30 that 30 tests with no edge would give with the dispersion of its grid: DSR of 1.00. B's, 0.54, is **below** the 0.85 that can be expected as the maximum of 60 tests with no edge, because its grid has a lot of dispersion (it includes both signs): DSR of 0.09. With all the tests counted, the placebo cannot be distinguished from chance. PBO gives a result that is confusing if read on its own: 0.55 for A and 0.14 for B. In A's grid all the combinations are good and alike, so which one comes first out of sample is a lottery and the PBO is around 0.5; but the chosen one **does not lose** out of sample in any of the 252 partitions. In B, the chance relationship is stable and the best in sample tends to be good out of sample (low PBO), but it loses in 17% of the partitions. That is why the auditor decides with the probability of loss and shows the PBO as context, as lesson 2.5.4 recommends. **Plateau** (lesson 2.5.6): the mean of the neighbors of the chosen combination in the in-sample grid, against the chosen one. **Capacity**: the capital at which the 95th percentile of the rebalancing orders reaches 10% of the stock's average daily volume, against the planned capital (USD 50 M). **Survivorship**: whether the universe leaves out any stock that stopped trading. ```python def p_meseta(res): """Neighbors of the chosen one in the in-sample grid.""" sr_in = res.rejilla.loc[:res.corte].apply(sharpe) ejes = [sorted({k[i] for k in sr_in.index}) for i in range(len(res.elegido))] pos = {k: [ejes[i].index(v) for i, v in enumerate(k)] for k in sr_in.index} yo = pos[res.elegido] vec = [sr_in[k] for k, p in pos.items() if k != res.elegido and max(abs(a - b) for a, b in zip(p, yo)) <= 1 and all(k[i] == res.elegido[i] for i in res.fijos)] ratio = np.mean(vec) / sr_in[res.elegido] return prueba(ratio, ">= 0.7", ratio >= 0.7, f"{len(vec)} neighbors: mean {np.mean(vec):.2f} " f"against {sr_in[res.elegido]:.2f}") def p_capacidad(res, limite=0.10): """Capital at which the 95th percentile of the orders is 10% of the stock's average daily volume.""" giro = (res.pesos - res.pesos.shift(1)).abs() part = (giro / datos["adv_usd"].loc[INICIO:]).stack() p95 = part[part > 0].quantile(0.95) cap = limite / p95 return prueba(cap / 1e6, f">= {res.capital / 1e6:.0f} M USD", cap >= res.capital, f"capacity {cap / 1e6:.0f} M USD") def p_supervivencia(res): """Does the universe include the stocks that stopped trading?""" fuera = set(deslist["ticker"]) - res.universo return prueba(len(fuera), "0 excluded", not fuera, f"{len(fuera)} delisted stocks outside the universe") for res in (A, B): for f in (p_meseta, p_capacidad, p_supervivencia): print(f"{res.nombre:>16} | {f.__name__:<15} | " f"{f(res)['detalle']}") ``` ```output value + momentum | p_meseta | 8 neighbors: mean 0.74 against 0.97 value + momentum | p_capacidad | capacity 54 M USD value + momentum | p_supervivencia | 0 delisted stocks outside the universe placebo ESG | p_meseta | 8 neighbors: mean 0.27 against 0.53 placebo ESG | p_capacidad | capacity 56 M USD placebo ESG | p_supervivencia | 10 delisted stocks outside the universe ``` A passes all three: its neighbors keep 76% of the chosen one's Sharpe (0.74 against 0.97), a capital of USD 54 M fits and its universe includes the 10 delisted stocks. B fails the plateau (0.27 against 0.53: the chosen window and quantile stand out from their neighbors) and survivorship (the 10 delisted stocks are left out), and passes capacity (USD 56 M). ## The verdict The full auditor runs the eight tests and only approves a strategy that passes all of them: ```python PRUEBAS = {"ejecución t+1": p_ejecucion, "costes x2": p_costes, "fuera de muestra": p_fuera, "Deflated Sharpe": p_dsr, "PBO y pérdida": p_pbo, "meseta": p_meseta, "capacidad": p_capacidad, "supervivencia": p_supervivencia} def auditar(res): filas = {n: f(res) for n, f in PRUEBAS.items()} tabla = pd.DataFrame(filas).T[["valor", "umbral", "pasa"]] veredicto = "APPROVED" if tabla["pasa"].all() else "NOT APPROVED" return tabla, veredicto informe = {} for res in (A, B): tabla, veredicto = auditar(res) informe[res.nombre] = tabla print(f"{res.nombre}: {veredicto} " f"({tabla['pasa'].sum()} of {len(tabla)} tests)") pd.concat({n: t["pasa"].map({True: "pass", False: "FAIL"}) for n, t in informe.items()}, axis=1) ``` ```output value + momentum: APPROVED (8 of 8 tests) placebo ESG: NOT APPROVED (4 of 8 tests) value + momentum placebo ESG ejecución t+1 pass pass costes x2 pass pass fuera de muestra pass pass Deflated Sharpe pass FAIL PBO y pérdida pass FAIL meseta pass FAIL capacidad pass pass supervivencia pass FAIL ``` The header figure summarizes the result. A, the strategy with the planted effect, passes all eight. B, the placebo, passes four and fails four, and none of the ones it fails would have been enough on its own if it had been looked at in isolation: the DSR requires knowing how many tests were run, the plateau requires keeping the grid, survivorship requires knowing the universe. B's full report: ```python t = informe["placebo ESG"].copy() t["valor"] = t["valor"].astype(float).round(2) t ``` ```output valor umbral pasa ejecución t+1 0.66 >= 0.5 and truncation identical True costes x2 0.53 >= 0.5 True fuera de muestra 1.05 >= 0.5 True Deflated Sharpe 0.09 >= 0.95 False PBO y pérdida 0.17 loss <= 10% False meseta 0.51 >= 0.7 False capacidad 56.04 >= 50 M USD True supervivencia 10.00 0 excluded False ``` The survivorship test is about procedure: it fails even if the effect on the result were small, because a backtest on survivors cannot be reproduced in real time, when it is not known which companies are going to disappear. In this case, moreover, the survivor universe inflated the placebo: ```python con_todos, _ = backtest(datos, senal_esg(datos, k, s), qb, TODOS) print(f"Placebo on survivors: {sharpe(B.rend):.2f}; " f"with the full universe: {sharpe(con_todos):.2f}") ``` ```output Placebo on survivors: 0.54; with the full universe: 0.43 ``` *Fig. — Cumulative growth, in logs, of the two audited strategies, net of costs; the vertical line separates the selection period (to July 2016) from the test period.* Two curves from 2007 to 2025. Value + momentum reaches 0.67 at the 2016 split and 1.50 at the end. The ESG placebo, highlighted, reaches 0.42 at the split and keeps rising to 0.82: it also wins out of sample, although its edge is zero by construction. The figure shows the problem of judging by the curve: the placebo, in amber, also rises after the split. Seen alone, and with the survivor universe, it is a strategy “that works out of sample”. The tests that give it away cannot be seen in the curve. ## What the auditor does not see, and paper trading Eight automatic tests cover part of the list. The rest requires judgment, and it is best done in writing: - **Hypothesis** (lesson 2.2.1): an explanation of why the edge exists, written before looking at results. B did not have one: “companies with better ESG go up” does not say who loses on the other side or why the market has not corrected it. - **Data**: errors, gaps, adjustments for splits and dividends, time zones (module 1.4). - **Code**: review by another person, unit tests of the engine, comparison with an independent engine (lessons 2.4.1 and 2.4.2). - **Risk and stress** (lessons 2.5.5 and 2.5.7): expected drawdown, size, scenarios where it loses. And after that, **paper trading**: the strategy runs in real time, with live data and simulated orders, for a period fixed in advance. It is not for measuring the Sharpe (a few months are not enough, lesson 2.5.5) but for checking that what the backtest assumes actually happens: that the signal computed live matches the one the backtest would give on the same data, that real fills and costs resemble the modeled ones and that operations (data arriving late, rejected orders, holidays) do not break anything. The differences between paper and backtest are the last finding of the audit; if they are large, it is back to research. Only then comes real money, with a small size. ## Exercises 1. A colleague computes momentum with `np.log1p(R).shift(-1)` by mistake (a backward shift). Build their result with everything else the same as strategy A and run the execution test on it. **Solution.** ```python def momentum_malo(R, L): with np.errstate(divide="ignore"): lr = np.log1p(R) return np.expm1(lr.shift(-1).rolling(L - 1).sum()) def senal_mala(d): return z_rango(momentum_malo(d["R"], L)) + z_rango(d["value"]) M = resultado("with look-ahead", senal_mala, q, TODOS, rejilla_a, elegido_a, rejilla_a.shape[1]) print(f"Sharpe: {sharpe(M.rend):.2f}") print(p_ejecucion(M)["detalle"]) ``` ```output Sharpe: 8.79 truncation DIFFERENT; Sharpe t+1 8.79, t+2 1.24 ``` With the error, each month's momentum includes the following month's return, and the Sharpe rises to 8.79: a number that should already be enough of an alarm. The truncation test confirms it: when the data after each date are deleted, the score changes. The lag test flags it too, because with one more month of delay the Sharpe falls to 1.24 (from 8.79), well below 50%. A look-ahead error is almost never this obvious; that is why the truncation test is always run, also when the Sharpe looks reasonable. 2. Repeat strategy A on the survivors and audit it. Which test fails and how much does the Sharpe change? **Solution.** ```python rejilla_s = pd.DataFrame({ (Ls, qs): backtest(datos, senal_vm(datos, Ls), qs, SUPERVIVIENTES)[0] for Ls in (3, 6, 9, 12, 15, 18) for qs in (0.1, 0.2, 0.3, 0.4, 0.5)}) S = resultado("A on survivors", lambda d: senal_vm(d, L), q, SUPERVIVIENTES, rejilla_s, elegido_a, 30) tabla_s, veredicto_s = auditar(S) print(f"Sharpe {sharpe(S.rend):.2f} against {sharpe(A.rend):.2f}: " f"{veredicto_s}") print(tabla_s.loc[~tabla_s["pasa"].astype(bool)].index.tolist()) ``` ```output Sharpe 1.18 against 1.12: NOT APPROVED ['supervivencia'] ``` On the survivors the Sharpe barely changes (1.18 against 1.12) and only the survivorship test fails. The strategy still has its planted effect, but the procedure is no longer possible in real time, and the auditor does not approve it until it is redone with the point-in-time universe. That the difference is small here guarantees nothing: in the placebo, the same error raised the Sharpe from 0.43 to 0.54, and in a strategy that buys companies in distress it can flip the sign of the result. 3. What would the auditor need to add to cover robustness across assets and periods (item 11 of the list)? Describe the test, its threshold and what new fields the `ResultadoBacktest` object would need. **Solution.** It would need the returns per asset (or per group: sector, country, asset class) of the chosen combination, and a field with the time partition (for example, five-year segments). The test would compute the Sharpe in each group and in each segment and would pass if most are positive (for example, at least two thirds of the groups and of the segments) and if no group explains most of the result on its own. As in lesson 2.5.6, the threshold has to be set within the domain of the hypothesis: if the edge should only exist in one type of asset, the test is applied to the assets of that type. And, like all thresholds, it is set before seeing the results. [^1]: Harvey, C. R. and Liu, Y. (2015), “Backtesting”, The Journal of Portfolio Management 42(1), 13-28, [doi:10.3905/jpm.2015.42.1.013](https://doi.org/10.3905/jpm.2015.42.1.013). ## Self-assessment 1. The ESG placebo passes the out-of-sample test: Sharpe of 0.53 in sample and 0.56 out of sample. Why is that not enough? - A. Because a chance relationship that holds over the whole sample also holds outside it: the out-of-sample check detects overfitting to an era, not a stable spurious effect or the selection among 60 tests - B. Because the out-of-sample period is too short - C. Because the test is badly coded - D. It is enough: the placebo is a valid strategy *Answer A: Because a chance relationship that holds over the whole sample also holds outside it: the out-of-sample check detects overfitting to an era, not a stable spurious effect or the selection among 60 tests.* The ESG score is independent of everything by construction, but in these data its relationship with returns is positive by chance in both periods. It is given away by the Deflated Sharpe (0.09: its Sharpe of 0.54 is below the maximum expected by chance with 60 tests, 0.85), the plateau and the probability of an out-of-sample loss. 2. The value + momentum strategy has a PBO of 0.55 and the placebo, 0.14. How is that explained? - A. The placebo is less overfitted than the real strategy - B. There is an error in the rank calculation - C. In the value + momentum grid all the combinations are good and similar: which one comes first out of sample is chance and the PBO is around 0.5, without the choice doing any harm (the chosen one loses in no partition). That is why PBO is read together with the probability of loss - D. PBO only works with daily data *Answer C: In the value + momentum grid all the combinations are good and similar: which one comes first out of sample is chance and the PBO is around 0.5, without the choice doing any harm (the chosen one loses in no partition). That is why PBO is read together with the probability of loss.* PBO measures whether choosing the best in sample helps to end up above the median out of sample. On a plateau, all are equally good and choosing adds nothing: PBO ≈ 0.5. What matters then is whether the chosen one loses out of sample: 0% of the 252 partitions in the real strategy, 17% in the placebo. 3. What does the signal truncation test detect? - A. That costs are correctly computed - B. That the strategy has enough capacity - C. That the universe includes the delisted stocks - D. That the score of a date uses later data: if deleting everything that comes after the date changes the score, the signal looks into the future *Answer D: That the score of a date uses later data: if deleting everything that comes after the date changes the score, the signal looks into the future.* It is the direct look-ahead test. In exercise 1, a mistaken `shift(-1)` gives a Sharpe of 8.79; the truncation test flags it as different. A lag test (executing one period later) complements it: an edge that disappears with a one-month delay is suspicious. 4. The value + momentum strategy recomputed on the survivors has a Sharpe of 1.18 instead of 1.12, and the auditor rejects it. Why does it fail if the Sharpe is similar? - A. Because the Sharpe has gone up - B. Because its universe excludes the 10 stocks that stopped trading: the result cannot be reproduced in real time, when it is not known which companies are going to disappear. The test is about procedure, not about the result - C. Because the plateau changes - D. Because costs are higher *Answer B: Because its universe excludes the 10 stocks that stopped trading: the result cannot be reproduced in real time, when it is not known which companies are going to disappear. The test is about procedure, not about the result.* A backtest on survivors uses information from the future (who survives), even if in a specific case the effect on the Sharpe is small. In the placebo, the survivor universe raises the Sharpe from 0.43 to 0.54. The test does not ask how much the result changes, but whether the procedure is possible in real time. 5. The value + momentum strategy passes all eight tests. What comes next? - A. Trade it with all the planned capital - B. Optimize more parameters to raise the Sharpe - C. Reviewing what the auditor does not see (hypothesis, data, code, stress) and a paper trading period that compares signals, fills and costs with what the backtest says, before trading live with a small size - D. Nothing: it is proven to work *Answer C: Reviewing what the auditor does not see (hypothesis, data, code, stress) and a paper trading period that compares signals, fills and costs with what the backtest says, before trading live with a small size.* The auditor is a necessary gate, not a sufficient one. In the course data, moreover, the strategy wins because the effect is planted; on real data, passing the auditor only says that the backtest does not have the errors the auditor looks for. ## Key points - An honest backtest is shown with tests, not with a Sharpe: next-bar execution, costs, out of sample, number of tests (DSR), PBO and loss, plateau, capacity and a point-in-time universe. - The tests are coded on a result object that stores everything needed: returns, weights, the full grid of what was tried, the in-sample and out-of-sample split, the universe and the planned capital. - The strategy with a planted effect passes all eight; the optimized ESG placebo passes four (execution, costs, out of sample and capacity) and fails the Deflated Sharpe (0.09), the out-of-sample loss (17%), the plateau (0.51) and survivorship. - No single test is enough: each one detects a different error, and a false positive that is stable over time can pass the out-of-sample check and have a low PBO. - The auditor does not replace manual review (hypothesis, data, code, stress) or paper trading, which checks that signals, fills and costs in real time match the backtest. ## Resources ### Research - [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min ### Documentation - [Python: dataclasses](https://docs.python.org/3/library/dataclasses.html) · docs.python.org ### Books - [Bailey, D. H. and López de Prado, M. (2014): The Deflated Sharpe Ratio. The Journal of Portfolio Management 40(5), 94-107](https://doi.org/10.3905/jpm.2014.40.5.094) · doi.org - [Bailey, D. H., Borwein, J., López de Prado, M. and Zhu, Q. J. (2016): The probability of backtest overfitting. The Journal of Computational Finance](https://doi.org/10.21314/JCF.2016.322) · doi.org - [Harvey, C. R. and Liu, Y. (2015): Backtesting. The Journal of Portfolio Management 42(1), 13-28](https://doi.org/10.3905/jpm.2015.42.1.013) · doi.org ### Data - [Level 2 data: monthly point-in-time characteristics of the universe (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/universo2_caracteristicas.parquet) · Download · universo2_caracteristicas.parquet Previous lesson: [2.5.7 · Synthetic data for stress testing: GBM, GARCH and bootstrapping](https://pythonparatrading.com/en/python-trading-course/level-2/synthetic-data-for-stress-testing/) Next lesson: [2.6.1 · CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index](https://pythonparatrading.com/en/python-trading-course/level-2/return-and-risk-metrics/)