--- title: "2.5.5 Monte Carlo: trades, blocks and permutations" description: "Trade resampling, block bootstrap and a signal permutation test on a trend portfolio with a planted effect and on its control." url: "https://pythonparatrading.com/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests.png" --- # 2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations > A backtest produces a single equity curve, and that curve is a sample of size one. This lesson generates thousands of plausible alternatives in three different ways, each with its own question: what drawdown to expect, how far to trust the Sharpe, and whether the signal gets the timing right or only collects the exposure. - Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/) - Module: 2.5 · Biases and validation - Duration: 70 min - Prerequisites: [2.1.1 · Hypothesis tests, p-values, bootstrap and regression with robust standard errors](https://pythonparatrading.com/en/python-trading-course/level-2/hypothesis-testing-bootstrap-robust-errors/), [2.3.1 · Trend following and momentum](https://pythonparatrading.com/en/python-trading-course/level-2/trend-following-and-momentum/), [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/) ## By the end of this lesson - Resample and shuffle a strategy's trades to estimate the distribution of the drawdown and the risk of ruin by leverage. - Apply the stationary bootstrap to daily returns, choose the block length (Politis and White) and build an equity fan. - Code a signal permutation test that preserves the returns and the risk scaling, and compute its p-value. - Check each method where there is a planted effect and on controls with no effect, and recognize what none of them corrects: selection among many strategies. ## An equity curve is a sample of size one A backtest returns an equity curve, a Sharpe and a maximum drawdown. They are figures from **one** history: the one that happened. Had the market dealt the same good and bad days in a different order, or had the strategy closed its trades in a different sequence, the drawdown would be different. Lesson 2.1.1 introduced the bootstrap to estimate the uncertainty of a statistic; here it is applied to the practical question of a trading system: which part of the result is structure and which part is the luck of the order. “Monte Carlo” in this context groups three families of resampling. Each one starts from a different object of the backtest and answers a different question: ```mermaid flowchart TB B[Backtest] --> O[Trades] B --> R[Returns] B --> S[Signal and market] O --> T[Trade resampling] R --> K[Block bootstrap] S --> P[Permutation] T --> T2[Drawdown and ruin] K --> K2[CI and fan] P --> P2[Timing p-value] ``` | Method | What it resamples | What it preserves | What it breaks | Question | |---|---|---|---|---| | Shuffle trades | The order of the trades | Each trade and the final result | The time clustering of losses | What drawdown is possible with these trades? | | Resample trades | Trades with replacement | The distribution of a trade | Order and overlap | What result and what ruin should be expected? | | Block bootstrap | Stretches of daily returns | Dependence within each block | Dependence between blocks | How far can I trust the Sharpe and the drawdown? | | Permute the signal | The direction relative to returns | Returns, risk scaling, average exposure | The signal-return relationship | Does it get the timing right or only the exposure? | None of the three knows how many strategies were tried before settling on this one. That question, the one about selection bias, belongs to lesson 2.5.4 and comes back at the end of this one. ## The strategy and its control The strategy is a classic **time series momentum** on the 24 synthetic futures of level 2: in each market, long if the return of the last 252 sessions is positive and short if it is negative, with a position scaled by the inverse of volatility (60-session EWMA) so that each market contributes a similar risk. The signal is computed at the close of each session and traded in the next one. Lesson 2.3.1 builds it step by step; the empirical reference in real markets is Moskowitz, Ooi and Pedersen (2012)[^1]. ```python import numpy as np import pandas as pd def sharpe(x): return np.mean(x) / np.std(x, ddof=1) * np.sqrt(252) def tendencia(px, L=252, riesgo=0.30): """Sign of the L-session return, scaled by vol.""" r = px.pct_change() vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252) signo = np.sign(px.pct_change(L)) # known at the close of t escala = riesgo / vol / px.shape[1] # risk per market pnl = (signo * escala).shift(1) * r # traded at t + 1 return signo, escala, r, pnl.loc["2001-01-02":] fut = pd.read_parquet("futuros2_continuos.parquet") px = fut.pivot(index="fecha", columns="mercado", values="cierre_ajustado") signo, escala, r, pnl = tendencia(px) cartera = pnl.sum(axis=1) print(f"Sessions: {len(cartera)}") print(f"Annual volatility: {cartera.std() * np.sqrt(252):.1%}") print(f"Sharpe: {sharpe(cartera):.2f}") ``` ```output Sessions: 6287 Annual volatility: 9.0% Sharpe: 1.00 ``` Twenty-five years, a volatility of 9.0% and a Sharpe of 1.00, without costs (lesson 2.4.7 adds them; here they do not change the method). The 1.00 is higher than the 0.15 per market of the README for two reasons: diversification across 24 weakly correlated markets and what the rule captures along the way, above all the carry (the README measures a correlation of 0.50 between the carry and the 252-session sign) and the index premium. The README gives 0.98 for the same rule with a different weighting (10% divided by the 63-session volatility): the figure does not depend on that detail. Each method needs a **control**: something with the same shape as the strategy and no effect. The most direct one is a twin of the futures themselves: the same daily returns, the same rows (each session keeps the 24 markets together, and with them their correlations), but in a shuffled order of sessions. This preserves the distribution of each day and the structure across markets and destroys any time memory: in the twin, a trend rule cannot have an edge. ```python rng = np.random.default_rng(2028) orden = rng.permutation(len(r) - 1) r_g = pd.DataFrame(r.to_numpy()[1:][orden], index=r.index[1:], columns=r.columns) px_g = (1 + r_g).cumprod() # same days, different order signo_g, escala_g, r_g, pnl_g = tendencia(px_g) gemelo = pnl_g.sum(axis=1) iu = np.triu_indices(24, 1) print(f"Mean correlation between markets: actual " f"{r.corr().to_numpy()[iu].mean():.3f}, " f"twin {r_g.corr().to_numpy()[iu].mean():.3f}") print(f"Trend Sharpe in the twin: {sharpe(gemelo):.2f}") ``` ```output Mean correlation between markets: actual 0.081, twin 0.081 Trend Sharpe in the twin: 0.03 ``` The same mean correlation between markets (0.081) and a Sharpe of 0.03: what you would expect from a memoryless series. Further down, the control that comes with the README itself is also used, the `ruptura` series of `regimenes2.parquet`. ## Trade resampling: drawdown and risk of ruin This is the most widespread form of Monte Carlo among traders: take the list of closed trades and generate alternative sequences. There are two variants. **Shuffling** (permuting without replacement) keeps exactly the same trades, and therefore the same final result, and only changes the order: it shows what drawdown could have come out with the same trades. **Resampling** (with replacement) also changes which trades come out and how many times: it serves for the distribution of the result and of the risk of ruin. In a trend strategy with a daily signal, a trade is a run of sessions with the same sign in one market: ```python def operaciones(signo, pnl): """A trade: a run of sessions with the same sign.""" filas = [] for m in pnl.columns: s = signo[m].shift(1).loc[pnl.index] racha = (s != s.shift()).cumsum() g = pnl[m].groupby(racha) fechas = pnl.index.to_series().groupby(racha) filas.append(pd.DataFrame({ "mercado": m, "salida": fechas.last(), "sesiones": g.size(), "pnl": g.sum()})) return (pd.concat(filas).sort_values("salida", kind="stable") .reset_index(drop=True)) ops = operaciones(signo, pnl) print(f"{len(ops)} trades; winners: " f"{(ops['pnl'] > 0).mean():.0%}") print("Sessions per trade:", ops["sesiones"].quantile([0.25, 0.5, 0.75, 0.99]) .round().astype(int).to_dict()) (100 * ops["pnl"]).quantile([0.01, 0.1, 0.5, 0.9, 0.99]).round(3) ``` ```output 3229 trades; winners: 32% Sessions per trade: {0.25: 1, 0.5: 4, 0.75: 16, 0.99: 631} 0.01 -0.827 0.10 -0.254 0.50 -0.050 0.90 0.263 0.99 3.477 Name: pnl, dtype: float64 ``` The profile is typical of trend following: 3,229 trades, only 32% winners, half of them last 4 sessions or less (the signal changes sign several times when the 12-month return hovers around zero) and a few last for years. The median trade loses 0.050% of capital; the 99th percentile one makes 3.48%. The result is made by the few long trades, and that is why the order in which the small losses arrive matters so much for the drawdown. ```python rng = np.random.default_rng(7) x = ops["pnl"].to_numpy() N = 2000 def dd_max(m): """Maximum drawdown of each row of a matrix of results.""" eq = np.cumprod(1 + m, axis=-1) pico = np.maximum.accumulate(eq, axis=-1) return (eq / pico - 1).min(axis=-1) dd_hist = dd_max(x) dd_perm = dd_max(rng.permuted(np.tile(x, (N, 1)), axis=1)) dd_boot = dd_max(rng.choice(x, size=(N, len(x)))) q = [0.05, 0.5, 0.95] print(f"Drawdown of the actual sequence: {dd_hist:.1%}") print(f"Shuffled with a worse drawdown: " f"{(dd_perm <= dd_hist).mean():.1%}") pd.DataFrame({ "shuffle": np.quantile(dd_perm, q), "resample": np.quantile(dd_boot, q)}, index=["5%", "median", "95%"]).mul(100).round(1) ``` ```output Drawdown of the actual sequence: -21.9% Shuffled with a worse drawdown: 0.0% shuffle resample 5% -15.0 -15.3 median -10.0 -10.1 95% -7.4 -7.2 ``` `rng.permuted(..., axis=1)` shuffles each row separately: 2,000 different orders of the same trades in a single call. The result is uncomfortable. With the trades in random order, the median drawdown is −10.0% and the 5th percentile one, −15.0%. The actual sequence had −21.9%: **worse than every** shuffled sequence. *Fig. — Maximum drawdown of 2,000 random orderings of the trend portfolio's 3,229 trades, against that of the real sequence.* Histogram of maximum drawdowns in one-point bins, between −22% and −5%, centred near −10%. The highlighted line marks the real trade sequence's drawdown, −21.9%: none of the 2,000 random orderings is worse. Real losses arrived more clustered than shuffling assumes. It is not bad luck. Shuffling assumes the trades are interchangeable, and in a 24-market portfolio they are not: they overlap in time, and when the market turns (the end of a trend, a crisis) several markets lose at once. The next cell measures it: ```python eq = np.cumprod(1 + x) fondo = int(np.argmin(eq / np.maximum.accumulate(eq))) pico = int(np.argmax(eq[:fondo + 1])) print(f"Worst stretch: {ops['salida'][pico]:%Y-%m-%d} to " f"{ops['salida'][fondo]:%Y-%m-%d}, {fondo - pico} trades") mes = ops["salida"].dt.to_period("M").to_numpy() real = pd.Series(x).groupby(mes).sum().std() barajado = np.mean([pd.Series(rng.permutation(x)).groupby(mes) .sum().std() for _ in range(200)]) print(f"Standard deviation of the monthly result: actual {real:.2%}, " f"shuffled {barajado:.2%}") ``` ```output Worst stretch: 2001-01-02 to 2003-01-16, 378 trades Standard deviation of the monthly result: actual 3.30%, shuffled 2.52% ``` With the trades in their actual month, the monthly result has a standard deviation of 3.30%; with the same trades spread at random across the same months, 2.52%. Gains and losses cluster in time, and the worst stretch of the actual sequence, from January 2001 to January 2003, chains 378 trades. With that caveat, resampling serves for a question the backtest does not answer: **how much leverage** the strategy can take. If ruin is defined as a drawdown of 20% or 40% (the point at which a manager would shut the strategy down or a client would pull their money), it is enough to multiply the trades by the leverage: ```python filas = {} for apal in (1, 2, 3, 4): dd = dd_max(apal * rng.choice(x, size=(N, len(x)))) filas[f"x{apal}"] = { "dd_median": np.median(dd), "P(dd < -20%)": (dd < -0.20).mean(), "P(dd < -40%)": (dd < -0.40).mean()} pd.DataFrame(filas).T.round(3) ``` ```output dd_median P(dd < -20%) P(dd < -40%) x1 -0.101 0.005 0.000 x2 -0.192 0.428 0.000 x3 -0.278 0.952 0.046 x4 -0.359 1.000 0.289 ``` Without leverage, the probability of going through a 20% drawdown in 25 years is 0.5%; with twice the size, 43%; with three times, 95%. That of reaching 40% is zero with twice the size (none of 2,000 paths), 4.6% with three times and 29% with four times. The median drawdown grows somewhat less than proportionally (−10.1%, −19.2%, −27.8% and −35.9%), because with compounded capital each loss is applied to an already smaller capital; the probability of crossing a fixed threshold, on the other hand, shoots up. And all these figures inherit the optimistic bias of the previous cell: the actual sequence already had −21.9% without leverage. ## Block bootstrap on daily returns The bootstrap of daily returns avoids the overlap problem: each day already contains all the open positions, with their correlation. But Efron's classic bootstrap (drawing single days with replacement) assumes the days are independent, and the returns of a portfolio are not: their volatility comes in clusters (lesson 1.6.2). The solution is to resample **blocks** of consecutive days. Künsch (1989) proposed fixed-length blocks; the **circular bootstrap** joins the end of the series to the beginning so that every day has the same probability of appearing, and the **stationary bootstrap** of Politis and Romano (1994) uses blocks of random length, geometric with mean $b$: $$ P(L = k) = \frac{1}{b}\left(1 - \frac{1}{b}\right)^{k-1}, \qquad E[L] = b $$ With blocks of random length, the resampled series is stationary, which fixed blocks do not guarantee. The mechanism fits in a loop: ```python def indices_estacionario(n, largo, b, rng): """Indices of a stationary resample (Politis and Romano). Each step moves on to the next day with probability 1 - 1/b or jumps to a random day with probability 1/b: blocks of geometric length with mean b, wrapping around circularly.""" idx = np.empty(largo, dtype=np.int64) idx[0] = rng.integers(n) salto = rng.random(largo) < 1 / b nuevo = rng.integers(n, size=largo) for t in range(1, largo): idx[t] = nuevo[t] if salto[t] else (idx[t - 1] + 1) % n return idx rng = np.random.default_rng(11) print(indices_estacionario(len(cartera), 16, 5, rng)) ``` ```output [ 841 842 3221 5917 5918 5275 3452 3453 3454 3455 3456 3457 3458 5179 5180 5181] ``` You can see the blocks: 841-842, a single day, 5917-5918, another single day, seven consecutive days from 3452 and another three at the end, with a mean of 5 because `b=5`. The `arch` library has the efficient version (`StationaryBootstrap`, `CircularBlockBootstrap`) and the automatic **block length** selector of Politis and White (2004), with the correction by Patton, Politis and White (2009), which estimates the optimal $b$ from the autocorrelation of the series: ```python from arch.bootstrap import optimal_block_length pd.concat({"r": optimal_block_length(cartera), "r²": optimal_block_length(cartera ** 2)}).round(1) ``` ```output stationary circular r None 1.2 1.4 r² None 47.6 54.5 ``` The returns have almost no autocorrelation, and the selector asks for blocks of 1 session. Their squares, which measure volatility, ask for 48 (stationary) or 55 (circular). The choice depends on which statistic you want: the mean depends on the returns; the drawdown, on the volatility clusters. The prudent thing is to look at several lengths: ```python from arch.bootstrap import StationaryBootstrap def drawdown(x): eq = np.cumprod(1 + np.asarray(x)) return (eq / np.maximum.accumulate(eq) - 1).min() def stats(x): return np.array([sharpe(x), drawdown(x)]) print(f"Historical: Sharpe {sharpe(cartera):.2f}, " f"drawdown {drawdown(cartera):.1%}") filas = {} for b in (1, 5, 20, 75, 250): bs = StationaryBootstrap(b, cartera.to_numpy(), seed=3) v = bs.apply(stats, 1000) filas[b] = {"sharpe_2.5": np.quantile(v[:, 0], 0.025), "sharpe_97.5": np.quantile(v[:, 0], 0.975), "dd_median": np.median(v[:, 1]), "dd_5%": np.quantile(v[:, 1], 0.05)} pd.DataFrame(filas).T.rename_axis("block").round(3) ``` ```output Historical: Sharpe 1.00, drawdown -14.7% sharpe_2.5 sharpe_97.5 dd_median dd_5% block 1 0.591 1.391 -0.179 -0.264 5 0.613 1.374 -0.178 -0.270 20 0.630 1.380 -0.178 -0.265 75 0.600 1.399 -0.177 -0.262 250 0.661 1.326 -0.169 -0.235 ``` Three readings. The first: the 95% interval of the Sharpe goes from 0.6 to 1.4 with any block. It resembles Lo's (2002)[^2] approximation for i.i.d. returns, $\sigma(\widehat{SR}) \approx \sqrt{(1 + SR^2/2)/T}$ with $T$ in years: with 1.00 and 25 years, 0.24, and ±1.96 times that gives 0.52 to 1.48; the bootstrap comes out somewhat narrower. Twenty-five years of data are not enough to tell a Sharpe of 0.6 from one of 1.4. The second: in this portfolio the drawdown barely depends on the block. The median goes from −17.9% with single days to −17.7% with 75-session blocks, and the 5th percentile one from −26.4% to −26.2%. The reason lies in the construction: each position is scaled by the inverse of its volatility, and that removes from the portfolio almost all the volatility clustering a long block would preserve. In an unscaled series it is different. The next cell repeats the test with the `IDA` index future bought and held: ```python ida = r["IDA"].loc["2001-01-02":].to_numpy() b_ida = optimal_block_length(ida ** 2)["stationary"].iloc[0] print(f"IDA: Politis and White block for r²: {b_ida:.0f}") print(f"Historical drawdown: {drawdown(ida):.1%}") for b in (1, 75): v = StationaryBootstrap(b, ida, seed=3).apply( lambda x: np.array([drawdown(x)]), 500)[:, 0] print(f"Block {b:>2}: median drawdown {np.median(v):.1%}, " f"5th percentile {np.quantile(v, 0.05):.1%}") ``` ```output IDA: Politis and White block for r²: 156 Historical drawdown: -62.2% Block 1: median drawdown -53.0%, 5th percentile -74.9% Block 75: median drawdown -65.3%, 5th percentile -86.8% ``` In `IDA` the selector asks for blocks of 156 sessions for the squares (48 in the portfolio), and the median drawdown goes from −53.0% with single days to −65.3% with 75-session blocks; the 5th percentile, from −74.9% to −86.8%. Long blocks preserve the volatility clusters, and with them the chained falls; single days spread them out and the drawdown comes out optimistic: with them, the historical drawdown of `IDA` (−62.2%) would look worse than normal. The practical rule: the block matters for the drawdown to the extent that the series has volatility clusters, and the prudent thing is to look at several lengths. The third: the historical drawdown of the portfolio (−14.7%) is milder than the bootstrap median with any block (between −16.9% and −17.9%). The history that happened was somewhat lucky in the order of its days: the risk worth expecting is that of the bootstrap, not that of the backtest. (The −21.9% of the trade sequence is not comparable: there each trade is booked in full on the day it exits.) ### The equity fan With the stationary bootstrap you can generate paths of any length. A thousand five-year paths with blocks of 75 sessions on average give the fan in the figure at the top: ```python rng = np.random.default_rng(5) n, largo, b = len(cartera), 5 * 252, 75 caminos = np.array([ cartera.to_numpy()[indices_estacionario(n, largo, b, rng)] for _ in range(1000)]) capital = np.cumprod(1 + caminos, axis=1) bandas = pd.DataFrame( np.quantile(capital, [0.05, 0.25, 0.5, 0.75, 0.95], axis=0).T, columns=["p5", "p25", "p50", "p75", "p95"]) print(bandas.iloc[[251, 503, 755, 1007, 1259]] .set_axis([1, 2, 3, 4, 5]).rename_axis("year").round(2)) print(f"P(loss at 5 years): {(capital[:, -1] < 1).mean():.1%}") ``` ```output p5 p25 p50 p75 p95 year 1 0.94 1.02 1.09 1.16 1.25 2 0.96 1.10 1.20 1.30 1.45 3 1.01 1.17 1.29 1.44 1.67 4 1.07 1.26 1.41 1.59 1.88 5 1.14 1.36 1.54 1.77 2.10 P(loss at 5 years): 1.2% ``` The median reaches 1.54 in five years; the 25-75% band, from 1.36 to 1.77. The 5th percentile is the line that matters for deciding the size: it goes through 0.94 in the first year (a 6% loss), through 0.96 in the second, does not get back to 1 until the third (1.01) and at five years is at 1.14. Even with a Sharpe of 1.00, 1.2% of the paths lose money at five years and one in twenty spends two years in losses. That is what you should show anyone who is going to invest before they live through it. ## Permuting the signal: does it get the timing right? The bootstrap says how much the Sharpe varies; it does not say whether it differs from what a strategy with no skill would give. A **permutation test** builds that reference directly: the relationship between signal and returns is broken and the statistic is recomputed many times. The p-value is the fraction of permutations that match or exceed what was observed, with the usual correction so that it is never zero: $$ p = \frac{1 + \#\{i : S^{(i)} \ge S_{\text{obs}}\}}{1 + N} $$ The delicate part is deciding **what** to permute. The question is whether the direction chosen at each moment adds anything, so only the sign is permuted; the volatility scaling and the returns stay in place. And there are two ways of permuting the sign: **shuffling** the sessions (which also breaks the persistence of the signal) or **shifting it circularly** by a random number of sessions (which preserves its runs, its turnover and its autocorrelation, and only changes its alignment with the market). White (2000) and Hansen (2005)[^3] extend the idea to the case of many rules at once. ```python def p_valor_perm(signo, escala, r, n_perm=1000, seed=0, modo="circular"): """Observed Sharpe, p-value and null distribution. Only the DIRECTION (the sign) is permuted; the volatility scaling and the returns stay in place.""" rng = np.random.default_rng(seed) s = signo.shift(1).to_numpy() e = escala.shift(1).to_numpy() rr = r.to_numpy() ok = ~(np.isnan(s).any(axis=1) | np.isnan(e).any(axis=1) | np.isnan(rr).any(axis=1)) s, e, rr = s[ok], e[ok], rr[ok] obs = sharpe((s * e * rr).sum(axis=1)) nula = np.empty(n_perm) for i in range(n_perm): if modo == "shuffle": sp = s[rng.permutation(len(s))] else: # circular shift of at least one year sp = np.roll(s, rng.integers(252, len(s) - 252), axis=0) nula[i] = sharpe((sp * e * rr).sum(axis=1)) p = (1 + (nula >= obs).sum()) / (1 + n_perm) return obs, p, nula res, nulas = {}, {} for nombre, args in {"trend": (signo, escala, r), "twin": (signo_g, escala_g, r_g)}.items(): for modo in ("shuffle", "circular"): obs, p, nula = p_valor_perm(*args, modo=modo) nulas[(nombre, modo)] = nula res[(nombre, modo)] = {"sharpe": obs, "p": p, "null_mean": nula.mean(), "null_95": np.quantile(nula, 0.95)} pd.DataFrame(res).T.round(3) ``` ```output sharpe p null_mean null_95 trend shuffle 0.999 0.001 0.202 0.519 circular 0.999 0.001 0.048 0.552 twin shuffle 0.026 0.536 0.039 0.376 circular 0.026 0.248 -0.066 0.212 ``` *Fig. — Signal permutation test (circular shift, 1,000 times): null distribution of the Sharpe ratio and observed Sharpe, in the futures and in their memoryless twin.* Two histograms. Left, futures with the planted effect: the null is centred at 0.05 and the observed Sharpe, 1.00 (highlighted), lies to the right of every permutation; p = 0.001. Right, the twin with shuffled sessions: the null is centred at −0.07 and the observed Sharpe, 0.03, falls inside it; p = 0.25. The method tells the two cases apart. In the futures with a planted effect, the observed Sharpe (1.00) beats every permutation, both shuffling and shifting: p = 0.001, the minimum possible with 1,000. In the memoryless twin, the 0.03 falls inside its null: p = 0.25 with the circular shift (0.54 shuffling). Note the mean of the null: 0.20 when shuffling and 0.05 with the circular shift, not 0. Any permutation preserves the proportion of long and short sessions of each market, and with it the part of the result that comes from the average exposure: ```python largo = (signo.shift(1).loc[pnl.index] > 0).mean() largo.sort_values().round(2).iloc[[0, 1, -2, -1]].to_dict() ``` ```output {'MEA': 0.42, 'DIC': 0.43, 'IDB': 0.7, 'IDA': 0.71} ``` The portfolio is long 71% of the sessions in `IDA` and 70% in `IDB`, and only 42% in gold (`MEA`). The indices carry the equity premium written into the generator, so a permutation that keeps them long 70% of the time makes something with any order; the same happens with the part of the carry the rule collects. The test does not ask whether the strategy makes money, but whether the chosen **timing** adds anything on top of that exposure. A strategy that is always long in a bull market would make money and would not pass the test. ```mermaid flowchart TB Q{What is the question?} -- risk --> D[Block bootstrap] Q -- timing? --> P[Permutation] Q -- right pick? --> S[DSR and PBO] D --> C{Losses together?} C -- yes --> B[Long blocks] C -- no --> T[Trades or i.i.d.] ``` ### Power: failing to reject is not having no effect All 24 markets have the same planted effect. Market by market: ```python p_merc = pd.Series({ m: p_valor_perm(signo[[m]], escala[[m]], r[[m]], n_perm=500, seed=1)[1] for m in r.columns}) print(f"Markets with p < 0.05: {(p_merc < 0.05).sum()} of 24") print(p_merc.sort_values().round(3).head(6).to_dict()) ``` ```output Markets with p < 0.05: 7 of 24 {'AGB': 0.002, 'BOD': 0.002, 'BOC': 0.01, 'AGC': 0.012, 'BOA': 0.034, 'AGA': 0.042} ``` Only 7 of the 24 pass at 5%, with agricultural commodities and bonds in the lead. With a true momentum Sharpe of around 0.15 per market and 25 years of data, the standard error of the Sharpe (≈ 0.2) is larger than the effect: the test almost never detects it in a single market. The portfolio detects it because it adds up 24 weak, weakly correlated effects. Discarding 17 markets as “not significant” would be discarding a real effect; the correct conclusion is that a single market does not have enough data. ### And in the noise, how many false positives? The README's control is the `ruptura` series of `regimenes2.parquet` after its break (2013-05-22): i.i.d. normal returns with zero mean, where no trend rule has an edge. A single noise series is not enough to know whether the test is well calibrated, so 200 simulated series like it are added: ```python reg = pd.read_parquet("regimenes2.parquet") ru = (reg[reg["serie"] == "ruptura"].set_index("fecha")[["precio"]] .loc["2013-05-22":]) # after the break: i.i.d., mean 0 s_ru, e_ru, r_ru, _ = tendencia(ru, riesgo=0.10) obs, p, _ = p_valor_perm(s_ru, e_ru, r_ru) print(f"“ruptura” series since 2013: Sharpe {obs:.2f}, p = {p:.3f}") rng = np.random.default_rng(99) ps = [] for i in range(200): ruido = pd.DataFrame( np.cumprod(1 + 0.013 * rng.standard_normal(len(ru))), index=ru.index) ps.append(p_valor_perm(*tendencia(ruido, riesgo=0.10)[:3], n_perm=200, seed=i)[1]) ps = np.array(ps) print(f"Noise series with p < 0.05: {(ps < 0.05).mean():.1%}") print(np.histogram(ps, bins=5, range=(0, 1))[0]) ``` ```output “ruptura” series since 2013: Sharpe 0.55, p = 0.032 Noise series with p < 0.05: 4.5% [36 47 33 27 57] ``` The README series gives a Sharpe of 0.55 and p = 0.032: a **false positive**. There is no error in the test: in 200 noise series like it, 4.5% come out with p < 0.05, and the p-values are spread roughly uniformly between 0 and 1 (36, 47, 33, 27 and 57 per quintile). That is what a 5% level means: one in twenty controls comes out “significant.” That is why a single p-value below 0.05 proves nothing, and why a control that passes the test does not invalidate the method: it invalidates the habit of testing a single thing and believing the result. ## What no resampling corrects The three methods work on a strategy that has already been chosen. If that strategy is the best of many, they all inherit the bias of the choice. The `ruido2_estrategias.parquet` set has 1,000 strategies, 995 of them with no edge at all (the truth is in `ruido2_verdad.csv`): ```python rs = pd.read_parquet("ruido2_estrategias.parquet").pivot( index="fecha", columns="estrategia", values="rendimiento") verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia") sin_ventaja = verdad.index[verdad["sharpe_real"] == 0] sr = rs[sin_ventaja].apply(sharpe) mejor = sr.idxmax() print(f"{len(sin_ventaja)} strategies with no edge; the best: " f"{mejor}, Sharpe {sr.max():.2f}") bs = StationaryBootstrap(5, rs[mejor].to_numpy(), seed=2) v = bs.apply(sharpe, 2000)[:, 0] print(f"95% interval: {np.quantile(v, 0.025):.2f} to " f"{np.quantile(v, 0.975):.2f}") print(f"Resamples with Sharpe <= 0: {(v <= 0).mean():.2%}") ``` ```output 995 strategies with no edge; the best: e0598, Sharpe 1.50 95% interval: 0.46 to 2.54 Resamples with Sharpe <= 0: 0.15% ``` The best of the 995 has a Sharpe of 1.50 over four years, a bootstrap interval from 0.46 to 2.54 and only 0.15% of resamples with a negative Sharpe. Everything points to an excellent strategy, and its true Sharpe is zero. The bootstrap answers its own question well (how much the Sharpe of **this** series varies) and cannot answer another one it was not asked: how many series were looked at before choosing it. With 995 trials and a standard error of the Sharpe of 0.5, a maximum of 1.5 is what you would expect. The correction is the Deflated Sharpe Ratio or the PBO of lesson 2.5.4, which need the number of tests, and the experiment journal of lesson 2.2.4, which is where that number gets written down. In summary, the order of work with a candidate strategy: 1. Block bootstrap of the daily returns, with several block lengths: interval of the Sharpe, of the drawdown and a fan to decide the size. 2. Trade resampling only as a floor for the risk, checking where the actual sequence lands. 3. Signal permutation test (circular if the signal is persistent) to separate skill from exposure, also on a control with no effect. 4. Deflated Sharpe or PBO with the real number of tests (lesson 2.5.4); lesson 2.5.8 puts everything together in a checklist. ## Exercises 1. Repeat the risk of ruin with the stationary bootstrap of the **daily returns** (blocks of 75) instead of the trades: probability of a 20% drawdown in 25 years with leverage 1 and 2. Compare it with the trades table. **Solution.** ```python rng = np.random.default_rng(21) n = len(cartera) caminos = np.array([cartera.to_numpy()[ indices_estacionario(n, n, 75, rng)] for _ in range(500)]) for apal in (1, 2): dd = dd_max(apal * caminos) print(f"x{apal}: median drawdown {np.median(dd):.1%}, " f"P(dd < -20%) {(dd < -0.20).mean():.1%}") ``` ```output x1: median drawdown -17.6%, P(dd < -20%) 29.8% x2: median drawdown -33.0%, P(dd < -20%) 99.6% ``` With the daily returns in blocks, the median drawdown over 25 years without leverage is −17.6% and the probability of going through −20%, 29.8%, versus the 0.5% given by the trades treated as independent. With leverage 2, the median drawdown is −33.0% and −20% arrives in 99.6% of the paths. The block bootstrap preserves the overlap of the positions and the volatility clusters; trade resampling does not. For deciding the size, this is the reference figure, and the difference between the two is the size of the error trade resampling makes in a portfolio. 2. Apply the circular permutation test to the same rule with a 63-session window instead of 252, on the futures and on the twin. Does the test detect an effect that the README plants at 252 sessions? **Solution.** ```python for nombre, precios in (("trend", px), ("twin", px_g)): s63, e63, r63, _ = tendencia(precios, L=63) obs, p, nula = p_valor_perm(s63, e63, r63, n_perm=500) print(f"{nombre}: Sharpe {obs:.2f}, p = {p:.3f}, " f"null mean {nula.mean():.2f}") ``` ```output trend: Sharpe 0.63, p = 0.020, null mean 0.07 twin: Sharpe 0.00, p = 0.403, null mean -0.04 ``` With 63 sessions the rule still captures part of the effect in the futures: Sharpe 0.63 and p = 0.020. The 3-month sign is correlated with the 12-month one, and the carry, which is persistent, shows up in the return of any window. In the twin there is nothing to capture: Sharpe 0.00 and p = 0.403. That a window different from the planted one also works is the idea of lesson 2.5.6: a real effect usually gives a parameter plateau, not a peak. 3. A strategy is long every day in an index that rises 8% a year. What p-value does the signal permutation test give, and why? What does that tell you about what the test measures? **Solution.** If the position is always +1, any permutation (shuffling or shifting) returns the same series of signs: every permutation has exactly the observed Sharpe and the p-value is (1 + N)/(1 + N) = 1. The strategy makes money, but it has no timing skill at all: its whole result is exposure. The permutation test measures only the value of timing relative to a portfolio with the same average exposure; the return of the exposure (the market premium) has to be judged with other tools, such as alpha against an index (lesson 2.6.2). [^1]: Moskowitz, Ooi and Pedersen (2012), “Time series momentum,” Journal of Financial Economics 104(2), 228-250, [doi:10.1016/j.jfineco.2011.11.003](https://doi.org/10.1016/j.jfineco.2011.11.003). [^2]: Lo, A. W. (2002), “The Statistics of Sharpe Ratios,” Financial Analysts Journal 58(4), 36-52, [doi:10.2469/faj.v58.n4.2453](https://doi.org/10.2469/faj.v58.n4.2453). [^3]: Hansen, P. R. (2005), “A Test for Superior Predictive Ability,” Journal of Business & Economic Statistics 23(4), 365-380, [doi:10.1198/073500105000000063](https://doi.org/10.1198/073500105000000063). ## Self-assessment 1. When the 3,229 trades of the trend portfolio are shuffled 2,000 times, no sequence has a worse drawdown than the actual sequence (−21.9%). What does it indicate? - A. That the strategy was unlucky and the actual drawdown will not happen again - B. That there is an error in the drawdown calculation - C. That more simulations are needed - D. That the actual losses came clustered in time, and shuffling the trades (which assumes independence) underestimates the drawdown *Answer D: That the actual losses came clustered in time, and shuffling the trades (which assumes independence) underestimates the drawdown.* Shuffling assumes that the order of the trades is interchangeable. In a portfolio with 24 markets the trades overlap and the losses arrive together: the standard deviation of the monthly result is 3.30% with the actual order and 2.52% with the shuffled order. Trade resampling is a floor for the risk, not an estimate. 2. With the stationary bootstrap, the median drawdown of the trend portfolio barely changes between blocks of 1 and of 75 sessions (−17.9% and −17.7%), while that of buy-and-hold IDA goes from −53.0% to −65.3%. Why? - A. Because the portfolio has more sessions than IDA - B. Because the stationary bootstrap does not work for indices - C. Because the portfolio scales each position by the inverse of its volatility and has few volatility clusters left; in IDA they are still there, and long blocks preserve them - D. Because with long blocks the drawdown always comes out worse *Answer C: Because the portfolio scales each position by the inverse of its volatility and has few volatility clusters left; in IDA they are still there, and long blocks preserve them.* The drawdown depends on chained falls being preserved, and that only happens if the series has volatility clusters that a long block can keep. The Politis and White selector anticipates it: it asks for blocks of 48 sessions for the squares of the portfolio and of 156 for those of IDA. The Sharpe interval barely changes with the block in either case. 3. In the permutation test, why is only the sign of the position permuted and not the volatility scaling? - A. Because the question is whether getting the direction right at each moment adds anything; shuffling the scaling too would mix in a second thing (the risk adjustment) and the null would no longer be comparable - B. For computation speed - C. Because the scaling is always constant - D. Because that way the p-value comes out lower *Answer A: Because the question is whether getting the direction right at each moment adds anything; shuffling the scaling too would mix in a second thing (the risk adjustment) and the null would no longer be comparable.* A permutation test answers a specific question: it breaks the relationship you want to test (direction versus return) and preserves everything else. If the volatility scaling is shuffled, the null takes large positions on turbulent days and its Sharpe falls for another reason. 4. The null of the test with the shuffled signal has a mean of 0.20, not 0. Why? - A. Because the random number generator is biased - B. Because shuffling preserves the proportion of long and short days of each market: the null collects the average exposure (the index premium, the carry). The test measures whether the chosen timing adds anything on top of that exposure - C. Because futures always go up - D. Because there are few permutations *Answer B: Because shuffling preserves the proportion of long and short days of each market: the null collects the average exposure (the index premium, the carry). The test measures whether the chosen timing adds anything on top of that exposure.* The portfolio is long 71% of the sessions in IDA and 70% in IDB, indices that carry the equity premium. Any permutation preserves those percentages and, with them, the part of the result that comes from the average exposure. That is why a permutation test does not confuse exposure with skill. 5. The best of 995 strategies with no edge has a Sharpe of 1.50 and its 95% bootstrap interval goes from 0.46 to 2.54. What is wrong? - A. The bootstrap is badly coded - B. Nothing: the strategy has an edge - C. The bootstrap resamples the series already chosen: it does not know that the best of 995 was picked. Selection is corrected with the Deflated Sharpe or the PBO (lesson 2.5.4) - D. Longer blocks are needed *Answer C: The bootstrap resamples the series already chosen: it does not know that the best of 995 was picked. Selection is corrected with the Deflated Sharpe or the PBO (lesson 2.5.4).* Resampling a series tells you how much its statistic varies by chance within that series, not how many series were looked at before choosing it. Among 995 strategies with no edge, the best has a high Sharpe by construction, and its interval is centered on that inflated value. ## Key points - Each Monte Carlo method preserves one structure and breaks another: shuffling trades breaks the clustering of losses; the block bootstrap preserves volatility clusters within the block; permuting the signal breaks only the relationship between direction and return. - In the trend portfolio, the drawdown of the actual sequence of trades (−21.9%) is worse than that of the 2,000 shuffled sequences: with overlapping trades, trade resampling underestimates the risk. - Block length matters for the drawdown when the series has volatility clusters (IDA: median of −53.0% with blocks of 1 and −65.3% with blocks of 75) and hardly at all in a volatility-scaled portfolio or for the Sharpe interval (from 0.6 to 1.4 with any block). - The historical drawdown of the portfolio (−14.7%) is milder than the bootstrap median (≈ −17.8%): the backtest shows a somewhat lucky history, not the risk you should expect. - The permutation test separates timing skill from average exposure: p = 0.001 in the trend with a planted effect, p = 0.25 in its memoryless twin, and 4.5% false positives at 5% across 200 noise series. - With 25 years of data and a weak effect per market, only 7 of the 24 markets pass the test individually: failing to reject is not the same as having no effect. - No resampling corrects selection: the best of 995 strategies with no edge has a bootstrap interval that excludes zero. ## Resources ### Research - [Stochastic Processes - Random Walks](https://pythonparatrading.com/en/stochastic-processes/) · Research-0005 · 11 min ### Documentation - [arch: bootstrap (StationaryBootstrap, CircularBlockBootstrap, optimal_block_length)](https://arch.readthedocs.io/en/latest/bootstrap/bootstrap.html) · arch.readthedocs.io ### Books - [Politis, D. N. and Romano, J. P. (1994): The Stationary Bootstrap. Journal of the American Statistical Association 89(428), 1303-1313](https://doi.org/10.1080/01621459.1994.10476870) · doi.org - [Politis, D. N. and White, H. (2004): Automatic Block-Length Selection for the Dependent Bootstrap. Econometric Reviews 23(1), 53-70 (correction in 28(4), 2009)](https://doi.org/10.1081/ETC-120028836) · doi.org - [White, H. (2000): A Reality Check for Data Snooping. Econometrica 68(5), 1097-1126](https://doi.org/10.1111/1468-0262.00152) · doi.org - [Moskowitz, T., Ooi, Y. H. and Pedersen, L. H. (2012): Time series momentum. Journal of Financial Economics 104(2), 228-250](https://doi.org/10.1016/j.jfineco.2011.11.003) · doi.org ### Data - [Level 2 data: continuous futures, 24 markets, 2000-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/futuros2_continuos.parquet) · Download · futuros2_continuos.parquet Previous lesson: [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/) Next lesson: [2.5.6 · Parameter sensitivity: plateaus versus peaks, robustness across assets and periods](https://pythonparatrading.com/en/python-trading-course/level-2/parameter-sensitivity-and-robustness/)