Monte Carlo: trade resampling, block bootstrap and signal permutations.

An equity curve is a sample of size one

A backtest returns an equity curve, a Sharpe and a maximum drawdown. They are figures from one history: the one that happened. Had the market dealt the same good and bad days in a different order, or had the strategy closed its trades in a different sequence, the drawdown would be different. Lesson 2.1.1 introduced the bootstrap to estimate the uncertainty of a statistic; here it is applied to the practical question of a trading system: which part of the result is structure and which part is the luck of the order.

“Monte Carlo” in this context groups three families of resampling. Each one starts from a different object of the backtest and answers a different question:

flowchart TB
  B[Backtest] --> O[Trades]
  B --> R[Returns]
  B --> S[Signal and market]
  O --> T[Trade resampling]
  R --> K[Block bootstrap]
  S --> P[Permutation]
  T --> T2[Drawdown and ruin]
  K --> K2[CI and fan]
  P --> P2[Timing p-value]

None of the three knows how many strategies were tried before settling on this one. That question, the one about selection bias, belongs to lesson 2.5.4 and comes back at the end of this one.

The strategy and its control

The strategy is a classic time series momentum on the 24 synthetic futures of level 2: in each market, long if the return of the last 252 sessions is positive and short if it is negative, with a position scaled by the inverse of volatility (60-session EWMA) so that each market contributes a similar risk. The signal is computed at the close of each session and traded in the next one. Lesson 2.3.1 builds it step by step; the empirical reference in real markets is Moskowitz, Ooi and Pedersen (2012)11Note 1 · Moskowitz, Ooi and Pedersen (2012), “Time series momentum,” Journal of Financial Economics 104(2), 228-250, doi:10.1016/j.jfineco.2011.11.003..

import numpy as np
import pandas as pd

def sharpe(x):
    return np.mean(x) / np.std(x, ddof=1) * np.sqrt(252)

def tendencia(px, L=252, riesgo=0.30):
    """Sign of the L-session return, scaled by vol."""
    r = px.pct_change()
    vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252)
    signo = np.sign(px.pct_change(L))       # known at the close of t
    escala = riesgo / vol / px.shape[1]     # risk per market
    pnl = (signo * escala).shift(1) * r     # traded at t + 1
    return signo, escala, r, pnl.loc["2001-01-02":]

fut = pd.read_parquet("futuros2_continuos.parquet")
px = fut.pivot(index="fecha", columns="mercado",
               values="cierre_ajustado")
signo, escala, r, pnl = tendencia(px)
cartera = pnl.sum(axis=1)
print(f"Sessions: {len(cartera)}")
print(f"Annual volatility: {cartera.std() * np.sqrt(252):.1%}")
print(f"Sharpe: {sharpe(cartera):.2f}")

Sessions: 6287
Annual volatility: 9.0%
Sharpe: 1.00

Twenty-five years, a volatility of 9.0% and a Sharpe of 1.00, without costs (lesson 2.4.7 adds them; here they do not change the method). The 1.00 is higher than the 0.15 per market of the README for two reasons: diversification across 24 weakly correlated markets and what the rule captures along the way, above all the carry (the README measures a correlation of 0.50 between the carry and the 252-session sign) and the index premium. The README gives 0.98 for the same rule with a different weighting (10% divided by the 63-session volatility): the figure does not depend on that detail.

Each method needs a control: something with the same shape as the strategy and no effect. The most direct one is a twin of the futures themselves: the same daily returns, the same rows (each session keeps the 24 markets together, and with them their correlations), but in a shuffled order of sessions. This preserves the distribution of each day and the structure across markets and destroys any time memory: in the twin, a trend rule cannot have an edge.

rng = np.random.default_rng(2028)
orden = rng.permutation(len(r) - 1)
r_g = pd.DataFrame(r.to_numpy()[1:][orden], index=r.index[1:],
                   columns=r.columns)
px_g = (1 + r_g).cumprod()                  # same days, different order
signo_g, escala_g, r_g, pnl_g = tendencia(px_g)
gemelo = pnl_g.sum(axis=1)
iu = np.triu_indices(24, 1)
print(f"Mean correlation between markets: actual "
      f"{r.corr().to_numpy()[iu].mean():.3f}, "
      f"twin {r_g.corr().to_numpy()[iu].mean():.3f}")
print(f"Trend Sharpe in the twin: {sharpe(gemelo):.2f}")

Mean correlation between markets: actual 0.081, twin 0.081
Trend Sharpe in the twin: 0.03

The same mean correlation between markets (0.081) and a Sharpe of 0.03: what you would expect from a memoryless series. Further down, the control that comes with the README itself is also used, the ruptura series of regimenes2.parquet.

Trade resampling: drawdown and risk of ruin

This is the most widespread form of Monte Carlo among traders: take the list of closed trades and generate alternative sequences. There are two variants. Shuffling (permuting without replacement) keeps exactly the same trades, and therefore the same final result, and only changes the order: it shows what drawdown could have come out with the same trades. Resampling (with replacement) also changes which trades come out and how many times: it serves for the distribution of the result and of the risk of ruin.

In a trend strategy with a daily signal, a trade is a run of sessions with the same sign in one market:

def operaciones(signo, pnl):
    """A trade: a run of sessions with the same sign."""
    filas = []
    for m in pnl.columns:
        s = signo[m].shift(1).loc[pnl.index]
        racha = (s != s.shift()).cumsum()
        g = pnl[m].groupby(racha)
        fechas = pnl.index.to_series().groupby(racha)
        filas.append(pd.DataFrame({
            "mercado": m, "salida": fechas.last(),
            "sesiones": g.size(), "pnl": g.sum()}))
    return (pd.concat(filas).sort_values("salida", kind="stable")
            .reset_index(drop=True))

ops = operaciones(signo, pnl)
print(f"{len(ops)} trades; winners: "
      f"{(ops['pnl'] > 0).mean():.0%}")
print("Sessions per trade:",
      ops["sesiones"].quantile([0.25, 0.5, 0.75, 0.99])
      .round().astype(int).to_dict())
(100 * ops["pnl"]).quantile([0.01, 0.1, 0.5, 0.9, 0.99]).round(3)

3229 trades; winners: 32%
Sessions per trade: {0.25: 1, 0.5: 4, 0.75: 16, 0.99: 631}
0.01   -0.827
0.10   -0.254
0.50   -0.050
0.90    0.263
0.99    3.477
Name: pnl, dtype: float64

The profile is typical of trend following: 3,229 trades, only 32% winners, half of them last 4 sessions or less (the signal changes sign several times when the 12-month return hovers around zero) and a few last for years. The median trade loses 0.050% of capital; the 99th percentile one makes 3.48%. The result is made by the few long trades, and that is why the order in which the small losses arrive matters so much for the drawdown.

rng = np.random.default_rng(7)
x = ops["pnl"].to_numpy()
N = 2000

def dd_max(m):
    """Maximum drawdown of each row of a matrix of results."""
    eq = np.cumprod(1 + m, axis=-1)
    pico = np.maximum.accumulate(eq, axis=-1)
    return (eq / pico - 1).min(axis=-1)

dd_hist = dd_max(x)
dd_perm = dd_max(rng.permuted(np.tile(x, (N, 1)), axis=1))
dd_boot = dd_max(rng.choice(x, size=(N, len(x))))

q = [0.05, 0.5, 0.95]
print(f"Drawdown of the actual sequence: {dd_hist:.1%}")
print(f"Shuffled with a worse drawdown: "
      f"{(dd_perm <= dd_hist).mean():.1%}")
pd.DataFrame({
    "shuffle": np.quantile(dd_perm, q),
    "resample": np.quantile(dd_boot, q)},
    index=["5%", "median", "95%"]).mul(100).round(1)

Drawdown of the actual sequence: -21.9%
Shuffled with a worse drawdown: 0.0%
        shuffle  resample
5%        -15.0     -15.3
median    -10.0     -10.1
95%        -7.4      -7.2

rng.permuted(..., axis=1) shuffles each row separately: 2,000 different orders of the same trades in a single call. The result is uncomfortable. With the trades in random order, the median drawdown is −10.0% and the 5th percentile one, −15.0%. The actual sequence had −21.9%: worse than every shuffled sequence.

It is not bad luck. Shuffling assumes the trades are interchangeable, and in a 24-market portfolio they are not: they overlap in time, and when the market turns (the end of a trend, a crisis) several markets lose at once. The next cell measures it:

eq = np.cumprod(1 + x)
fondo = int(np.argmin(eq / np.maximum.accumulate(eq)))
pico = int(np.argmax(eq[:fondo + 1]))
print(f"Worst stretch: {ops['salida'][pico]:%Y-%m-%d} to "
      f"{ops['salida'][fondo]:%Y-%m-%d}, {fondo - pico} trades")

mes = ops["salida"].dt.to_period("M").to_numpy()
real = pd.Series(x).groupby(mes).sum().std()
barajado = np.mean([pd.Series(rng.permutation(x)).groupby(mes)
                    .sum().std() for _ in range(200)])
print(f"Standard deviation of the monthly result: actual {real:.2%}, "
      f"shuffled {barajado:.2%}")

Worst stretch: 2001-01-02 to 2003-01-16, 378 trades
Standard deviation of the monthly result: actual 3.30%, shuffled 2.52%

With the trades in their actual month, the monthly result has a standard deviation of 3.30%; with the same trades spread at random across the same months, 2.52%. Gains and losses cluster in time, and the worst stretch of the actual sequence, from January 2001 to January 2003, chains 378 trades.

With that caveat, resampling serves for a question the backtest does not answer: how much leverage the strategy can take. If ruin is defined as a drawdown of 20% or 40% (the point at which a manager would shut the strategy down or a client would pull their money), it is enough to multiply the trades by the leverage:

filas = {}
for apal in (1, 2, 3, 4):
    dd = dd_max(apal * rng.choice(x, size=(N, len(x))))
    filas[f"x{apal}"] = {
        "dd_median": np.median(dd),
        "P(dd < -20%)": (dd < -0.20).mean(),
        "P(dd < -40%)": (dd < -0.40).mean()}
pd.DataFrame(filas).T.round(3)

    dd_median  P(dd < -20%)  P(dd < -40%)
x1     -0.101         0.005         0.000
x2     -0.192         0.428         0.000
x3     -0.278         0.952         0.046
x4     -0.359         1.000         0.289

Without leverage, the probability of going through a 20% drawdown in 25 years is 0.5%; with twice the size, 43%; with three times, 95%. That of reaching 40% is zero with twice the size (none of 2,000 paths), 4.6% with three times and 29% with four times. The median drawdown grows somewhat less than proportionally (−10.1%, −19.2%, −27.8% and −35.9%), because with compounded capital each loss is applied to an already smaller capital; the probability of crossing a fixed threshold, on the other hand, shoots up. And all these figures inherit the optimistic bias of the previous cell: the actual sequence already had −21.9% without leverage.

Block bootstrap on daily returns

The bootstrap of daily returns avoids the overlap problem: each day already contains all the open positions, with their correlation. But Efron's classic bootstrap (drawing single days with replacement) assumes the days are independent, and the returns of a portfolio are not: their volatility comes in clusters (lesson 1.6.2). The solution is to resample blocks of consecutive days. Künsch (1989) proposed fixed-length blocks; the circular bootstrap joins the end of the series to the beginning so that every day has the same probability of appearing, and the stationary bootstrap of Politis and Romano (1994) uses blocks of random length, geometric with mean $b$:

With blocks of random length, the resampled series is stationary, which fixed blocks do not guarantee. The mechanism fits in a loop:

def indices_estacionario(n, largo, b, rng):
    """Indices of a stationary resample (Politis and Romano).

    Each step moves on to the next day with probability 1 - 1/b or
    jumps to a random day with probability 1/b: blocks of
    geometric length with mean b, wrapping around circularly."""
    idx = np.empty(largo, dtype=np.int64)
    idx[0] = rng.integers(n)
    salto = rng.random(largo) < 1 / b
    nuevo = rng.integers(n, size=largo)
    for t in range(1, largo):
        idx[t] = nuevo[t] if salto[t] else (idx[t - 1] + 1) % n
    return idx

rng = np.random.default_rng(11)
print(indices_estacionario(len(cartera), 16, 5, rng))

[ 841  842 3221 5917 5918 5275 3452 3453 3454 3455 3456 3457 3458 5179
 5180 5181]

You can see the blocks: 841-842, a single day, 5917-5918, another single day, seven consecutive days from 3452 and another three at the end, with a mean of 5 because b=5. The arch library has the efficient version (StationaryBootstrap, CircularBlockBootstrap) and the automatic block length selector of Politis and White (2004), with the correction by Patton, Politis and White (2009), which estimates the optimal $b$ from the autocorrelation of the series:

from arch.bootstrap import optimal_block_length

pd.concat({"r": optimal_block_length(cartera),
           "r²": optimal_block_length(cartera ** 2)}).round(1)

         stationary  circular
r  None         1.2       1.4
r² None        47.6      54.5

The returns have almost no autocorrelation, and the selector asks for blocks of 1 session. Their squares, which measure volatility, ask for 48 (stationary) or 55 (circular). The choice depends on which statistic you want: the mean depends on the returns; the drawdown, on the volatility clusters. The prudent thing is to look at several lengths:

from arch.bootstrap import StationaryBootstrap

def drawdown(x):
    eq = np.cumprod(1 + np.asarray(x))
    return (eq / np.maximum.accumulate(eq) - 1).min()

def stats(x):
    return np.array([sharpe(x), drawdown(x)])

print(f"Historical: Sharpe {sharpe(cartera):.2f}, "
      f"drawdown {drawdown(cartera):.1%}")
filas = {}
for b in (1, 5, 20, 75, 250):
    bs = StationaryBootstrap(b, cartera.to_numpy(), seed=3)
    v = bs.apply(stats, 1000)
    filas[b] = {"sharpe_2.5": np.quantile(v[:, 0], 0.025),
                "sharpe_97.5": np.quantile(v[:, 0], 0.975),
                "dd_median": np.median(v[:, 1]),
                "dd_5%": np.quantile(v[:, 1], 0.05)}
pd.DataFrame(filas).T.rename_axis("block").round(3)

Historical: Sharpe 1.00, drawdown -14.7%
       sharpe_2.5  sharpe_97.5  dd_median  dd_5%
block
1           0.591        1.391     -0.179 -0.264
5           0.613        1.374     -0.178 -0.270
20          0.630        1.380     -0.178 -0.265
75          0.600        1.399     -0.177 -0.262
250         0.661        1.326     -0.169 -0.235

Three readings. The first: the 95% interval of the Sharpe goes from 0.6 to 1.4 with any block. It resembles Lo's (2002)22Note 2 · Lo, A. W. (2002), “The Statistics of Sharpe Ratios,” Financial Analysts Journal 58(4), 36-52, doi:10.2469/faj.v58.n4.2453. approximation for i.i.d. returns, $\sigma(\widehat{SR}) \approx \sqrt{(1 + SR^2/2)/T}$ with $T$ in years: with 1.00 and 25 years, 0.24, and ±1.96 times that gives 0.52 to 1.48; the bootstrap comes out somewhat narrower. Twenty-five years of data are not enough to tell a Sharpe of 0.6 from one of 1.4.

The second: in this portfolio the drawdown barely depends on the block. The median goes from −17.9% with single days to −17.7% with 75-session blocks, and the 5th percentile one from −26.4% to −26.2%. The reason lies in the construction: each position is scaled by the inverse of its volatility, and that removes from the portfolio almost all the volatility clustering a long block would preserve. In an unscaled series it is different. The next cell repeats the test with the IDA index future bought and held:

ida = r["IDA"].loc["2001-01-02":].to_numpy()
b_ida = optimal_block_length(ida ** 2)["stationary"].iloc[0]
print(f"IDA: Politis and White block for r²: {b_ida:.0f}")
print(f"Historical drawdown: {drawdown(ida):.1%}")
for b in (1, 75):
    v = StationaryBootstrap(b, ida, seed=3).apply(
        lambda x: np.array([drawdown(x)]), 500)[:, 0]
    print(f"Block {b:>2}: median drawdown {np.median(v):.1%}, "
          f"5th percentile {np.quantile(v, 0.05):.1%}")

IDA: Politis and White block for r²: 156
Historical drawdown: -62.2%
Block  1: median drawdown -53.0%, 5th percentile -74.9%
Block 75: median drawdown -65.3%, 5th percentile -86.8%

In IDA the selector asks for blocks of 156 sessions for the squares (48 in the portfolio), and the median drawdown goes from −53.0% with single days to −65.3% with 75-session blocks; the 5th percentile, from −74.9% to −86.8%. Long blocks preserve the volatility clusters, and with them the chained falls; single days spread them out and the drawdown comes out optimistic: with them, the historical drawdown of IDA (−62.2%) would look worse than normal. The practical rule: the block matters for the drawdown to the extent that the series has volatility clusters, and the prudent thing is to look at several lengths.

The third: the historical drawdown of the portfolio (−14.7%) is milder than the bootstrap median with any block (between −16.9% and −17.9%). The history that happened was somewhat lucky in the order of its days: the risk worth expecting is that of the bootstrap, not that of the backtest. (The −21.9% of the trade sequence is not comparable: there each trade is booked in full on the day it exits.)

The equity fan

With the stationary bootstrap you can generate paths of any length. A thousand five-year paths with blocks of 75 sessions on average give the fan in the figure at the top:

rng = np.random.default_rng(5)
n, largo, b = len(cartera), 5 * 252, 75
caminos = np.array([
    cartera.to_numpy()[indices_estacionario(n, largo, b, rng)]
    for _ in range(1000)])
capital = np.cumprod(1 + caminos, axis=1)
bandas = pd.DataFrame(
    np.quantile(capital, [0.05, 0.25, 0.5, 0.75, 0.95], axis=0).T,
    columns=["p5", "p25", "p50", "p75", "p95"])
print(bandas.iloc[[251, 503, 755, 1007, 1259]]
      .set_axis([1, 2, 3, 4, 5]).rename_axis("year").round(2))
print(f"P(loss at 5 years): {(capital[:, -1] < 1).mean():.1%}")

        p5   p25   p50   p75   p95
year
1     0.94  1.02  1.09  1.16  1.25
2     0.96  1.10  1.20  1.30  1.45
3     1.01  1.17  1.29  1.44  1.67
4     1.07  1.26  1.41  1.59  1.88
5     1.14  1.36  1.54  1.77  2.10
P(loss at 5 years): 1.2%

The median reaches 1.54 in five years; the 25-75% band, from 1.36 to 1.77. The 5th percentile is the line that matters for deciding the size: it goes through 0.94 in the first year (a 6% loss), through 0.96 in the second, does not get back to 1 until the third (1.01) and at five years is at 1.14. Even with a Sharpe of 1.00, 1.2% of the paths lose money at five years and one in twenty spends two years in losses. That is what you should show anyone who is going to invest before they live through it.

Permuting the signal: does it get the timing right?

The bootstrap says how much the Sharpe varies; it does not say whether it differs from what a strategy with no skill would give. A permutation test builds that reference directly: the relationship between signal and returns is broken and the statistic is recomputed many times. The p-value is the fraction of permutations that match or exceed what was observed, with the usual correction so that it is never zero:

The delicate part is deciding what to permute. The question is whether the direction chosen at each moment adds anything, so only the sign is permuted; the volatility scaling and the returns stay in place. And there are two ways of permuting the sign: shuffling the sessions (which also breaks the persistence of the signal) or shifting it circularly by a random number of sessions (which preserves its runs, its turnover and its autocorrelation, and only changes its alignment with the market). White (2000) and Hansen (2005)33Note 3 · Hansen, P. R. (2005), “A Test for Superior Predictive Ability,” Journal of Business & Economic Statistics 23(4), 365-380, doi:10.1198/073500105000000063. extend the idea to the case of many rules at once.

def p_valor_perm(signo, escala, r, n_perm=1000, seed=0,
                 modo="circular"):
    """Observed Sharpe, p-value and null distribution.

    Only the DIRECTION (the sign) is permuted; the volatility
    scaling and the returns stay in place."""
    rng = np.random.default_rng(seed)
    s = signo.shift(1).to_numpy()
    e = escala.shift(1).to_numpy()
    rr = r.to_numpy()
    ok = ~(np.isnan(s).any(axis=1) | np.isnan(e).any(axis=1)
           | np.isnan(rr).any(axis=1))
    s, e, rr = s[ok], e[ok], rr[ok]
    obs = sharpe((s * e * rr).sum(axis=1))
    nula = np.empty(n_perm)
    for i in range(n_perm):
        if modo == "shuffle":
            sp = s[rng.permutation(len(s))]
        else:  # circular shift of at least one year
            sp = np.roll(s, rng.integers(252, len(s) - 252), axis=0)
        nula[i] = sharpe((sp * e * rr).sum(axis=1))
    p = (1 + (nula >= obs).sum()) / (1 + n_perm)
    return obs, p, nula

res, nulas = {}, {}
for nombre, args in {"trend": (signo, escala, r),
                     "twin": (signo_g, escala_g, r_g)}.items():
    for modo in ("shuffle", "circular"):
        obs, p, nula = p_valor_perm(*args, modo=modo)
        nulas[(nombre, modo)] = nula
        res[(nombre, modo)] = {"sharpe": obs, "p": p,
                               "null_mean": nula.mean(),
                               "null_95": np.quantile(nula, 0.95)}
pd.DataFrame(res).T.round(3)

                sharpe      p  null_mean  null_95
trend shuffle    0.999  0.001      0.202    0.519
      circular   0.999  0.001      0.048    0.552
twin  shuffle    0.026  0.536      0.039    0.376
      circular   0.026  0.248     -0.066    0.212

The method tells the two cases apart. In the futures with a planted effect, the observed Sharpe (1.00) beats every permutation, both shuffling and shifting: p = 0.001, the minimum possible with 1,000. In the memoryless twin, the 0.03 falls inside its null: p = 0.25 with the circular shift (0.54 shuffling).

Note the mean of the null: 0.20 when shuffling and 0.05 with the circular shift, not 0. Any permutation preserves the proportion of long and short sessions of each market, and with it the part of the result that comes from the average exposure:

largo = (signo.shift(1).loc[pnl.index] > 0).mean()
largo.sort_values().round(2).iloc[[0, 1, -2, -1]].to_dict()

{'MEA': 0.42, 'DIC': 0.43, 'IDB': 0.7, 'IDA': 0.71}

The portfolio is long 71% of the sessions in IDA and 70% in IDB, and only 42% in gold (MEA). The indices carry the equity premium written into the generator, so a permutation that keeps them long 70% of the time makes something with any order; the same happens with the part of the carry the rule collects. The test does not ask whether the strategy makes money, but whether the chosen timing adds anything on top of that exposure. A strategy that is always long in a bull market would make money and would not pass the test.

flowchart TB
  Q{What is the question?} -- risk --> D[Block bootstrap]
  Q -- timing? --> P[Permutation]
  Q -- right pick? --> S[DSR and PBO]
  D --> C{Losses together?}
  C -- yes --> B[Long blocks]
  C -- no --> T[Trades or i.i.d.]

Power: failing to reject is not having no effect

All 24 markets have the same planted effect. Market by market:

p_merc = pd.Series({
    m: p_valor_perm(signo[[m]], escala[[m]], r[[m]],
                    n_perm=500, seed=1)[1]
    for m in r.columns})
print(f"Markets with p < 0.05: {(p_merc < 0.05).sum()} of 24")
print(p_merc.sort_values().round(3).head(6).to_dict())

Markets with p < 0.05: 7 of 24
{'AGB': 0.002, 'BOD': 0.002, 'BOC': 0.01, 'AGC': 0.012, 'BOA': 0.034, 'AGA': 0.042}

Only 7 of the 24 pass at 5%, with agricultural commodities and bonds in the lead. With a true momentum Sharpe of around 0.15 per market and 25 years of data, the standard error of the Sharpe (≈ 0.2) is larger than the effect: the test almost never detects it in a single market. The portfolio detects it because it adds up 24 weak, weakly correlated effects. Discarding 17 markets as “not significant” would be discarding a real effect; the correct conclusion is that a single market does not have enough data.

And in the noise, how many false positives?

The README's control is the ruptura series of regimenes2.parquet after its break (2013-05-22): i.i.d. normal returns with zero mean, where no trend rule has an edge. A single noise series is not enough to know whether the test is well calibrated, so 200 simulated series like it are added:

reg = pd.read_parquet("regimenes2.parquet")
ru = (reg[reg["serie"] == "ruptura"].set_index("fecha")[["precio"]]
      .loc["2013-05-22":])          # after the break: i.i.d., mean 0
s_ru, e_ru, r_ru, _ = tendencia(ru, riesgo=0.10)
obs, p, _ = p_valor_perm(s_ru, e_ru, r_ru)
print(f"“ruptura” series since 2013: Sharpe {obs:.2f}, p = {p:.3f}")

rng = np.random.default_rng(99)
ps = []
for i in range(200):
    ruido = pd.DataFrame(
        np.cumprod(1 + 0.013 * rng.standard_normal(len(ru))),
        index=ru.index)
    ps.append(p_valor_perm(*tendencia(ruido, riesgo=0.10)[:3],
                           n_perm=200, seed=i)[1])
ps = np.array(ps)
print(f"Noise series with p < 0.05: {(ps < 0.05).mean():.1%}")
print(np.histogram(ps, bins=5, range=(0, 1))[0])

“ruptura” series since 2013: Sharpe 0.55, p = 0.032
Noise series with p < 0.05: 4.5%
[36 47 33 27 57]

The README series gives a Sharpe of 0.55 and p = 0.032: a false positive. There is no error in the test: in 200 noise series like it, 4.5% come out with p < 0.05, and the p-values are spread roughly uniformly between 0 and 1 (36, 47, 33, 27 and 57 per quintile). That is what a 5% level means: one in twenty controls comes out “significant.” That is why a single p-value below 0.05 proves nothing, and why a control that passes the test does not invalidate the method: it invalidates the habit of testing a single thing and believing the result.

What no resampling corrects

The three methods work on a strategy that has already been chosen. If that strategy is the best of many, they all inherit the bias of the choice. The ruido2_estrategias.parquet set has 1,000 strategies, 995 of them with no edge at all (the truth is in ruido2_verdad.csv):

rs = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")
sin_ventaja = verdad.index[verdad["sharpe_real"] == 0]
sr = rs[sin_ventaja].apply(sharpe)
mejor = sr.idxmax()
print(f"{len(sin_ventaja)} strategies with no edge; the best: "
      f"{mejor}, Sharpe {sr.max():.2f}")
bs = StationaryBootstrap(5, rs[mejor].to_numpy(), seed=2)
v = bs.apply(sharpe, 2000)[:, 0]
print(f"95% interval: {np.quantile(v, 0.025):.2f} to "
      f"{np.quantile(v, 0.975):.2f}")
print(f"Resamples with Sharpe <= 0: {(v <= 0).mean():.2%}")

995 strategies with no edge; the best: e0598, Sharpe 1.50
95% interval: 0.46 to 2.54
Resamples with Sharpe <= 0: 0.15%

The best of the 995 has a Sharpe of 1.50 over four years, a bootstrap interval from 0.46 to 2.54 and only 0.15% of resamples with a negative Sharpe. Everything points to an excellent strategy, and its true Sharpe is zero. The bootstrap answers its own question well (how much the Sharpe of this series varies) and cannot answer another one it was not asked: how many series were looked at before choosing it. With 995 trials and a standard error of the Sharpe of 0.5, a maximum of 1.5 is what you would expect. The correction is the Deflated Sharpe Ratio or the PBO of lesson 2.5.4, which need the number of tests, and the experiment journal of lesson 2.2.4, which is where that number gets written down.

In summary, the order of work with a candidate strategy:

1. Block bootstrap of the daily returns, with several block lengths: interval of the Sharpe, of the drawdown and a fan to decide the size.
2. Trade resampling only as a floor for the risk, checking where the actual sequence lands.
3. Signal permutation test (circular if the signal is persistent) to separate skill from exposure, also on a control with no effect.
4. Deflated Sharpe or PBO with the real number of tests (lesson 2.5.4); lesson 2.5.8 puts everything together in a checklist.

Exercises

1. Repeat the risk of ruin with the stationary bootstrap of the daily returns (blocks of 75) instead of the trades: probability of a 20% drawdown in 25 years with leverage 1 and 2. Compare it with the trades table.

rng = np.random.default_rng(21)
n = len(cartera)
caminos = np.array([cartera.to_numpy()[
    indices_estacionario(n, n, 75, rng)] for _ in range(500)])
for apal in (1, 2):
    dd = dd_max(apal * caminos)
    print(f"x{apal}: median drawdown {np.median(dd):.1%}, "
          f"P(dd < -20%) {(dd < -0.20).mean():.1%}")

x1: median drawdown -17.6%, P(dd < -20%) 29.8%
x2: median drawdown -33.0%, P(dd < -20%) 99.6%

2. Apply the circular permutation test to the same rule with a 63-session window instead of 252, on the futures and on the twin. Does the test detect an effect that the README plants at 252 sessions?

for nombre, precios in (("trend", px), ("twin", px_g)):
    s63, e63, r63, _ = tendencia(precios, L=63)
    obs, p, nula = p_valor_perm(s63, e63, r63, n_perm=500)
    print(f"{nombre}: Sharpe {obs:.2f}, p = {p:.3f}, "
          f"null mean {nula.mean():.2f}")

trend: Sharpe 0.63, p = 0.020, null mean 0.07
twin: Sharpe 0.00, p = 0.403, null mean -0.04

3. A strategy is long every day in an index that rises 8% a year. What p-value does the signal permutation test give, and why? What does that tell you about what the test measures?