Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO).

The Sharpe of the best of N

Lesson 2.5.1 showed the problem with numbers: among 1,000 strategies with no edge, the maximum Sharpe is around 2 even though none of them is worth anything. Lesson 2.5.2 saw the consequence in a walk-forward and 2.5.3, in the distribution of paths of a cross-validation. This lesson gives the tools to correct a Sharpe for luck, at two levels:

• For one strategy, how much evidence does its Sharpe give, given the length of the series and the shape of its distribution? That is the Probabilistic Sharpe Ratio (PSR).
• For the best of N, does it beat what you would expect from the best of N trials with no edge? That is the Deflated Sharpe Ratio (DSR). And, if you have the full matrix of trials, how often does the process of choosing the best one end up choosing one worse than the median out-of-sample? That is the probability of backtest overfitting (PBO).

flowchart TB
  S[Measured Sharpe] --> P[PSR: against 0]
  P --> D[DSR: against max]
  N[N trials] --> D
  M[T x N matrix] --> B[PBO via CSCV]
  D --> X[Decision]
  B --> X

All three are by David Bailey and Marcos López de Prado (with Borwein and Zhu for the PBO), and the formulas in this lesson follow their papers, linked in the resources.

The probabilistic Sharpe (PSR)

The Sharpe estimated with T observations has an error. With normal and independent returns, its standard error is approximately $\sqrt{(1 + \widehat{SR}^2/2)/T}$ in per-period units (Lo, 2002). With fat tails and skewness it is larger: a strategy that makes a little almost always and loses a lot every now and then (negative skewness, high kurtosis) can have a high Sharpe for years without that saying much. The PSR measures the probability that the true Sharpe exceeds a threshold $SR^*$, with $\widehat{\gamma}_3$ the skewness and $\widehat{\gamma}_4$ the kurtosis (3 for the normal) of the returns:

All the Sharpes in the formula are at the frequency of the data (daily here), not annualized.

import numpy as np
import pandas as pd
from scipy.stats import norm

est = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")

def psr(sr, sr_umbral, t, asim, curt):
    """Probabilistic Sharpe Ratio. Per-period Sharpes; `curt` is the
    kurtosis (3 for the normal), not the excess."""
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr**2)
    z = (sr - sr_umbral) * np.sqrt(t - 1) / den
    if isinstance(z, pd.Series):  # keeps the names
        return pd.Series(norm.cdf(z), index=z.index)
    return norm.cdf(z)

T = len(est)
sr = est.mean() / est.std()           # daily Sharpe
asim, curt = est.skew(), est.kurt() + 3
psr0 = psr(sr, 0, T, asim, curt)

pasan = psr0 > 0.95
reales = verdad["sharpe_real"] > 0
print(f"T = {T} sessions; PSR(0) > 0.95: {pasan.sum()} strategies, "
      f"{(pasan & reales).sum()} with a real edge")

T = 1003 sessions; PSR(0) > 0.95: 57 strategies, 4 with a real edge

Tested one by one against a Sharpe of 0, more than fifty strategies seem to have an edge with 95% probability, and only four have one. That is to be expected: with 1,000 tests at 5%, about 50 false positives (lesson 2.1.8 covers multiple testing corrections). The PSR is not the problem: it is answering the question for an isolated strategy, and here there are a thousand.

ruido = reales.index[~reales.to_numpy()]
mejores_ruido = sr[ruido].nlargest(2).index
elegidas = list(reales.index[reales.to_numpy()]) + list(mejores_ruido)
tabla = pd.DataFrame({
    "annual_sharpe": sr[elegidas] * np.sqrt(252),
    "skew": asim[elegidas], "kurtosis": curt[elegidas],
    "PSR(0)": psr0[elegidas],
    "real": verdad.loc[elegidas, "sharpe_real"]})
tabla.round(3)

            annual_sharpe   skew  kurtosis  PSR(0)  real
estrategia
e0080               0.951  0.234     4.657   0.972   0.5
e0133               1.459 -0.284     7.113   0.998   0.8
e0258               0.622 -0.096     4.295   0.892   0.7
e0518               1.579  0.252     7.003   0.999   1.5
e0751               0.832 -0.133     4.008   0.951   0.6
e0598               1.503 -0.066     4.803   0.999   0.0
e0316               1.423  0.309     6.503   0.998   0.0

The two best of the noise, e0598 and e0316, have a measured Sharpe of 1.50 and 1.42 and a PSR(0) of 0.999 and 0.998: the same as e0133 and e0518, which do have an edge. By the Sharpe and its shape, nothing tells them apart. And two of the five good ones, e0258 and e0751, have a PSR(0) that grazes or falls short of 0.95: with 1,003 sessions, a true edge of 0.6-0.7 annual gets confused with noise. The kurtosis, from 4 to 7 in these series (with t innovations with 5 degrees of freedom), enlarges the denominator, although with daily Sharpes of the order of 0.1 its effect is small.

The Deflated Sharpe Ratio (DSR)

The right question for the best of N is not whether it beats 0, but whether it beats the maximum that N trials with no edge would give. The DSR is the PSR with the threshold at that expected maximum. Bailey and López de Prado (2014) approximate it with the dispersion of the Sharpes of the trials, $V[\{\widehat{SR}_n\}]$, and the same extreme value expression as in lesson 2.5.1:

with $\gamma \approx 0.5772$, the Euler-Mascheroni constant. The variance is that of the measured Sharpes of all the trials, not a theoretical one: if the trials are similar (highly correlated variants), their dispersion is smaller and the threshold goes down.

def sr_max_esperado(n, var_sr):
    g = np.euler_gamma
    return np.sqrt(var_sr) * ((1 - g) * norm.ppf(1 - 1 / n)
                              + g * norm.ppf(1 - 1 / (n * np.e)))

def dsr(sr_i, n, var_sr, t, asim_i, curt_i):
    umbral = sr_max_esperado(n, var_sr) if n > 1 else 0.0
    return psr(sr_i, umbral, t, asim_i, curt_i)

var_sr = sr.var()
print(f"Dispersion of the 1,000 Sharpes: {np.sqrt(var_sr * 252):.3f} "
      f"annual (theoretical with no edge: {np.sqrt(252 / T):.3f})")
for n in (10, 50, 1000):
    print(f"N = {n:>4}: SR_0 = "
          f"{sr_max_esperado(n, var_sr) * np.sqrt(252):.2f} annual")

tabla["DSR(1000)"] = dsr(sr[elegidas], 1000, var_sr, T,
                         asim[elegidas], curt[elegidas])
tabla["DSR(50)"] = dsr(sr[elegidas], 50, var_sr, T,
                       asim[elegidas], curt[elegidas])
cols = ["annual_sharpe", "PSR(0)", "DSR(50)", "DSR(1000)", "real"]
tabla[cols].round(3)

Dispersion of the 1,000 Sharpes: 0.486 annual (theoretical with no edge: 0.501)
N =   10: SR_0 = 0.77 annual
N =   50: SR_0 = 1.11 annual
N = 1000: SR_0 = 1.58 annual
            annual_sharpe  PSR(0)  DSR(50)  DSR(1000)  real
estrategia
e0080               0.951   0.972    0.377      0.102   0.5
e0133               1.459   0.998    0.754      0.404   0.8
e0258               0.622   0.892    0.167      0.028   0.7
e0518               1.579   0.999    0.828      0.497   1.5
e0751               0.832   0.951    0.293      0.068   0.6
e0598               1.503   0.999    0.783      0.437   0.0
e0316               1.423   0.998    0.737      0.373   0.0

The measured dispersion of the 1,000 Sharpes, 0.49 annual, almost matches the theoretical one of a Sharpe with no edge over 1,003 sessions: as it should, because almost all of them are noise. With N = 1,000, the expected maximum of the noise is an annual Sharpe of 1.58, and no strategy beats it with 95% probability: e0518, the truly best one, stays at 0.50, and e0598, the best noise, at 0.44. The DSR does not separate them because the data does not separate them: with four years of data, a true Sharpe of 1.5 cannot be told apart from the maximum of a thousand noises. That is the honest conclusion, and it is the one the DSR would give with real data.

The figure puts the three probabilities of each strategy on the same scale. Against 0 (squares) almost all pass; with 50 trials (diamonds) none reaches 0.95; with 1,000 (circles), the best stays at 0.50. And the order of the rows, by measured Sharpe, mixes good ones and noise: the best noise is the second row.

The figure at the top shows the same calculation as a function of N: the DSR of the two strategies falls as the number of declared trials increases.

ns = [1, 2, 3, 5, 7, 10, 15, 20, 30, 50, 70, 100, 150, 200, 300,
      500, 700, 1000]
curvas = pd.DataFrame({e: [dsr(sr[e], n, var_sr, T, asim[e], curt[e])
                           for n in ns] for e in ("e0518", "e0598")},
                      index=pd.Index(ns, name="N"))
print(curvas.loc[[1, 5, 10, 20, 50, 100, 1000]].round(3))
for e in curvas:
    ultimo = curvas.index[curvas[e] >= 0.95].max()
    print(f"{e}: DSR ≥ 0.95 up to N = {ultimo}")

      e0518  e0598
N
1     0.999  0.999
5     0.977  0.966
10    0.948  0.928
20    0.905  0.874
50    0.828  0.783
100   0.757  0.705
1000  0.497  0.437
e0518: DSR ≥ 0.95 up to N = 7
e0598: DSR ≥ 0.95 up to N = 5

If e0518 had been the only idea tried, its Sharpe would be conclusive (DSR 0.999). Declaring ten trials, the DSR is already close to 0.95; with twenty, it falls short. The best noise has almost the same curve, a little lower. The practical lesson: the number of trials is part of the result, and declaring fewer than were made is a form of fraud, even if unintentional.

autoval = np.linalg.eigvalsh(est.corr().to_numpy())
n_ef = autoval.sum() ** 2 / (autoval ** 2).sum()
print(f"Participation ratio: {n_ef:.0f} effective trials")
d = dsr(sr[["e0518", "e0598"]], round(n_ef), var_sr, T,
        asim[["e0518", "e0598"]], curt[["e0518", "e0598"]])
print(d.round(3).to_string())

Participation ratio: 147 effective trials
estrategia
e0518    0.715
e0598    0.660

With 147 effective trials, the DSR of e0518 is 0.72 and that of e0598, 0.66: between the two bounds and far from 0.95.

The probability of backtest overfitting (PBO)

The DSR needs N and the dispersion of the trials. If the returns of all the trials were also saved (a matrix of T sessions by N variants), you can directly measure what happens when choosing the best one. That is the idea of combinatorially symmetric cross-validation (CSCV) by Bailey, Borwein, López de Prado and Zhu:

1. Split the matrix into S contiguous blocks of rows (S even; 16 here).
2. For each of the $\binom{S}{S/2}$ ways of choosing half of the blocks (12,870 with S = 16): that half is the “in-sample” set and the other, the “out-of-sample” set.
3. Choose the variant with the best in-sample Sharpe and look at its out-of-sample relative rank, $\overline{\omega}_c$ (1 if it is the best of N, close to 0 if it is the worst), and its logit, $\lambda_c = \ln\left(\overline{\omega}_c / (1 - \overline{\omega}_c)\right)$.
4. The PBO is the proportion of partitions with $\lambda_c \le 0$: the chosen one lands in the bad half out-of-sample.

flowchart TB
  M[T x N matrix] --> B[S blocks]
  B --> C[Partitions]
  C --> E[Best in-sample]
  E --> R[Out-of-sample rank]
  R --> L[Logit]
  L --> P[PBO: logit ≤ 0]

If the choice adds nothing, the out-of-sample rank of the chosen one is uniform and the PBO, 0.5. If the choice gets it right (the best in-sample tends to be the best out-of-sample), the logits are positive and the PBO, low. The partition is symmetric: in-sample and out-of-sample have the same size and each block is on each side half of the time.

from itertools import combinations

def pbo_cscv(m, s=16):
    """m: T x N matrix of returns. Returns the PBO, the logits and,
    per partition, the in- and out-of-sample Sharpe of the chosen one."""
    t, n = m.shape
    bloques = np.array_split(np.arange(t), s)
    suma = np.array([m[b].sum(0) for b in bloques])
    suma2 = np.array([(m[b] ** 2).sum(0) for b in bloques])
    filas = np.array([len(b) for b in bloques])

    def sharpe(idx):
        k = filas[idx].sum()
        media = suma[idx].sum(0) / k
        return media / np.sqrt(suma2[idx].sum(0) / k - media ** 2)

    logits, sh_dentro, sh_fuera = [], [], []
    for dentro in combinations(range(s), s // 2):
        dentro = list(dentro)
        fuera = [i for i in range(s) if i not in dentro]
        a, b = sharpe(dentro), sharpe(fuera)
        j = np.argmax(a)
        w = (b < b[j]).sum() + 1        # rank from 1 (worst) to n
        w = w / (n + 1)
        logits.append(np.log(w / (1 - w)))
        sh_dentro.append(a[j]); sh_fuera.append(b[j])
    logits = np.array(logits)
    return ((logits <= 0).mean(), logits,
            np.array(sh_dentro) * np.sqrt(252),
            np.array(sh_fuera) * np.sqrt(252))

pbo, logits, sh_d, sh_f = pbo_cscv(est.to_numpy())
print(f"Partitions: {len(logits)}; PBO = {pbo:.3f}")
print(f"Sharpe of the chosen one: IS {sh_d.mean():.2f}, "
      f"OOS {sh_f.mean():.2f}; "
      f"P(OOS < 0) = {(sh_f < 0).mean():.2f}")

Partitions: 12870; PBO = 0.398
Sharpe of the chosen one: IS 2.36, OOS 0.14; P(OOS < 0) = 0.37

The PBO of the 1,000 strategies is 0.40: in four out of ten partitions, the best in-sample falls in the bad half out-of-sample. The mean Sharpe of the chosen one goes from 2.36 in-sample to 0.14 out-of-sample (the degradation), and in a little over a third of the partitions it loses money out-of-sample. Without the five strategies with an edge, the PBO gets close to 0.5:

solo_ruido = est[ruido].to_numpy()
pbo_r, _, d_r, f_r = pbo_cscv(solo_ruido)
print(f"995 strategies with no edge: PBO = {pbo_r:.3f}; "
      f"Sharpe IS {d_r.mean():.2f}, OOS {f_r.mean():.2f}")

995 strategies with no edge: PBO = 0.455; Sharpe IS 2.31, OOS 0.05

The five good ones lower the PBO by only about six points because, among a thousand, the best in-sample is usually noise: it is the same conclusion as the DSR, by another route. The figure shows, at the top, the logits of the 12,870 partitions of the full matrix: a bell centered slightly above 0, with a long left tail and 40% of the mass below 0, which is what the PBO measures. With a choice that got it right, the bell would be shifted to the right. The bottom panel is the case of the next section.

Reading the PBO carefully

The PBO answers a specific question: does choosing the best in-sample give something better than the median out-of-sample? It does not say whether the median makes money. The eight futures momentum variants of lesson 2.5.2 show it:

fut = pd.read_parquet("futuros2_continuos.parquet")
precio = fut.pivot(index="fecha", columns="mercado",
                   values="cierre_ajustado")
rf = precio.pct_change(fill_method=None)
vol = rf.ewm(span=60, min_periods=60).std() * np.sqrt(252)
mom = {}
for n in (10, 21, 42, 63, 126, 189, 252, 378):
    senal = np.sign(precio.pct_change(n, fill_method=None))
    peso = (senal * 0.40 / vol / precio.shape[1]).shift(1)
    mom[f"mom{n}"] = (peso * rf).sum(axis=1, min_count=1)
mom = pd.DataFrame(mom).loc["2002":]
mezcla = pd.concat([mom, (-mom).add_prefix("rev_")], axis=1)

filas = {}
for nombre, m in (("8 momentum", mom),
                  ("8 + 8 reversal", mezcla)):
    p, _, d, f = pbo_cscv(m.to_numpy())
    filas[nombre] = {"PBO": p, "IS": d.mean(), "OOS": f.mean(),
                     "P(OOS < 0)": (f < 0).mean()}
pd.DataFrame(filas).T.round(3)

                  PBO     IS    OOS  P(OOS < 0)
8 momentum      0.127  1.074  0.928       0.002
8 + 8 reversal  0.002  1.074  0.928       0.002

With eight momentum variants, all with an edge and highly correlated, the PBO is 0.13: in most partitions the chosen one also lands in the good half out-of-sample. The choice gets it right because in this data there is a true parameter (252) with a clear peak (lesson 2.5.2) and its neighbors, 189 and 378, are also good. The degradation is small, from 1.07 to 0.93, and the chosen one loses money out-of-sample in 0.2% of the partitions. That is the bottom panel of the logits figure: there are only eight possible rank values, and more than half of the partitions fall on the highest.

Now add eight reversal variants (the same ones with the sign flipped, all losers). The PBO drops to 0.002, but the chosen strategy is the same and so is its out-of-sample Sharpe: the choice “gets it right” more often because discarding obvious losers is easy. The PBO of a set of variants depends on which variants are put into it. With good and almost interchangeable variants, the opposite case, the PBO would rise toward 0.5 even if the family made money out-of-sample: which one comes first would be chance, and the choice would add nothing, but it would do no harm either.

The rule: the PBO, degradation and the probability of loss are read together, and the set of variants is fixed before looking. A low PBO can be manufactured by adding bad variants; a high PBO with rare losses says that what works is the family, not the parameter.

What to record and how to report

The three tools need data that only exists if it was saved during research (lesson 2.2.4):

Three limitations to keep in mind. The DSR depends on N, and N is hard to count honestly: it includes ideas half discarded, variants tried by colleagues and those from the literature that inspired the strategy. The approximation of the expected maximum assumes independent trials; with correlated trials it is conservative. And the PBO, like CPCV, mixes in-sample and out-of-sample without respecting the arrow of time: it assumes the process is stable throughout the sample.

Exercises

1. Compute the PSR of e0518 against an annual Sharpe threshold of 1 (a minimum useful value, not 0). What is the probability that its true Sharpe exceeds 1?

umbral = 1 / np.sqrt(252)      # at daily frequency
p1 = psr(sr["e0518"], umbral, T, asim["e0518"], curt["e0518"])
print(f"PSR(1 annual) of e0518: {p1:.3f}")

PSR(1 annual) of e0518: 0.877

2. Repeat the PBO of the 1,000 strategies with S = 8 and S = 12 blocks. Is the result stable?

for s in (8, 12, 16):
    p, lg, _, f = pbo_cscv(est.to_numpy(), s=s)
    print(f"S = {s:>2}: {len(lg):>5} partitions, PBO = {p:.3f}, "
          f"P(OOS < 0) = {(f < 0).mean():.2f}")

S =  8:    70 partitions, PBO = 0.371, P(OOS < 0) = 0.36
S = 12:   924 partitions, PBO = 0.425, P(OOS < 0) = 0.39
S = 16: 12870 partitions, PBO = 0.398, P(OOS < 0) = 0.37

3. A fund presents you with a strategy with a Sharpe of 2.1 over six years of daily data and says it is “the only one they tried.” Its team has four researchers who have been working on variants for two years. What N would you use for the DSR, and what information would you ask for?