---
title: "2.5.4 Deflated Sharpe Ratio and probability of overfitting"
description: "PSR, Deflated Sharpe and PBO coded from scratch and applied to 1,000 synthetic strategies: which Sharpe survives the number of trials and fat tails."
url: "https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo.png"
---

# 2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)

> A Sharpe of 1.5 does not mean the same if it is the only one tried as if it is the best of a thousand. This lesson codes Bailey and López de Prado's three tools for discounting luck: the probabilistic Sharpe, the Deflated Sharpe and the probability of backtest overfitting, and tests them where the answer is known.

- Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/)
- Module: 2.5 · Biases and validation
- Duration: 70 min
- Prerequisites: [2.5.1 · Backtest biases: look-ahead, survivorship, selection and narrative](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-biases/), [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/), [0.7.2 · Probability and distributions, correlation versus causation and simple regression](https://pythonparatrading.com/en/python-trading-course/level-0/probability-distributions-and-regression/)

## By the end of this lesson

- Compute the Probabilistic Sharpe Ratio (PSR) of a return series, with its skewness, its kurtosis and its length.
- Compute the Deflated Sharpe Ratio (DSR): the PSR against the expected maximum Sharpe of N trials with no edge, and see how it changes with N.
- Estimate the probability of backtest overfitting (PBO) with combinatorially symmetric cross-validation (CSCV) on a matrix of variants.
- Interpret the PBO together with the out-of-sample degradation and probability of loss, and know what has to be recorded during research to be able to compute them.

## The Sharpe of the best of N

Lesson 2.5.1 showed the problem with numbers: among 1,000 strategies with no edge, the maximum Sharpe is around 2 even
though none of them is worth anything. Lesson 2.5.2 saw the consequence in a walk-forward and 2.5.3, in the
distribution of paths of a cross-validation. This lesson gives the tools to **correct** a Sharpe for luck, at two
levels:

- For **one** strategy, how much evidence does its Sharpe give, given the length of the series and the shape of its
  distribution? That is the **Probabilistic Sharpe Ratio** (PSR).
- For the **best of N**, does it beat what you would expect from the best of N trials with no edge? That is the
  **Deflated Sharpe Ratio** (DSR). And, if you have the full matrix of trials, how often does the process of choosing
  the best one end up choosing one worse than the median out-of-sample? That is the **probability of backtest
  overfitting** (PBO).

```mermaid
flowchart TB
  S[Measured Sharpe] --> P[PSR: against 0]
  P --> D[DSR: against max]
  N[N trials] --> D
  M[T x N matrix] --> B[PBO via CSCV]
  D --> X[Decision]
  B --> X
```

All three are by David Bailey and Marcos López de Prado (with Borwein and Zhu for the PBO), and the formulas in this
lesson follow their papers, linked in the resources.

<aside class="callout" data-kind="aviso">

The data is that of `ruido2_estrategias.parquet`: daily returns of 1,000 fictitious strategies over the 1,003
sessions from 2022 to 2025, in 50 families of 20 variants with a correlation of 0.5 within each family. **All have
zero edge except five**, whose true Sharpe (from 0.5 to 1.5) is in `ruido2_verdad.csv`. They are synthetic and the
answer is planted: they serve to see what each tool detects, not to look for strategies.

</aside>

## The probabilistic Sharpe (PSR)

The Sharpe estimated with T observations has an error. With normal and independent returns, its standard error is
approximately $\sqrt{(1 + \widehat{SR}^2/2)/T}$ in per-period units (Lo, 2002). With fat tails and skewness it is
larger: a strategy that makes a little almost always and loses a lot every now and then (negative skewness, high
kurtosis) can have a high Sharpe for years without that saying much. The PSR measures the probability that the true
Sharpe exceeds a threshold $SR^*$, with $\widehat{\gamma}_3$ the skewness and $\widehat{\gamma}_4$ the kurtosis (3 for
the normal) of the returns:

$$
\widehat{PSR}(SR^*) = \Phi\left(\frac{(\widehat{SR} - SR^*)\sqrt{T-1}}{\sqrt{1 - \widehat{\gamma}_3\,\widehat{SR} + \frac{\widehat{\gamma}_4 - 1}{4}\,\widehat{SR}^2}}\right)
$$

All the Sharpes in the formula are **at the frequency of the data** (daily here), not annualized.

```python
import numpy as np
import pandas as pd
from scipy.stats import norm

est = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")

def psr(sr, sr_umbral, t, asim, curt):
    """Probabilistic Sharpe Ratio. Per-period Sharpes; `curt` is the
    kurtosis (3 for the normal), not the excess."""
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr**2)
    z = (sr - sr_umbral) * np.sqrt(t - 1) / den
    if isinstance(z, pd.Series):  # keeps the names
        return pd.Series(norm.cdf(z), index=z.index)
    return norm.cdf(z)

T = len(est)
sr = est.mean() / est.std()           # daily Sharpe
asim, curt = est.skew(), est.kurt() + 3
psr0 = psr(sr, 0, T, asim, curt)

pasan = psr0 > 0.95
reales = verdad["sharpe_real"] > 0
print(f"T = {T} sessions; PSR(0) > 0.95: {pasan.sum()} strategies, "
      f"{(pasan & reales).sum()} with a real edge")
```
```output
T = 1003 sessions; PSR(0) > 0.95: 57 strategies, 4 with a real edge
```

Tested one by one against a Sharpe of 0, more than fifty strategies seem to have an edge with 95% probability, and
only four have one. That is to be expected: with 1,000 tests at 5%, about 50 false positives (lesson 2.1.8 covers
multiple testing corrections). The PSR is not the problem: it is answering the question for an isolated strategy, and
here there are a thousand.

```python
ruido = reales.index[~reales.to_numpy()]
mejores_ruido = sr[ruido].nlargest(2).index
elegidas = list(reales.index[reales.to_numpy()]) + list(mejores_ruido)
tabla = pd.DataFrame({
    "annual_sharpe": sr[elegidas] * np.sqrt(252),
    "skew": asim[elegidas], "kurtosis": curt[elegidas],
    "PSR(0)": psr0[elegidas],
    "real": verdad.loc[elegidas, "sharpe_real"]})
tabla.round(3)
```
```output
            annual_sharpe   skew  kurtosis  PSR(0)  real
estrategia
e0080               0.951  0.234     4.657   0.972   0.5
e0133               1.459 -0.284     7.113   0.998   0.8
e0258               0.622 -0.096     4.295   0.892   0.7
e0518               1.579  0.252     7.003   0.999   1.5
e0751               0.832 -0.133     4.008   0.951   0.6
e0598               1.503 -0.066     4.803   0.999   0.0
e0316               1.423  0.309     6.503   0.998   0.0
```

The two best of the noise, e0598 and e0316, have a measured Sharpe of 1.50 and 1.42 and a PSR(0) of 0.999 and 0.998:
the same as e0133 and e0518, which do have an edge. By the Sharpe and its shape, nothing tells them apart. And two of
the five good ones, e0258 and e0751, have a PSR(0) that grazes or falls short of 0.95: with 1,003 sessions, a true
edge of 0.6-0.7 annual gets confused with noise. The kurtosis, from 4 to 7 in these series (with t innovations with 5
degrees of freedom), enlarges the denominator, although with daily Sharpes of the order of 0.1 its effect is small.

## The Deflated Sharpe Ratio (DSR)

The right question for the best of N is not whether it beats 0, but whether it beats **the maximum that N trials with
no edge would give**. The DSR is the PSR with the threshold at that expected maximum. Bailey and López de Prado (2014)
approximate it with the dispersion of the Sharpes of the trials, $V[\{\widehat{SR}_n\}]$, and the same extreme value
expression as in lesson 2.5.1:

$$
SR_0 = \sqrt{V[\{\widehat{SR}_n\}]}\left[(1-\gamma)\,\Phi^{-1}\!\left(1-\tfrac{1}{N}\right) + \gamma\,\Phi^{-1}\!\left(1-\tfrac{1}{N e}\right)\right],
\qquad DSR = \widehat{PSR}(SR_0)
$$

with $\gamma \approx 0.5772$, the Euler-Mascheroni constant. The variance is that of the **measured** Sharpes of all
the trials, not a theoretical one: if the trials are similar (highly correlated variants), their dispersion is
smaller and the threshold goes down.

```python
def sr_max_esperado(n, var_sr):
    g = np.euler_gamma
    return np.sqrt(var_sr) * ((1 - g) * norm.ppf(1 - 1 / n)
                              + g * norm.ppf(1 - 1 / (n * np.e)))

def dsr(sr_i, n, var_sr, t, asim_i, curt_i):
    umbral = sr_max_esperado(n, var_sr) if n > 1 else 0.0
    return psr(sr_i, umbral, t, asim_i, curt_i)

var_sr = sr.var()
print(f"Dispersion of the 1,000 Sharpes: {np.sqrt(var_sr * 252):.3f} "
      f"annual (theoretical with no edge: {np.sqrt(252 / T):.3f})")
for n in (10, 50, 1000):
    print(f"N = {n:>4}: SR_0 = "
          f"{sr_max_esperado(n, var_sr) * np.sqrt(252):.2f} annual")

tabla["DSR(1000)"] = dsr(sr[elegidas], 1000, var_sr, T,
                         asim[elegidas], curt[elegidas])
tabla["DSR(50)"] = dsr(sr[elegidas], 50, var_sr, T,
                       asim[elegidas], curt[elegidas])
cols = ["annual_sharpe", "PSR(0)", "DSR(50)", "DSR(1000)", "real"]
tabla[cols].round(3)
```
```output
Dispersion of the 1,000 Sharpes: 0.486 annual (theoretical with no edge: 0.501)
N =   10: SR_0 = 0.77 annual
N =   50: SR_0 = 1.11 annual
N = 1000: SR_0 = 1.58 annual
            annual_sharpe  PSR(0)  DSR(50)  DSR(1000)  real
estrategia
e0080               0.951   0.972    0.377      0.102   0.5
e0133               1.459   0.998    0.754      0.404   0.8
e0258               0.622   0.892    0.167      0.028   0.7
e0518               1.579   0.999    0.828      0.497   1.5
e0751               0.832   0.951    0.293      0.068   0.6
e0598               1.503   0.999    0.783      0.437   0.0
e0316               1.423   0.998    0.737      0.373   0.0
```

The measured dispersion of the 1,000 Sharpes, 0.49 annual, almost matches the theoretical one of a Sharpe with no edge
over 1,003 sessions: as it should, because almost all of them are noise. With N = 1,000, the expected maximum of the
noise is an annual Sharpe of 1.58, and no strategy beats it with 95% probability: e0518, the truly best one, stays at
0.50, and e0598, the best noise, at 0.44. The DSR does not separate them because the data does not separate them: with
four years of data, a true Sharpe of 1.5 cannot be told apart from the maximum of a thousand noises. That is the
honest conclusion, and it is the one the DSR would give with real data.

*Fig. — PSR against 0 and Deflated Sharpe with 50 and 1,000 trials for the five synthetic strategies with an edge and the two best noise ones, with their true Sharpe.* Seven rows ordered by measured Sharpe, each with three joined points: PSR(0), DSR with N = 50 and DSR with N = 1,000. e0518 (true 1.5): 0.999, 0.83 and 0.50. e0598 (noise, highlighted): 0.999, 0.78 and 0.44. e0133 (true 0.8): 0.998, 0.75 and 0.40. e0316 (noise): 0.998, 0.74 and 0.37. e0080, e0751 and e0258 (true 0.5, 0.6 and 0.7) have PSR of 0.97, 0.95 and 0.89 and DSR with 1,000 trials below 0.11. None reaches 0.95 on the DSR.

The figure puts the three probabilities of each strategy on the same scale. Against 0 (squares) almost all pass;
with 50 trials (diamonds) none reaches 0.95; with 1,000 (circles), the best stays at 0.50. And the order of the rows,
by measured Sharpe, mixes good ones and noise: the best noise is the second row.

The figure at the top shows the same calculation as a function of N: the DSR of the two strategies falls as the
number of declared trials increases.

```python
ns = [1, 2, 3, 5, 7, 10, 15, 20, 30, 50, 70, 100, 150, 200, 300,
      500, 700, 1000]
curvas = pd.DataFrame({e: [dsr(sr[e], n, var_sr, T, asim[e], curt[e])
                           for n in ns] for e in ("e0518", "e0598")},
                      index=pd.Index(ns, name="N"))
print(curvas.loc[[1, 5, 10, 20, 50, 100, 1000]].round(3))
for e in curvas:
    ultimo = curvas.index[curvas[e] >= 0.95].max()
    print(f"{e}: DSR ≥ 0.95 up to N = {ultimo}")
```
```output
      e0518  e0598
N
1     0.999  0.999
5     0.977  0.966
10    0.948  0.928
20    0.905  0.874
50    0.828  0.783
100   0.757  0.705
1000  0.497  0.437
e0518: DSR ≥ 0.95 up to N = 7
e0598: DSR ≥ 0.95 up to N = 5
```

If e0518 had been the only idea tried, its Sharpe would be conclusive (DSR 0.999). Declaring ten trials, the DSR is
already close to 0.95; with twenty, it falls short. The best noise has almost the same curve, a little lower. The
practical lesson: the number of trials **is** part of the result, and declaring fewer than were made is a form of
fraud, even if unintentional.

<aside class="callout" data-kind="nota">

How many “effective” trials are 1,000 strategies in 50 families of 20 correlated variants? Fewer than 1,000 and more
than 50: the variants of a family are not independent, but they are not identical either. One option is to group the
trials by correlation and count the groups; another simple heuristic is the participation ratio of the eigenvalues
of the correlation matrix, $(\sum \lambda)^2 / \sum \lambda^2$. In any case, DSR(50) and DSR(1000) bound the result,
and none of the strategies passes with either of them.

</aside>

```python
autoval = np.linalg.eigvalsh(est.corr().to_numpy())
n_ef = autoval.sum() ** 2 / (autoval ** 2).sum()
print(f"Participation ratio: {n_ef:.0f} effective trials")
d = dsr(sr[["e0518", "e0598"]], round(n_ef), var_sr, T,
        asim[["e0518", "e0598"]], curt[["e0518", "e0598"]])
print(d.round(3).to_string())
```
```output
Participation ratio: 147 effective trials
estrategia
e0518    0.715
e0598    0.660
```

With 147 effective trials, the DSR of e0518 is 0.72 and that of e0598, 0.66: between the two bounds and far from 0.95.

## The probability of backtest overfitting (PBO)

The DSR needs N and the dispersion of the trials. If the **returns** of all the trials were also saved (a matrix of
T sessions by N variants), you can directly measure what happens when choosing the best one. That is the idea of
combinatorially symmetric cross-validation (CSCV) by Bailey, Borwein, López de Prado and Zhu:

1. Split the matrix into S contiguous blocks of rows (S even; 16 here).
2. For each of the $\binom{S}{S/2}$ ways of choosing half of the blocks (12,870 with S = 16): that half is the
   “in-sample” set and the other, the “out-of-sample” set.
3. Choose the variant with the best in-sample Sharpe and look at its out-of-sample **relative rank**,
   $\overline{\omega}_c$ (1 if it is the best of N, close to 0 if it is the worst), and its logit,
   $\lambda_c = \ln\left(\overline{\omega}_c / (1 - \overline{\omega}_c)\right)$.
4. The PBO is the proportion of partitions with $\lambda_c \le 0$: the chosen one lands in the bad half
   out-of-sample.

```mermaid
flowchart TB
  M[T x N matrix] --> B[S blocks]
  B --> C[Partitions]
  C --> E[Best in-sample]
  E --> R[Out-of-sample rank]
  R --> L[Logit]
  L --> P[PBO: logit ≤ 0]
```

If the choice adds nothing, the out-of-sample rank of the chosen one is uniform and the PBO, 0.5. If the choice gets
it right (the best in-sample tends to be the best out-of-sample), the logits are positive and the PBO, low. The
partition is symmetric: in-sample and out-of-sample have the same size and each block is on each side half of the
time.

```python
from itertools import combinations

def pbo_cscv(m, s=16):
    """m: T x N matrix of returns. Returns the PBO, the logits and,
    per partition, the in- and out-of-sample Sharpe of the chosen one."""
    t, n = m.shape
    bloques = np.array_split(np.arange(t), s)
    suma = np.array([m[b].sum(0) for b in bloques])
    suma2 = np.array([(m[b] ** 2).sum(0) for b in bloques])
    filas = np.array([len(b) for b in bloques])

    def sharpe(idx):
        k = filas[idx].sum()
        media = suma[idx].sum(0) / k
        return media / np.sqrt(suma2[idx].sum(0) / k - media ** 2)

    logits, sh_dentro, sh_fuera = [], [], []
    for dentro in combinations(range(s), s // 2):
        dentro = list(dentro)
        fuera = [i for i in range(s) if i not in dentro]
        a, b = sharpe(dentro), sharpe(fuera)
        j = np.argmax(a)
        w = (b < b[j]).sum() + 1        # rank from 1 (worst) to n
        w = w / (n + 1)
        logits.append(np.log(w / (1 - w)))
        sh_dentro.append(a[j]); sh_fuera.append(b[j])
    logits = np.array(logits)
    return ((logits <= 0).mean(), logits,
            np.array(sh_dentro) * np.sqrt(252),
            np.array(sh_fuera) * np.sqrt(252))

pbo, logits, sh_d, sh_f = pbo_cscv(est.to_numpy())
print(f"Partitions: {len(logits)}; PBO = {pbo:.3f}")
print(f"Sharpe of the chosen one: IS {sh_d.mean():.2f}, "
      f"OOS {sh_f.mean():.2f}; "
      f"P(OOS < 0) = {(sh_f < 0).mean():.2f}")
```
```output
Partitions: 12870; PBO = 0.398
Sharpe of the chosen one: IS 2.36, OOS 0.14; P(OOS < 0) = 0.37
```

*Fig. — Distribution of the logit of the chosen variant's out-of-sample rank across the 12,870 CSCV partitions (S = 16): 1,000 synthetic strategies above, 8 futures momentum lookbacks below.* Two histograms with the zero line marked; the bars left of zero, highlighted, are the partitions where the chosen one lands in the bad half out of sample. Above, the 1,000 strategies: a bell centred slightly above 0, from −6.5 to 3.5, with 40% left of zero (PBO 0.40; the chosen one loses out of sample in 37%). Below, 8 momentum variants: 8 possible values that grow to the right, with more than half the partitions (6,816) at the highest one, PBO 0.13, and the chosen one loses out of sample in only 0.2%.

The PBO of the 1,000 strategies is 0.40: in four out of ten partitions, the best in-sample falls in the bad half
out-of-sample. The mean Sharpe of the chosen one goes from 2.36 in-sample to 0.14 out-of-sample (the
**degradation**), and in a little over a third of the partitions it loses money out-of-sample. Without the five
strategies with an edge, the PBO gets close to 0.5:

```python
solo_ruido = est[ruido].to_numpy()
pbo_r, _, d_r, f_r = pbo_cscv(solo_ruido)
print(f"995 strategies with no edge: PBO = {pbo_r:.3f}; "
      f"Sharpe IS {d_r.mean():.2f}, OOS {f_r.mean():.2f}")
```
```output
995 strategies with no edge: PBO = 0.455; Sharpe IS 2.31, OOS 0.05
```

The five good ones lower the PBO by only about six points because, among a thousand, the best in-sample is usually
noise: it is the same conclusion as the DSR, by another route. The figure shows, at the top, the logits of the 12,870
partitions of the full matrix: a bell centered slightly above 0, with a long left tail and 40% of the mass below 0,
which is what the PBO measures. With a choice that got it right, the bell would be shifted to the right. The bottom
panel is the case of the next section.

## Reading the PBO carefully

The PBO answers a specific question: does choosing the best in-sample give something better than the median
out-of-sample? It does not say whether the median makes money. The eight futures momentum variants of lesson 2.5.2
show it:

```python
fut = pd.read_parquet("futuros2_continuos.parquet")
precio = fut.pivot(index="fecha", columns="mercado",
                   values="cierre_ajustado")
rf = precio.pct_change(fill_method=None)
vol = rf.ewm(span=60, min_periods=60).std() * np.sqrt(252)
mom = {}
for n in (10, 21, 42, 63, 126, 189, 252, 378):
    senal = np.sign(precio.pct_change(n, fill_method=None))
    peso = (senal * 0.40 / vol / precio.shape[1]).shift(1)
    mom[f"mom{n}"] = (peso * rf).sum(axis=1, min_count=1)
mom = pd.DataFrame(mom).loc["2002":]
mezcla = pd.concat([mom, (-mom).add_prefix("rev_")], axis=1)

filas = {}
for nombre, m in (("8 momentum", mom),
                  ("8 + 8 reversal", mezcla)):
    p, _, d, f = pbo_cscv(m.to_numpy())
    filas[nombre] = {"PBO": p, "IS": d.mean(), "OOS": f.mean(),
                     "P(OOS < 0)": (f < 0).mean()}
pd.DataFrame(filas).T.round(3)
```
```output
                  PBO     IS    OOS  P(OOS < 0)
8 momentum      0.127  1.074  0.928       0.002
8 + 8 reversal  0.002  1.074  0.928       0.002
```

With eight momentum variants, all with an edge and highly correlated, the PBO is 0.13: in most partitions the chosen
one also lands in the good half out-of-sample. The choice gets it right because in this data there is a true parameter
(252) with a clear peak (lesson 2.5.2) and its neighbors, 189 and 378, are also good. The degradation is small, from
1.07 to 0.93, and the chosen one loses money out-of-sample in 0.2% of the partitions. That is the bottom panel of the
logits figure: there are only eight possible rank values, and more than half of the partitions fall on the highest.

Now add eight reversal variants (the same ones with the sign flipped, all losers). The PBO drops to 0.002, but the
chosen strategy is the same and so is its out-of-sample Sharpe: the choice “gets it right” more often because
discarding obvious losers is easy. The PBO of a set of variants depends on which variants are put into it. With good
and almost interchangeable variants, the opposite case, the PBO would rise toward 0.5 even if the family made money
out-of-sample: which one comes first would be chance, and the choice would add nothing, but it would do no harm
either.

| Situation | PBO | Degradation | P(out-of-sample loss) | Reading |
|---|---|---|---|---|
| A thousand trials, almost all noise | 0.40 | 2.36 to 0.14 | 0.37 | Overfitting: the choice is almost chance and the in-sample Sharpe, illusory |
| Good variants with a clear peak | 0.13 | 1.07 to 0.93 | 0.002 | The choice gets it right and the family has an edge |
| The same mixed with clear losers | 0.002 | 1.07 to 0.93 | 0.002 | Lower PBO by construction of the set, not by merit |
| Good, interchangeable variants | Close to 0.5 | Small | Low | Choice irrelevant; what works is the family |

The rule: **the PBO, degradation and the probability of loss are read together**, and the set of variants is fixed
before looking. A low PBO can be manufactured by adding bad variants; a high PBO with rare losses says that what works
is the family, not the parameter.

## What to record and how to report

The three tools need data that only exists if it was saved during research (lesson 2.2.4):

| Tool | What it needs | What to report |
|---|---|---|
| PSR | The return series of the strategy | PSR(0) and, if desired, PSR against a minimum useful Sharpe |
| DSR | The number N of trials and the dispersion of their Sharpes | DSR with the declared N and with a prudent N |
| PBO | The T × N matrix of returns of all the trials | PBO, degradation and probability of loss, with S |

Three limitations to keep in mind. The DSR depends on N, and N is hard to count honestly: it includes ideas half
discarded, variants tried by colleagues and those from the literature that inspired the strategy. The approximation of
the expected maximum assumes independent trials; with correlated trials it is conservative. And the PBO, like CPCV,
mixes in-sample and out-of-sample without respecting the arrow of time: it assumes the process is stable throughout
the sample.

<aside class="callout" data-kind="riesgo">

None of these corrections fixes a biased backtest (lesson 2.5.1). If the strategy executes on the same bar or uses a
universe of survivors, its Sharpe is inflated before any selection, and the DSR only discounts the part that comes
from choosing among trials.

</aside>

## Exercises

1. Compute the PSR of e0518 against an annual Sharpe threshold of 1 (a minimum useful value, not 0). What is the
   probability that its true Sharpe exceeds 1?

**Solution.**

```python
umbral = 1 / np.sqrt(252)      # at daily frequency
p1 = psr(sr["e0518"], umbral, T, asim["e0518"], curt["e0518"])
print(f"PSR(1 annual) of e0518: {p1:.3f}")
```
```output
PSR(1 annual) of e0518: 0.877
```

Against 1 annual, the probability drops from 0.999 to 0.88: a measured Sharpe of 1.58 over four years leaves an
uncertainty of ±0.5 annual per standard error, and 1 sits just over one standard error away. Its true Sharpe is 1.5,
so in this case the correct answer is “yes, it exceeds it,” but the data only says so with moderate probability.
Testing against 0 answers whether there is any edge at all; against a minimum useful value, whether it is worth
trading.

2. Repeat the PBO of the 1,000 strategies with S = 8 and S = 12 blocks. Is the result stable?

**Solution.**

```python
for s in (8, 12, 16):
    p, lg, _, f = pbo_cscv(est.to_numpy(), s=s)
    print(f"S = {s:>2}: {len(lg):>5} partitions, PBO = {p:.3f}, "
          f"P(OOS < 0) = {(f < 0).mean():.2f}")
```
```output
S =  8:    70 partitions, PBO = 0.371, P(OOS < 0) = 0.36
S = 12:   924 partitions, PBO = 0.425, P(OOS < 0) = 0.39
S = 16: 12870 partitions, PBO = 0.398, P(OOS < 0) = 0.37
```

The PBO moves a few points with S, because the size of the blocks and the number of partitions change, but the
conclusion does not: in every case the chosen one lands in the bad half out-of-sample in a large proportion of the
partitions (from 0.37 to 0.43). S is fixed before looking and reported along with the result.

3. A fund presents you with a strategy with a Sharpe of 2.1 over six years of daily data and says it is “the only one
   they tried.” Its team has four researchers who have been working on variants for two years. What N would you use
   for the DSR, and what information would you ask for?

**Solution.**

“The only one they tried” is almost never true in a team that has been doing research for two years: each researcher
will have tried dozens of variants (horizons, filters, universes, costs). I would use a prudent N, in the hundreds,
and compute the DSR with several values to see whether the conclusion depends on it. I would ask for the experiment
log (dates, variants and results, including the discarded ones), the daily series of the strategy to estimate its
skewness and kurtosis, and, if it exists, the matrix of returns of the variants to compute the PBO. If there is no
log, the lack of information is itself a data point: the Sharpe cannot be deflated and should be treated with great
caution.

## Self-assessment

1. Strategy e0518 has an annual Sharpe of 1.58 over 1,003 sessions and a PSR(0) of 0.999. Why does its Deflated Sharpe with N = 1,000 trials drop to 0.50?

   - A. Because with 1,000 trials with no edge, the expected maximum Sharpe is around 1.58 annual: beating zero is not enough, you have to beat the best of the noise
   - B. Because its kurtosis is high
   - C. Because 1,003 sessions are too few for any test
   - D. Because the DSR only works for noise strategies

   *Answer A: Because with 1,000 trials with no edge, the expected maximum Sharpe is around 1.58 annual: beating zero is not enough, you have to beat the best of the noise.* The DSR is the PSR with the threshold SR* at the expected maximum Sharpe of N candidates with no edge. With the measured dispersion of the 1,000 Sharpes (≈ 0.49 annual), that maximum is 1.58: e0518 sits right on it.

2. What goes into the denominator of the PSR, and why?

   - A. Annual volatility, to annualize
   - B. The number of strategies tried
   - C. The risk-free rate
   - D. Skewness and kurtosis: with fat tails or negative skewness, the estimated Sharpe is more uncertain and more evidence is needed

   *Answer D: Skewness and kurtosis: with fat tails or negative skewness, the estimated Sharpe is more uncertain and more evidence is needed.* The standard error of the estimated Sharpe grows with kurtosis and with negative skewness: that is the expression Bailey and López de Prado use in the PSR. The number of trials enters the DSR through the threshold, not the denominator.

3. In the CSCV on the 1,000 strategies, the PBO is 0.40. What does it mean?

   - A. That 40% of the strategies are overfitted
   - B. That in 40% of the partitions, the best in-sample strategy ends up below the median out-of-sample
   - C. That the chosen strategy loses 40% of its Sharpe
   - D. That there is a 40% probability of making money

   *Answer B: That in 40% of the partitions, the best in-sample strategy ends up below the median out-of-sample.* The PBO is the frequency with which the logit of the out-of-sample rank of the chosen strategy is ≤ 0, that is, with which the chosen one lands in the bad half. With pure noise it would be ≈ 0.5; the five with an edge lower it a little.

4. Eight futures momentum variants give a PBO of 0.13. Adding their eight inverses (reversal, all losers), the PBO drops to 0.002, with the same chosen variant and the same out-of-sample Sharpe (0.93). What do you conclude?

   - A. That the sixteen-variant version is a better strategy
   - B. That the PBO has a calculation error
   - C. That the PBO depends on the set of variants: adding obvious losers lowers it without improving anything. It is read together with degradation and the probability of loss, and the set is fixed before looking
   - D. That the reversal variants have an edge

   *Answer C: That the PBO depends on the set of variants: adding obvious losers lowers it without improving anything. It is read together with degradation and the probability of loss, and the set is fixed before looking.* The PBO measures whether the selection process gets the ranking right, not whether the variants make money. Discarding obvious losers is easy, so adding them “improves” the PBO without changing the chosen strategy or its result: that is why it cannot be read on its own.

5. What must be saved during research to be able to compute the DSR and the PBO at the end?

   - A. The number of variants tried and the return series of each one, including the discarded ones
   - B. Only the best strategy
   - C. Only the Sharpe of each variant
   - D. Nothing: they are computed with the final strategy

   *Answer A: The number of variants tried and the return series of each one, including the discarded ones.* The DSR needs N and the dispersion of the Sharpes of the trials; the PBO, the full T × N matrix of returns. If only the winner is saved, the correction is impossible (lesson 2.2.4).

## Key points

- The PSR turns a Sharpe into a probability that the true one exceeds a threshold, taking into account the length of the series, skewness and kurtosis.
- The DSR is the PSR with the threshold at the expected maximum Sharpe of N trials with no edge. With the 1,000 trials of `ruido2`, that maximum is 1.58 annual and no strategy beats it with 95%.
- The best noise (e0598, Sharpe 1.50) and the best with an edge (e0518, 1.58) have DSRs of 0.44 and 0.50: with this data they are indistinguishable. Declaring 7 trials or fewer, e0518 would pass the 95%.
- The CSCV PBO counts how often the best in-sample falls in the bad half out-of-sample: 0.40 in `ruido2`, with the chosen strategy's Sharpe going from 2.36 in-sample to 0.14 out-of-sample.
- The PBO is read with degradation and the probability of loss, and the set of variants is fixed beforehand: adding eight obvious losers lowers the futures momentum PBO from 0.13 to 0.002 without changing anything.
- Without a record of all the trials (N and their returns) no correction is possible.

## Resources

### Documentation

- [SciPy: scipy.stats.norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) · docs.scipy.org

### Books

- [Bailey, D. and López de Prado, M. (2014): The Deflated Sharpe Ratio. Journal of Portfolio Management 40(5), 94-107](https://doi.org/10.3905/jpm.2014.40.5.094) · doi.org
- [Bailey, D. and López de Prado, M. (2012): The Sharpe Ratio Efficient Frontier. Journal of Risk 15(2), 3-44](https://doi.org/10.21314/JOR.2012.255) · doi.org
- [Bailey, Borwein, López de Prado and Zhu (2016): The Probability of Backtest Overfitting. Journal of Computational Finance](https://doi.org/10.21314/JCF.2016.322) · doi.org
- [Lo, A. (2002): The Statistics of Sharpe Ratios. Financial Analysts Journal 58(4), 36-52](https://doi.org/10.2469/faj.v58.n4.2453) · doi.org
- [López de Prado, M. (2018): Advances in Financial Machine Learning. Wiley (chapter 14: PSR and DSR)](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com

### Data

- [Level 2 data: 1,000 noise strategies and five with an edge (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/ruido2_estrategias.parquet) · Download · ruido2_estrategias.parquet

Previous lesson: [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/)

Next lesson: [2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations](https://pythonparatrading.com/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests/)