---
title: "5.4.4 Backtesting ML signals and Deflated Sharpe"
description: "From model prediction to a portfolio with costs, with a log of every experiment and the Deflated Sharpe at the end: the honest ridge nets ≈ 0."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe.png"
---

# 5.4.4 · Backtesting ML signals with costs and Deflated Sharpe

> A positive, stable IC is not money. Between the prediction and the result lie the rule that turns scores into weights, the turnover that rule generates and what each trade costs. This lesson walks that path with the ridge from the walk-forward of 5.4.3, logs every variant it tries and, at the end, discounts the selection with the Deflated Sharpe from lesson 2.5.4.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.4 · Validation for financial series
- Duration: 70 min
- Prerequisites: [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/), [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/), [2.4.7 · Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-transaction-cost-modeling/), [5.1.3 · Economic metrics for trading models: beyond accuracy](https://pythonparatrading.com/en/python-trading-course/level-5/economic-metrics-for-trading-models/)

## By the end of this lesson

- Turn a model's scores into a decile long-short portfolio, measure its turnover and its return gross and net of costs, and reproduce the README figures.
- Compute the turnover cost a signal can bear before losing money and test rules that reduce turnover (less frequent rebalancing, smoothed scores, bands).
- Keep a log of every experiment with its return series and apply the PSR and the Deflated Sharpe to the best one, with the declared and the effective number of trials.
- Check that the Deflated Sharpe corrects for selection but does not detect a leak, and present an honest result with its standard error.

## From IC to money

The walk-forward of lesson 5.4.3 ended with a ridge with an IC of 0.0185 (t = 4.9) over 574 weeks: a real, stable and
statistically clear signal. The question of this lesson is the one that matters to whoever has to trade it: how much
does a portfolio built on that signal earn, after paying to change it every week? The path has four steps, and a fifth
that tends to be forgotten:

```mermaid
flowchart TB
  S[Score] --> C[Portfolio rule]
  C --> W[Weights per week]
  W --> T[Turnover and costs]
  T --> N[Net return]
  N --> R[Log]
  R --> D[Deflated Sharpe]
```

The fifth is the log: every model and portfolio variant you try is a trial, and the Sharpe of the best one only means
something after discounting how many trials there were (lesson 2.5.4).

<aside class="callout" data-kind="aviso">

Synthetic data from the level 5 panel, with a known answer: the oracle (`alfa_verdadero`) marks the ceiling of what
any model can earn. The costs are the README ones: 10 bp per side on turnover. The figures describe this panel, not a
real market; in real stocks costs depend on liquidity, portfolio size and market impact (lesson 4.3.1).

</aside>

## The scores: the walk-forward of 5.4.3

Each lesson runs separately, so this one redoes in a single cell what it needs from 5.4.3: the panel as arrays, the
ridge from sufficient statistics and the walk-forward with the time rule. It adds two reference models: the oracle
and the ridge with `revision_analistas`, the leaky feature.

```python
import numpy as np
import pandas as pd
from scipy.stats import norm, rankdata

feat = pd.read_parquet("panel5_features.parquet")
feat = feat.sort_values(["fecha", "ticker"])
etiq = pd.read_parquet("panel5_etiquetas.parquet")
etiq = etiq.sort_values(["fecha", "ticker"])
verdad = pd.read_parquet("panel5_verdad.parquet")
verdad = verdad.sort_values(["fecha", "ticker"])
TODAS = [c for c in feat.columns if c not in
         ("fecha", "ticker", "sector")]
fechas = pd.DatetimeIndex(feat["fecha"].unique())
nW, nA = len(fechas), feat["ticker"].nunique()

F = feat[TODAS].to_numpy(dtype=float).reshape(nW, nA, -1)
with np.errstate(invalid="ignore", divide="ignore"):
    n = np.isfinite(F).sum(axis=1, keepdims=True)
    G = np.nan_to_num(norm.ppf(
        (rankdata(F, axis=1, nan_policy="omit") - 0.5) / n))
R = etiq["ret_sig_semana"].to_numpy().reshape(nW, nA)
fin = etiq["fin_sig_semana"].to_numpy().reshape(nW, nA)[:, 0]
con_feat = np.isfinite(F[:, :, 0]).all(axis=1)
valida = con_feat & np.isfinite(R).all(axis=1)
Yc = np.zeros_like(R)
Yc[valida] = R[valida] - R[valida].mean(axis=1, keepdims=True)
anyo = fechas.year.to_numpy()
semanas = np.flatnonzero(valida & (anyo >= 2015))  # backtest

def estadisticos(cols):
    X1 = np.concatenate([np.ones((nW, nA, 1)), G[:, :, cols]], 2)
    pv = valida.astype(float)
    return (np.einsum("wni,wnj->wij", X1, X1) * pv[:, None, None],
            np.einsum("wni,wn->wi", X1, Yc) * pv[:, None])

def walk_forward(cols, ventana=None, cada=None, anual=False,
                 semivida=None, alfa=10.0):
    S_xx, S_xy = estadisticos(cols)
    k = len(cols)
    pred, beta = np.full((nW, nA), np.nan), None
    for i, w in enumerate(semanas):
        if (beta is None or (cada and i % cada == 0) or
                (anual and anyo[w] != anyo[semanas[i - 1]])):
            u = int(np.searchsorted(fin, fechas[w])) - 1
            p = valida.astype(float)
            p[u + 1:] = 0.0
            if ventana:
                p[:max(0, u + 1 - ventana)] = 0.0
            if semivida:
                p *= 0.5 ** ((u - np.arange(nW)) / semivida)
            A, b = np.tensordot(p, S_xx, 1), np.tensordot(p, S_xy, 1)
            mx, my = A[0, 1:] / A[0, 0], b[0] / A[0, 0]
            C = A[1:, 1:] - A[0, 0] * np.outer(mx, mx)
            c = b[1:] - A[0, 0] * mx * my
            beta = np.linalg.solve(C + alfa * np.eye(k), c)
        pred[w] = G[w][:, cols] @ beta
    return pred

sin_fuga = [i for i, c in enumerate(TODAS)
            if c != "revision_analistas"]
modelos = {
    "exp. annual": walk_forward(sin_fuga, anual=True),
    "exp. weekly": walk_forward(sin_fuga, cada=1),
    "rolling 3 years": walk_forward(sin_fuga, ventana=156, cada=1),
    "half-life 2 years": walk_forward(sin_fuga, semivida=104, cada=1),
}
fuga = walk_forward(list(range(len(TODAS))), anual=True)
oraculo = verdad["alfa_verdadero"].to_numpy().reshape(nW, nA)
print(f"{len(semanas)} backtest weeks: "
      f"{fechas[semanas[0]].date()} to {fechas[semanas[-1]].date()}")
```
```output
574 backtest weeks: 2015-01-02 to 2025-12-26
```

## From scores to weights, with costs

The portfolio rule is the README one: each week, long the top-score decile and short the bottom one, equally
weighted, one dollar per leg (two of gross notional). Between one week and the next, weights drift with returns;
**turnover** is the sum of the absolute weight changes needed to get back to the target portfolio, and the cost is
10 bp per unit of turnover:

$$
\text{turnover}_t = \sum_i \left| w_{i,t} - \tilde w_{i,t-1} \right|, \qquad
r^{\text{net}}_t = \sum_i w_{i,t}\, r_{i,t+1} - c \cdot \text{turnover}_t
$$

with $\tilde w_{i,t-1} = w_{i,t-1}(1 + r_{i,t})$ the previous week's weight after moving with the market and
$c$ = 0.001. The function supports three variants that reduce turnover and are used further down:

- `cada`: rebalance only every n weeks (in between, the weights drift).
- `suavizado`: trade on an exponential average of the score's rank, $s_t = \lambda\,\text{rank}_t +
  (1 - \lambda)\,s_{t-1}$, instead of on that week's score. It is the idea of Gârleanu and Pedersen (2013): move only
  part of the way toward the target portfolio.
- `banda`: enter in the extreme 10% and do not exit while still in the 30% (a band between buying and holding, like those studied
  by Novy-Marx and Velikov, 2016).

```python
def cartera(score, q=0.1, cada=1, suavizado=None, banda=None,
            coste=0.001):
    """Gross and net weekly returns and mean turnover."""
    w_ant = np.zeros(nA)
    s_prev = None
    bruto, neto, rot = [], [], []
    for i, w in enumerate(semanas):
        rk = (rankdata(score[w]) - 0.5) / nA
        if suavizado:
            s_prev = rk if s_prev is None else (
                suavizado * rk + (1 - suavizado) * s_prev)
            rk = (rankdata(s_prev) - 0.5) / nA
        if i % cada == 0:
            largo, corto = rk >= 1 - q, rk <= q
            if banda:
                largo |= (w_ant > 0) & (rk >= 1 - banda)
                corto |= (w_ant < 0) & (rk <= banda)
            peso = np.zeros(nA)
            peso[largo] = 1 / largo.sum()
            peso[corto] = -1 / corto.sum()
        else:
            peso = w_ant
        r = float(peso @ R[w])
        t = np.abs(peso - w_ant).sum()
        bruto.append(r)
        neto.append(r - coste * t)
        rot.append(t)
        w_ant = peso * (1 + R[w])
    return np.array(bruto), np.array(neto), float(np.mean(rot[1:]))

def sharpe(x):
    return x.mean() / x.std(ddof=1) * np.sqrt(52)

filas = {}
for nombre, sc in (("oracle", oraculo),
                   ("ridge (exp. annual)", modelos["exp. annual"]),
                   ("ridge + leak", fuga)):
    b, nt, rot = cartera(sc)
    filas[nombre] = {"gross ret.": f"{b.mean() * 52:.1%}",
                     "vol.": f"{b.std(ddof=1) * np.sqrt(52):.1%}",
                     "gross Sharpe": round(sharpe(b), 2),
                     "turnover": f"{rot:.0%}",
                     "net ret.": f"{nt.mean() * 52:.1%}",
                     "net Sharpe": round(sharpe(nt), 2)}
pd.DataFrame(filas).T
```
```output
                    gross ret.  vol. gross Sharpe turnover net ret. net Sharpe
oracle                   17.1%  9.6%         1.78     259%     3.6%       0.38
ridge (exp. annual)      12.1%  9.7%         1.25     265%    -1.7%      -0.18
ridge + leak            136.5%  6.9%        19.89     287%   121.6%      17.74
```

The three rows are the README ones. The oracle, which knows the true alpha, has a gross Sharpe of 1.78 and a net one
of 0.38: even the best possible information barely survives a weekly turnover of 259%. The honest ridge goes from
1.25 gross to −0.18 net. The leaky one, from 19.9 to 17.7: an impossible number, which in a real backtest should set
off the alarm before any test.

The figure at the top shows the difference over time for the ridge: the gross curve rises steadily; the net one, with
the same shape, ends below where it started.

## The cost arithmetic

The arithmetic is simple and worth doing before any backtest. With a weekly turnover $\tau$ and a cost $c$ per unit of
turnover, the annual cost is $52\,\tau\,c$. The signal can bear the cost as long as its annual gross return exceeds
it:

```python
b, nt, rot = cartera(modelos["exp. annual"])
coste_anual = 52 * rot * 0.001
equilibrio = b.mean() * 52 / (52 * rot)
print(f"annual gross return {b.mean() * 52:.1%}; "
      f"annual cost {coste_anual:.1%}")
print(f"break-even cost: {equilibrio * 1e4:.1f} bp per side")
```
```output
annual gross return 12.1%; annual cost 13.8%
break-even cost: 8.8 bp per side
```

The ridge earns 12.1% gross a year and pays 13.8% in costs. It could bear about 8.8 bp per side: with the README's 10,
it loses. There are two levers: more gross return per unit of turnover (a better signal) or less turnover for the same
return (a better portfolio rule). The second is the one you can touch without modeling again.

## An experiment log

Before trying variants, the log. Each experiment stores its full configuration, its metrics and its **series of net
returns**: without the series you cannot compute the dispersion of the Sharpes or the PBO at the end (lesson 2.2.4).
The log also stores the variants that do not work, which are the ones that tell you most about luck.

The grid is small and reasonable, the one anyone would try: the four walk-forward models of 5.4.3 (expanding annual and
weekly, three-year rolling, weights with a two-year half-life) times five portfolio rules (deciles, quintiles, deciles
every 4 weeks, smoothed ranks with λ = 0.25 and a 30% band). Twenty experiments.

```python
reglas = {"deciles": {}, "quintiles": {"q": 0.2},
          "every 4 weeks": {"cada": 4},
          "smoothed 0.25": {"suavizado": 0.25},
          "band 30%": {"banda": 0.3}}
registro, series = [], {}
for m, sc in modelos.items():
    for r, kw in reglas.items():
        b, nt, rot = cartera(sc, **kw)
        clave = f"{m} | {r}"
        registro.append({"experiment": clave, "model": m,
                         "rule": r, "params": kw,
                         "gross Sharpe": sharpe(b),
                         "net Sharpe": sharpe(nt), "turnover": rot})
        series[clave] = nt
registro = pd.DataFrame(registro).set_index("experiment")
netos = pd.DataFrame(series, index=fechas[semanas])
print(f"logged experiments: {len(registro)}")
(registro[["gross Sharpe", "net Sharpe", "turnover"]]
 .sort_values("net Sharpe", ascending=False).head(8).round(2))
```
```output
logged experiments: 20
                                   gross Sharpe  net Sharpe  turnover
experiment
exp. annual | smoothed 0.25                0.88        0.37      0.86
exp. weekly | smoothed 0.25                0.79        0.25      0.88
exp. annual | every 4 weeks                0.59        0.18      0.73
exp. annual | band 30%                     1.09        0.12      1.44
exp. weekly | every 4 weeks                0.46        0.05      0.73
half-life 2 years | smoothed 0.25          0.56        0.05      0.86
exp. weekly | band 30%                     1.00        0.03      1.49
exp. annual | deciles                      1.25       -0.18      2.65
```

*Fig. — The 20 logged experiments (4 walk-forward models by 5 portfolio rules) on the level 5 panel: weekly turnover against Sharpe net of 10 bp per side.* Cloud of 20 points. High-turnover rules (weekly deciles and quintiles, 207% to 270%) have net Sharpes from -0.18 to -0.95. Low-turnover rules (every 4 weeks, smoothed, band; 72% to 149%) sit between -0.62 and 0.37. The best, expanding yearly ridge with smoothed ranks (highlighted), gives 0.37 at 86% turnover, almost on the line of the oracle with weekly deciles (0.38).

The figure places the twenty experiments on the turnover-net Sharpe plane. The rule is visible: the high-turnover
variants (weekly deciles and quintiles, on the right) lose money with any model; the low-turnover ones (rebalancing
every 4 weeks, smoothing, band) stay close to zero or somewhat above it with the expanding models, and below it with
the rolling window, which lesson 5.4.3 already ruled out because of its IC. The best is the annual expanding ridge with
smoothed ranks: a net Sharpe of 0.37, with a turnover of 86%. Almost the same as the oracle with weekly deciles.

## The Deflated Sharpe of the best one

Is 0.37 a result? Three questions, from least to most demanding. The functions are those of lesson 2.5.4, with the
Sharpes at the frequency of the data (weekly):

$$
\widehat{PSR}(SR^*) = \Phi\left(\frac{(\widehat{SR} - SR^*)\sqrt{T-1}}{\sqrt{1 - \widehat{\gamma}_3\,\widehat{SR} + \frac{\widehat{\gamma}_4 - 1}{4}\,\widehat{SR}^2}}\right),
\qquad
SR_0 = \sqrt{V[\widehat{SR}_n]}\left[(1-\gamma)\,\Phi^{-1}\!\left(1-\tfrac{1}{N}\right) + \gamma\,\Phi^{-1}\!\left(1-\tfrac{1}{N e}\right)\right]
$$

```python
def psr(sr, umbral, t, asim, curt):
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr ** 2)
    return float(norm.cdf((sr - umbral) * np.sqrt(t - 1) / den))

def sr_max_esperado(n, var_sr):
    g = np.euler_gamma
    return np.sqrt(var_sr) * ((1 - g) * norm.ppf(1 - 1 / n)
                              + g * norm.ppf(1 - 1 / (n * np.e)))

sr_sem = netos.mean() / netos.std()       # weekly Sharpes
mejor = sr_sem.idxmax()
x = netos[mejor]
T, asim, curt = len(x), x.skew(), x.kurt() + 3
sr = sr_sem[mejor]
et_anual = np.sqrt((1 + sr ** 2 / 2) / T) * np.sqrt(52)
print(f"best: {mejor}")
print(f"net Sharpe {sr * np.sqrt(52):.2f} ± {et_anual:.2f} "
      f"(standard error, {T} weeks)")
print(f"PSR(0) = {psr(sr, 0, T, asim, curt):.3f}")

autoval = np.linalg.eigvalsh(netos.corr().to_numpy())
n_ef = autoval.sum() ** 2 / (autoval ** 2).sum()
for etiqueta, n_int in (("effective trials", n_ef),
                        ("20 logged", 20),
                        ("with the 13 from 5.4.3 x 5 rules", 65)):
    sr0 = sr_max_esperado(n_int, sr_sem.var())
    print(f"N = {n_int:5.1f} ({etiqueta}): SR_0 = "
          f"{sr0 * np.sqrt(52):.2f} annual, "
          f"DSR = {psr(sr, sr0, T, asim, curt):.3f}")
```
```output
best: exp. annual | smoothed 0.25
net Sharpe 0.37 ± 0.30 (standard error, 574 weeks)
PSR(0) = 0.890
N =   2.0 (effective trials): SR_0 = 0.20 annual, DSR = 0.713
N =  20.0 (20 logged): SR_0 = 0.71 annual, DSR = 0.128
N =  65.0 (with the 13 from 5.4.3 x 5 rules): SR_0 = 0.89 annual, DSR = 0.042
```

First question: can it be told apart from zero, as if it were the only trial? The standard error of an annual Sharpe
with eleven years of weekly data is ±0.30, almost as large as the Sharpe itself, and the PSR(0) is 0.89: even
without correction it does not reach 95%.

Second: does it beat the best one expected among the trials? The 20 experiments are highly correlated (they share
models and rules), and the participation ratio of the eigenvalues of their correlation matrix gives only 2 effective
trials: with such a favorable N, the threshold is a net Sharpe of 0.20 and the DSR, 0.71. With the 20 declared, the
threshold rises to 0.71 and the DSR drops to 0.13. And the honest count includes the trials of the previous lesson:
the walk-forward of 5.4.3 tested thirteen schemes before choosing the four used here, and each one could have been
combined with five rules; with 65, the DSR drops to 0.04. No way of counting reaches 0.95.

Third: what do you present? What the numbers allow: **the honest ridge gives a net Sharpe compatible with zero**,
between −0.4 and 0.4 depending on the portfolio rule with the expanding models, with a standard error of ±0.3. It is
the README's answer (“net Sharpe ≈ 0”) and it is the correct conclusion, even though the gross IC is clearly positive.

The figure sums up the second question: the DSR of the best experiment depending on how many trials are declared. No
reasonable way of counting brings it close to the 0.95 line.

*Fig. — Deflated Sharpe of the best of the 20 experiments (net Sharpe 0.37 over 574 weeks) by the number of declared trials.* Falling curve on a log axis from 1 to 100 trials. With a single trial it is PSR(0), 0.89, below the 0.95 line. With the 2 effective trials from the participation ratio, 0.72. With the 20 logged (highlighted), 0.13. With 65, counting the previous lesson's schemes, 0.04.

## The DSR does not see leaks

The ridge with `revision_analistas` needs no correction to look good, and no correction uncovers it:

```python
_, nt_fuga, rot_fuga = cartera(fuga)
sr_f = nt_fuga.mean() / nt_fuga.std(ddof=1)
var_con = pd.concat([sr_sem, pd.Series({"leak": sr_f})]).var()
sr0 = sr_max_esperado(21, var_con)
xf = pd.Series(nt_fuga)
dsr_f = psr(sr_f, sr0, len(xf), xf.skew(), xf.kurt() + 3)
print(f"net Sharpe {sr_f * np.sqrt(52):.1f}; SR_0 with N = 21: "
      f"{sr0 * np.sqrt(52):.2f}; DSR = {dsr_f:.3f}")
```
```output
net Sharpe 17.7; SR_0 with N = 21: 7.58; DSR = 1.000
```

A net Sharpe of 17.7 leaves any threshold behind, even the 7.6 that comes out when the leak itself is included in the
dispersion of the trials: the DSR is 1.000. The DSR discounts **selection** (having chosen the best of N trials with
no edge), not **information from the future**: with a leak, the edge is real inside the backtest, because the
backtest sees it. The only things that give it away are the order of magnitude (a weekly net Sharpe of 17 in stocks
does not exist) and, above all, the audit of when each piece of data is known (lesson 5.2.2). Validation (5.4.1-5.4.3)
and the correction for selection (this lesson) are applied **after** that audit, not instead of it.

<aside class="callout" data-kind="riesgo">

Practical order before believing an ML backtest: (1) point-in-time audit of every feature; (2) validation without
overlap (purging, embargo, walk-forward); (3) realistic costs and turnover; (4) a log of every trial and DSR or PBO;
(5) comparison with a simple baseline and with the order of magnitude in the literature. Skipping the first one
invalidates the other four.

</aside>

## What to present

| Question | Answer in this panel |
|---|---|
| Does the model have signal? | Yes: IC 0.0185, t = 4.9 (lesson 5.4.3) |
| Does it make money with the starting rule? | No: net Sharpe −0.18 with weekly deciles |
| And with the best portfolio rule? | 0.37 ± 0.30, PSR(0) 0.89 |
| Does it survive selection? | No: DSR 0.13 with 20 trials, 0.71 with the 2 effective ones |
| What is the ceiling? | The oracle: 0.38 net with weekly deciles |
| Conclusion | Real signal, net profit compatible with zero; work pending on costs and execution |

The last row is not a failure: it is information. It says that the next effort is not in another model (the ridge
already captures a sizable part of what the oracle captures) but in execution: lower costs (lesson 4.3.1), optimized
portfolios with a turnover penalty (lesson 4.1.5) or a signal combined with other, less correlated ones (lesson
5.10.3).

## Exercises

1. Apply the best portfolio rule (smoothed ranks with λ = 0.25) to the oracle. What is the best net Sharpe this panel
   allows with that rule, and how far is the ridge from it?

**Solution.**

```python
for nombre, sc in (("oracle", oraculo),
                   ("ridge exp. annual", modelos["exp. annual"])):
    b, nt, rot = cartera(sc, suavizado=0.25)
    print(f"{nombre:<18} gross {sharpe(b):.2f}  net {sharpe(nt):.2f}"
          f"  turnover {rot:.0%}")
```
```output
oracle             gross 1.14  net 0.64  turnover 91%
ridge exp. annual  gross 0.88  net 0.37  turnover 86%
```

With the smoothed rule, the oracle goes from 0.38 to 0.64 net Sharpe and the ridge stays at 0.37: the rule does not only help the ridge, it helps any slow signal.
The distance between the two net figures is what remains to be gained with a better model; the distance between the
oracle's net and gross figures, what remains to be gained with better execution. Always comparing with the ceiling
tells you what to work on.

2. Compute the break-even cost per side of the best variant (annual expanding ridge, smoothing 0.25). At what real
   costs would it be tradable?

**Solution.**

```python
b, nt, rot = cartera(modelos["exp. annual"], suavizado=0.25)
print(f"turnover {rot:.0%} weekly; annual gross {b.mean() * 52:.1%};"
      f" break-even {b.mean() / rot * 1e4:.1f} bp per side")
```
```output
turnover 86% weekly; annual gross 7.7%; break-even 17.2 bp per side
```

With a weekly turnover of 86% and 7.7% gross a year, the smoothed ridge can bear 17.2 bp per side, twice as much as
the weekly decile version (8.8). That puts it within reach of portfolios in liquid stocks, where the total cost
(commission, bid-ask spread and market impact) can stay below that; but the margin over the 10 bp is about 7 bp,
impact grows with size, and the resulting Sharpe cannot be told apart from zero. The practical conclusion is the same:
before trading it you have to measure the real costs with the real portfolio.

3. Write down what fields the log of an experiment would have in a real project, beyond those in the cell.

**Solution.**

Besides the configuration and the net return series: the data version (and its fingerprint, for example a hash of the
file), the code version (commit), the seeds, the training and test periods, the validation scheme with its parameters
(purging, embargo, window, frequency), the cost model, the date and the person who launched it, and a note on why it
was tried. Also the experiments that failed or were aborted. With that, any number in the final report can be
reproduced and the number of trials can be counted without relying on memory (lesson 2.2.4). Tools such as MLflow
automate it, but a text file with one row per experiment is enough if it is maintained with discipline.

## Self-assessment

1. The walk-forward ridge has a gross Sharpe of 1.25 with a weekly turnover of 265% and costs of 10 bp per side. Why is the net −0.18?

   - A. Because the model is overfitted
   - B. Because turnover costs 2.65 × 52 × 10 bp ≈ 13.8% a year and the portfolio earns 12.1% gross: the cost eats the whole edge and then some
   - C. Because costs of 10 bp are unrealistic
   - D. Because the net Sharpe is computed with log returns

   *Answer B: Because turnover costs 2.65 × 52 × 10 bp ≈ 13.8% a year and the portfolio earns 12.1% gross: the cost eats the whole edge and then some.* The annual gross return is 12.1%; the annual cost, turnover times the cost per unit, 13.8%. The break-even point is about 8.8 bp per side: at 10, you lose.

2. The best of the 20 variants in the log (rebalancing with smoothed scores) has a net Sharpe of 0.37. What does its Deflated Sharpe say with N = 20?

   - A. That, given the dispersion of the 20 trials, the best one expected by chance has a net Sharpe of around 0.71: 0.37 is well below that, and the DSR is 0.13
   - B. That it is significant at 95%
   - C. That the DSR cannot be computed with weekly data
   - D. That the result is negative

   *Answer A: That, given the dispersion of the 20 trials, the best one expected by chance has a net Sharpe of around 0.71: 0.37 is well below that, and the DSR is 0.13.* Even without correction, its PSR(0) is 0.89, below 0.95. Corrected for having chosen the best of 20, the DSR falls to 0.13. It is a result compatible with zero.

3. With `revision_analistas`, the same ridge has a net Sharpe of 17.7 and a DSR of 1.000 for any N. What do you conclude?

   - A. That the strategy is excellent
   - B. That the DSR is coded wrong
   - C. That more trials have to be declared
   - D. That the DSR discounts selection among trials, not information from the future: a leak passes any statistical correction. A net Sharpe of 17 is the alarm

   *Answer D: That the DSR discounts selection among trials, not information from the future: a leak passes any statistical correction. A net Sharpe of 17 is the alarm.* The DSR answers “could this be the best of N trials with no edge?”. With a leak the edge is real inside the backtest, because the backtest sees the future. Only the point-in-time audit uncovers it.

4. What must be saved from each experiment to be able to compute the DSR and the PBO at the end?

   - A. The Sharpe of the best one
   - B. Only the parameters of the final model
   - C. The full configuration and the net return series of every variant, including the discarded ones, and the total number of variants tried (including those from previous lessons or notebooks)
   - D. Nothing: the DSR only needs the final Sharpe

   *Answer C: The full configuration and the net return series of every variant, including the discarded ones, and the total number of variants tried (including those from previous lessons or notebooks).* The DSR needs N and the dispersion of the Sharpes of all the trials; the PBO, the full matrix of returns. What is not logged cannot be discounted.

5. What do the smoothing and band rules do to improve the net result?

   - A. They increase the model's IC
   - B. They eliminate costs
   - C. They change the universe
   - D. They reduce turnover (from 265% to 86-144% a week) at the cost of following the signal with a lag: they lose gross Sharpe, but save more in costs than they lose

   *Answer D: They reduce turnover (from 265% to 86-144% a week) at the cost of following the signal with a lag: they lose gross Sharpe, but save more in costs than they lose.* With a signal that changes little from one week to the next, trading every small change in rank is expensive. Smoothing or requiring a band to enter and exit keeps most of the exposure with a fraction of the trades.

## Key points

- The README is reproduced: oracle with a gross Sharpe of 1.78 and net 0.38; ridge with all the features, 1.25 gross and −0.18 net, with a weekly turnover of 265% and 10 bp per side.
- The cost arithmetic is simple: turnover × 52 × cost per side. The ridge can bear about 8.8 bp per side before losing.
- Reducing turnover (smoothing scores, bands, rebalancing every 4 weeks) lowers the gross Sharpe and raises the net. The best of 20 variants reaches 0.37 net.
- That 0.37 has a standard error of ±0.30 over eleven years, a PSR(0) of 0.89 and a DSR of 0.13 with 20 trials: compatible with zero.
- The experiment log is part of the result: without the series of each variant there is no DSR or PBO.
- The DSR does not see leaks: the model with `revision_analistas` gives a net Sharpe of 17.7 and a DSR of 1.000.

## Resources

### Research

- [Introduction to Quant Research](https://pythonparatrading.com/en/quant-research-2/) · Research-0023 · 8 min

### Documentation

- [SciPy: scipy.stats.norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) · docs.scipy.org

### Books

- [Bailey, D. and López de Prado, M. (2014): The Deflated Sharpe Ratio. Journal of Portfolio Management 40(5), 94-107](https://doi.org/10.3905/jpm.2014.40.5.094) · doi.org
- [Novy-Marx, R. and Velikov, M. (2016): A Taxonomy of Anomalies and Their Trading Costs. Review of Financial Studies 29(1), 104-147](https://doi.org/10.1093/rfs/hhv063) · doi.org
- [Gârleanu, N. and Pedersen, L. H. (2013): Dynamic Trading with Predictable Returns and Transaction Costs. Journal of Finance 68(6), 2309-2340](https://doi.org/10.1111/jofi.12080) · doi.org
- [López de Prado, M. (2018): Advances in Financial Machine Learning. Wiley (chapters 11 to 14: backtesting, statistics and DSR)](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com

### Data

- [Level 5 data: true alpha of the panel (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_verdad.parquet) · Download · panel5_verdad.parquet

Previous lesson: [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/)

Next lesson: [5.5.1 · Regularized regressions: Ridge, Lasso, Elastic Net and logistic](https://pythonparatrading.com/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression/)