---
title: "2.1.1 Hypothesis tests, bootstrap and Newey-West errors"
description: "Is the mean different from zero? t-test, i.i.d. and block bootstrap, power and regression with HAC errors, tested on planted effects."
url: "https://pythonparatrading.com/en/python-trading-course/level-2/hypothesis-testing-bootstrap-robust-errors/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/hypothesis-testing-bootstrap-robust-errors.png"
---

# 2.1.1 · Hypothesis tests, p-values, bootstrap and regression with robust standard errors

> A backtest is a sample and its mean return is an estimate with error. This lesson measures that error with three tools (the t-test, the bootstrap and Newey-West standard errors) on synthetic data where you know which strategies have an edge and which do not, and shows why overlapping returns manufacture a t of 10.5 where the honest one is 1.8.

- Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/)
- Module: 2.1 · Statistics and econometrics for trading
- Duration: 70 min
- Prerequisites: [0.7.2 · Probability and distributions, correlation versus causation and simple regression](https://pythonparatrading.com/en/python-trading-course/level-0/probability-distributions-and-regression/), [1.6.2 · Stylized facts: fat tails, clustering, autocorrelation and leverage](https://pythonparatrading.com/en/python-trading-course/level-1/stylized-facts-of-financial-markets/), [1.6.3 · Seasonality: day of the week, month, turn of the month and intraday sessions](https://pythonparatrading.com/en/python-trading-course/level-1/seasonality-and-calendar-effects/)

## By the end of this lesson

- Set up a test on the mean return of a strategy and read its p-value, its type I and type II errors and its power.
- Compute confidence intervals with the t formula, with the i.i.d. bootstrap and with the block bootstrap, and know when they differ.
- Measure how overlapping returns inflate the t-statistic and correct it with Newey-West errors (`cov_type="HAC"`) or with non-overlapping samples.
- Estimate a multiple regression on a futures panel with errors robust to autocorrelation and to cross-market correlation (Driscoll-Kraay).

## A backtest is a sample

A four-year backtest with a mean return of 15% a year does not say that the strategy earns 15% a year. It says that,
in those four particular years, it earned that. Had the market rolled other dice, the mean would be different. The
question of this lesson is how different it could have turned out and, in particular, whether a result like that is
compatible with the true edge being **zero**.

Lesson 1.6.3 already used the t-statistic and the bootstrap for calendar effects. Here the method is formalized, its
power is measured and the two problems that produce the most false t's in practice are dealt with: observations that
overlap and assets that move together.

```mermaid
flowchart TB
  H[Null hypothesis] --> E[Statistic]
  E --> D[Distrib. under H0]
  D --> P[p-value]
  P --> R{p < α?}
  R -- yes --> RE[Reject H0]
  R -- no --> NR[Do not reject]
```

- **Null hypothesis** (H0): the expected mean return is zero. It is what you try to refute.
- **Statistic**: a number computed from the sample that would be small if H0 were true. For the mean, Student's t:

$$
t = \frac{\overline{r}}{s / \sqrt{n}}
$$

  with $\overline{r}$ the mean of the $n$ returns and $s$ their standard deviation.
- **Distribution under H0**: how the statistic would be distributed if H0 were true. With independent observations
  and large $n$, a standard normal; with the bootstrap, the one that comes out of resampling the data.
- **p-value**: the probability, if H0 were true, of obtaining a statistic at least as extreme as the one observed.
  It is not the probability that H0 is true.

When deciding with a threshold α (5% by convention) you can make two errors:

| | H0 true (no edge) | H0 false (there is an edge) |
|---|---|---|
| H0 is rejected | Type I error (false positive), probability α | Correct: the **power** |
| H0 is not rejected | Correct | Type II error (false negative) |

In trading, the type I error is the expensive one: you put money into a strategy with no edge and pay costs for
nothing. The type II error is quieter: you discard a good strategy because the sample was short.

A useful identity connects the t with the Sharpe. If $SR$ is the annual Sharpe and the sample lasts $T$ years,

$$
t = \frac{\overline{r}}{s}\sqrt{n} = SR_{\text{daily}}\sqrt{n} = SR \sqrt{T}
$$

A Sharpe of 1 over 4 years gives a t of 2; a Sharpe of 0.5 needs 16 years to reach 2.

## Four strategies with a known answer

`ruido2_estrategias.parquet` holds the daily returns of 1,000 fictional strategies over the 1,003 sessions from 2022
to 2025. They are synthetic data: 995 have no edge at all (true Sharpe 0) and five have a planted one, documented in
`ruido2_verdad.csv`. That one of them wins here says nothing about any real market; it serves to know whether a
method gets it right. Four are chosen:

- `e0518`, true Sharpe 1.5: the clearest edge in the set.
- `e0258`, true Sharpe 0.7: a real but modest edge.
- `e0001`, no edge: any one of them.
- the best of the 995 with no edge, by its measured Sharpe.

```python
import numpy as np
import pandas as pd
from scipy import stats

r = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv").set_index("estrategia")
sr = r.mean() / r.std() * np.sqrt(252)
nulas = verdad.index[verdad["sharpe_real"] == 0]
mejor_nula = sr[nulas].idxmax()
elegidas = ["e0518", "e0258", "e0001", mejor_nula]

filas = {}
for e in elegidas:
    t, p = stats.ttest_1samp(r[e], 0)
    filas[e] = {"sharpe_real": verdad.loc[e, "sharpe_real"],
                "sharpe_medido": sr[e],
                "media_anual_%": 100 * 252 * r[e].mean(),
                "t": t, "p": p,
                "sr_x_raiz_anos": sr[e] * np.sqrt(len(r) / 252)}
print(len(r), "sessions;", len(nulas), "strategies with no edge")
pd.DataFrame(filas).T.round(3)
```
```output
1003 sessions; 995 strategies with no edge
       sharpe_real  sharpe_medido  media_anual_%      t      p  sr_x_raiz_anos
e0518          1.5          1.579         33.165  3.150  0.002           3.150
e0258          0.7          0.622          8.492  1.240  0.215           1.240
e0001          0.0          0.003          0.063  0.007  0.995           0.007
e0598          0.0          1.503         26.607  2.999  0.003           2.999
```

The last column checks the identity: the t from `scipy.stats.ttest_1samp` matches the measured Sharpe times the square
root of the years (3.98 years). The table already shows the two errors:

- `e0518` (true Sharpe 1.5) comes out with t = 3.15 and p = 0.002: the test gets it right.
- `e0258` has a real edge, but its measured Sharpe is 0.62 and its p, 0.215: with four years it cannot be told apart
  from noise. It is a type II error, and it is not exceptional bad luck: with a true Sharpe of 0.7 and four years, the
  expected t is 1.4.
- The best of the nulls, `e0598`, has a measured Sharpe of 1.50 and p = 0.003: more “significant” than one of the
  good ones. It is the maximum of 995 tests; the p-value of a test is only valid for a hypothesis chosen before
  looking. Lesson 2.1.8 deals with that selection; here it is enough to remember that no method in this lesson
  corrects it.

## The bootstrap: the distribution without formulas

The t-test assumes that the mean is normally distributed. With fat-tailed returns (lesson 1.6.2) and a thousand
observations, the approximation is usually good, but you do not always have a formula: for the Sharpe, the drawdown or
the median you have to work harder. The **bootstrap** avoids the formula: it resamples the data with replacement
thousands of times, recomputes the statistic on each resample and uses the dispersion of those values as its error.

```mermaid
flowchart LR
  M[Sample of n days] --> R[Resample B times]
  R --> E[Statistic ×B]
  E --> I[CI percentiles]
```

The **i.i.d.** bootstrap draws individual days, so it destroys any dependence between consecutive days. The **block
bootstrap** draws stretches of consecutive days and keeps the dependence within each stretch. The `arch` library has
both: `IIDBootstrap` and `StationaryBootstrap` (Politis and Romano, 1994), whose blocks have a random length with a
given mean.

```python
from arch.bootstrap import IIDBootstrap, StationaryBootstrap

def media_anual(x):
    return 252 * x.mean()

def intervalos(x, bloque=20, reps=2000, semilla=7):
    x = np.asarray(x)
    m, se = media_anual(x), 252 * x.std(ddof=1) / np.sqrt(len(x))
    q = stats.t.ppf(0.975, len(x) - 1)
    out = {"t": (m - q * se, m + q * se)}
    for nombre, bs in [("iid", IIDBootstrap(x, seed=semilla)),
                       ("bloques", StationaryBootstrap(
                           bloque, x, seed=semilla))]:
        ic = bs.conf_int(media_anual, reps=reps, method="percentile")
        out[nombre] = tuple(ic.ravel())
    return out

ic = {(e, k): 100 * np.array(v) for e in elegidas
      for k, v in intervalos(r[e]).items()}
pd.DataFrame(ic, index=["inf_%", "sup_%"]).T.round(1)
```
```output
               inf_%  sup_%
e0518 t         12.5   53.8
      iid       13.5   55.0
      bloques    9.0   55.5
e0258 t         -4.9   21.9
      iid       -4.7   22.1
      bloques   -5.0   22.2
e0001 t        -18.8   19.0
      iid      -18.7   20.2
      bloques  -19.1   19.3
e0598 t          9.2   44.0
      iid        9.0   43.7
      bloques    6.8   46.0
```

*Fig. — 95% interval for the mean annual return of four synthetic strategies, by the t formula, i.i.d. bootstrap and block bootstrap.* Twelve horizontal segments, three per strategy, with a dot at the mean. The three methods nearly coincide for each strategy. e0518, true Sharpe 1.5: 9 to 55%. e0258, true Sharpe 0.7: −5 to 22%, crossing zero. e0001, no edge: −19 to 20%. The best of the 995 no-edge strategies, highlighted: 7 to 46%, clear of zero although its real edge is nil.

The three methods give almost the same interval for the four strategies; the block one comes out somewhat wider at
the bottom for `e0518` and `e0598` (9.0 versus 12.5 for the first), because of resampling randomness and because of
volatility clustering, which the blocks preserve. It is no coincidence that they agree: these daily returns have no
autocorrelation (only volatility clustering), so resampling individual days or blocks of 20 gives the same thing, and
with 1,003 days the mean is already distributed almost like a normal. It is the usual situation with **daily**
returns of a strategy that does not hold positions for many days: the t-test is enough.

The practical lesson is in the width. The interval for `e0518` runs from about 12% to 55% a year: the edge is real,
but four years do not say whether it is large or small. The one for `e0258`, from −6% to +22%, includes zero even
though the edge exists.

The bootstrap also gives a p-value. To test H0, the sample is centered at zero (its mean is subtracted, so the null
is true in the resamples) and you count what fraction of the resampled means is as extreme as the observed one:

```python
x = r["e0258"].to_numpy()
bs = StationaryBootstrap(20, x - x.mean(), seed=7)
medias = np.array([media_anual(d[0][0])
                   for d in bs.bootstrap(4000)])
p_bs = np.mean(np.abs(medias) >= abs(media_anual(x)))
print(f"observed annual mean: {100 * media_anual(x):.2f}%")
print(f"bootstrap p: {p_bs:.3f}  (t-test p: "
      f"{stats.ttest_1samp(x, 0).pvalue:.3f})")
```
```output
observed annual mean: 8.49%
bootstrap p: 0.232  (t-test p: 0.215)
```

The two p-values are similar (0.232 versus 0.215) and lead to the same conclusion. On daily returns without
autocorrelation, the bootstrap confirms the t-test; its value shows up when the formula does not exist or the
dependence does, as in the sections that follow.

## Power: how much history you need

The figure at the top answers the inverse question: if the edge exists, with what probability does a test detect it?
4,000 strategies of 1,003 days are simulated with Student's t innovations (fat tails) and four true Sharpes. It is a
seeded simulation, not the course data: that way the answer is known for each one.

```python
rng = np.random.default_rng(211)
n, sims = 1003, 4000
filas = {}
for sr_real in [0, 0.5, 1.0, 1.5]:
    z = rng.standard_t(5, (sims, n)) * np.sqrt(3 / 5)  # variance 1
    x = sr_real / np.sqrt(252) + z
    t = x.mean(1) / (x.std(1, ddof=1) / np.sqrt(n))
    p = 2 * stats.t.sf(np.abs(t), n - 1)
    filas[sr_real] = {
        "rechaza_5%": np.mean(p < 0.05),
        "teorica": stats.norm.cdf(sr_real * np.sqrt(n / 252) - 1.96),
        "p_mediano": np.median(p)}
pd.DataFrame(filas).T.round(3)
```
```output
     rechaza_5%  teorica  p_mediano
0.0       0.056    0.025      0.498
0.5       0.166    0.168      0.294
1.0       0.528    0.514      0.042
1.5       0.850    0.849      0.003
```

With Sharpe 0 (the null), 5.6% of the simulations reject at 5%: the nominal size, with the noise of 4,000
replications. The p-values are **uniform**: each 0.05 bin receives 5% of them, which is what you see in the left panel
of the figure. With Sharpe 1, the power is 52.8%: half of the time a Sharpe 1 strategy tested over four years does not
pass the threshold. With Sharpe 0.5, only 16.6%. The `teorica` column (the normal with mean $SR\sqrt{T}$, ignoring
rejection on the negative side) is almost exactly right.

Solving for $T$ in the condition “expected t = 1.96 + 0.84” (the 0.84 gives 80% power) yields the rule of thumb:

```python
for s in [0.25, 0.5, 1.0, 1.5, 2.0]:
    anos = ((stats.norm.ppf(0.975) + stats.norm.ppf(0.8)) / s) ** 2
    print(f"Sharpe {s:4.2f}: {anos:6.1f} years")
```
```output
Sharpe 0.25:  125.6 years
Sharpe 0.50:   31.4 years
Sharpe 1.00:    7.8 years
Sharpe 1.50:    3.5 years
Sharpe 2.00:    2.0 years
```

With a true Sharpe of 0.5, already a good figure for a real strategy after costs, you would need 31 years of history
to detect it with 80% probability. That is why short backtests with a high Sharpe should be read with suspicion (the
measured Sharpe of the lucky ones is inflated, lesson 2.5.4) and why low-Sharpe strategies are validated with many
assets at once rather than with a long history of only one.

## Overlapping returns: the t that multiplies

Many signals predict a long horizon: “the return over the next 21 sessions”. If that target is computed **every
day**, two consecutive observations share 20 of their 21 sessions. The mean is still correct; the standard error of
the classic formula is not. With overlap $h$, the autocovariances are positive up to lag $h-1$ and the true variance of
the mean is about $h$ times what $s^2/n$ assumes: the naive t comes out inflated by about $\sqrt{h}$.

The **Newey-West** estimator (1987) corrects the variance by adding the autocovariances $\hat\gamma_j$ up to a lag
$L$, with decreasing weights that guarantee the result is positive:

$$
\widehat{\operatorname{Var}}(\overline{y}) = \frac{1}{n}\left[\hat\gamma_0 + 2\sum_{j=1}^{L}\left(1-\frac{j}{L+1}\right)\hat\gamma_j\right]
$$

In statsmodels it is a regression on a constant with `cov_type="HAC"`. First, by hand and with the library, on a
strategy with no edge:

```python
import statsmodels.api as sm

def nw_var_media(y, L):
    y = np.asarray(y) - np.mean(y)
    n = len(y)
    g = [y[j:] @ y[:n - j] / n for j in range(L + 1)]
    lrv = g[0] + 2 * sum((1 - j / (L + 1)) * g[j]
                         for j in range(1, L + 1))
    return lrv / n

h = 21
lr = np.log1p(r["e0001"]).cumsum()
y = (lr.shift(-h) - lr).dropna()       # 21 sessions, every day
ols = sm.OLS(y.to_numpy(), np.ones(len(y))).fit()
hac = sm.OLS(y.to_numpy(), np.ones(len(y))).fit(
    cov_type="HAC", cov_kwds={"maxlags": h})
print(f"autocorrelation at lag 1: {y.autocorr(1):.3f}")
print(f"naive t: {ols.tvalues[0]:.2f}; "
      f"Newey-West t: {hac.tvalues[0]:.2f}")
print(f"standard error by hand: {np.sqrt(nw_var_media(y, h)):.6f}; "
      f"statsmodels: {hac.bse[0]:.6f}")
```
```output
autocorrelation at lag 1: 0.959
naive t: -0.97; Newey-West t: -0.25
standard error by hand: 0.007131; statsmodels: 0.007131
```

The autocorrelation at lag 1 is 0.96, close to the theoretical 20/21. The naive t is about four times the Newey-West
one, and the standard error computed by hand matches the statsmodels one (which uses the same Bartlett weights by
default). In this strategy both t's are far from 2, but the important question is what happens across all of them.

The 995 strategies with no edge make it possible to measure the **false positive rate** of each method on data whose
truth is known:

```python
lr = np.log1p(r[nulas]).cumsum()
Y = (lr.shift(-h) - lr).dropna()

def p_media(y, L=None):
    m = sm.OLS(y, np.ones(len(y)))
    f = m.fit() if L is None else m.fit(
        cov_type="HAC", cov_kwds={"maxlags": L})
    return f.pvalues[0]

tasas = {
    "daily, t": np.mean([stats.ttest_1samp(r[e], 0).pvalue < 0.05
                         for e in nulas]),
    "overlapping, OLS": np.mean([p_media(Y[e].to_numpy()) < 0.05
                                 for e in nulas])}
for L in [10, 21, 42]:
    tasas[f"overlapping, NW L={L}"] = np.mean(
        [p_media(Y[e].to_numpy(), L) < 0.05 for e in nulas])
tasas["no overlap, t"] = np.mean(
    [stats.ttest_1samp(Y[e].iloc[::h], 0).pvalue < 0.05
     for e in nulas])
pd.Series(tasas, name="falsos_positivos").round(3)
```
```output
daily, t                0.044
overlapping, OLS        0.653
overlapping, NW L=10    0.183
overlapping, NW L=21    0.094
overlapping, NW L=42    0.068
no overlap, t           0.039
Name: falsos_positivos, dtype: float64
```

*Fig. — False positives at 5% in 995 no-edge strategies by test: daily returns, 21-session returns computed every day (overlapping) and the same series without overlap.* Horizontal bars with the share of no-edge strategies declared significant at 5%. Daily with t: 4.4%. Overlapping with classical errors, highlighted: 65.3%. Overlapping with Newey-West at 10, 21 and 42 lags: 18.3, 9.4 and 6.8%. No overlap: 3.9%. A vertical line marks the nominal 5%.

With daily returns, the t-test gives 4.4% false positives: the nominal level. With the same strategies and a target
overlapping over 21 sessions, the regression with classic errors declares 65.3% of them significant: two out of three
strategies with no edge “predict” their own return. Newey-West corrects it almost completely, and the choice of $L$
matters: with 10 lags (fewer than the overlap) 18.3% remains; with 21, 9.4%; with 42, 6.8%. Sampling one observation
every 21 sessions (no overlap) brings it back to 3.9%, at the cost of being left with 47 observations.

<aside class="callout" data-kind="riesgo">

Overlap appears without anyone deciding it. A regression of “the 12-month return” on a signal with monthly data has
overlap 12; a backtest that opens a position every day and holds it for 10 days produces daily returns of a portfolio
of 10 overlapping positions; a 5-day *machine learning* target (level 3) has overlap 5. In every case, a t computed as
if the observations were independent is inflated.

</aside>

The bootstrap also fails with overlap if the i.i.d. one is used, and it works with blocks longer than the overlap:

```python
y = Y["e0001"].to_numpy()
ee = {"classic formula": y.std(ddof=1) / np.sqrt(len(y)),
      "Newey-West L=42": np.sqrt(nw_var_media(y, 42))}
for nombre, bs in [
        ("iid bootstrap", IIDBootstrap(y, seed=3)),
        ("blocks mean 63", StationaryBootstrap(63, y, seed=3))]:
    m = [d[0][0].mean() for d in bs.bootstrap(2000)]
    ee[nombre] = np.std(m)
pd.Series(ee, name="error_tipico_media").round(5)
```
```output
classic formula    0.00186
Newey-West L=42    0.00810
iid bootstrap      0.00191
blocks mean 63     0.00792
Name: error_tipico_media, dtype: float64
```

The i.i.d. bootstrap reproduces the error of the classic formula (0.0019) because, by drawing individual days, it
breaks the overlap; the block one, with blocks of 63 sessions on average, gives an error on the same scale as
Newey-West, about four times larger. The rule is the same for both methods: the window (lags or block) has to cover
the dependence.

## Multiple regression with robust errors: carry and momentum in futures

The most frequent case is a predictive regression: the future return of an asset on one or more of today's signals.
`futuros2_continuos.parquet` and `futuros2_carry.parquet` hold 24 synthetic futures markets from 2000 to 2025 with two
planted effects (level 2 README): the expected annual return is
$0.5\,c_{\text{exogeno}} + 0.1\,c_{\text{reversion}} + 0.15\,\sigma\,\operatorname{sign}(r_{252})$, with $\sigma$
the long-run volatility of each market and the last term a time-series momentum. The observed carry, `carry_anual`,
is the sum of the two parts: an exogenous one, with a premium, and another that only discounts the return of the price
to its anchor, with almost no premium. The natural regression is

$$
y_{i,t} = a + b_1\, c_{i,t} + b_2\, \sigma_{i,t}\operatorname{sign}(r_{i,t-252:t}) + \varepsilon_{i,t}
$$

with $y$ the annualized return of the next 21 sessions and $c$ the observed carry. The known answer is
$b_2 \approx 0.15$; $b_1$ is a mix of 0.5 and 0.1 depending on which part of the carry dominates (the README measures
0.15 with month-end data). $\sigma$ is estimated with all returns up to each date (expanding window): it is the
long-run volatility without looking ahead. The observations are daily, so there is an overlap of 21; in addition, the
24 markets share days and factors. The **Driscoll-Kraay** estimator (1998) handles both: for each date it sums the
regressor × residual products of all markets and applies Newey-West to that series. In statsmodels it is
`cov_type="hac-groupsum"` with the date index.

```python
cont = pd.read_parquet("futuros2_continuos.parquet")
carry = pd.read_parquet("futuros2_carry.parquet")
lp = np.log(cont.pivot(index="fecha", columns="mercado",
                       values="cierre_ajustado"))
c = carry.pivot(index="fecha", columns="mercado",
                values="carry_anual")
fwd = (lp.shift(-h) - lp) * 252 / h            # annualized
# long-run σ of each market, without looking ahead
vol = lp.diff().expanding(252).std() * np.sqrt(252)
mom = np.sign(lp - lp.shift(252)) * vol
panel = pd.concat({"y": fwd, "carry": c, "mom": mom}, axis=1)
panel = panel.stack(future_stack=True).dropna()
X = sm.add_constant(panel[["carry", "mom"]])
fecha = panel.index.get_level_values("fecha").factorize()[0]

ols = sm.OLS(panel["y"], X).fit()
dk = sm.OLS(panel["y"], X).fit(
    cov_type="hac-groupsum", cov_kwds={"time": fecha, "maxlags": h})
print(len(panel), "rows,", panel.index.get_level_values(
    "mercado").nunique(), "markets")
pd.DataFrame({"coef": ols.params, "t_OLS": ols.tvalues,
              "t_DK": dk.tvalues}).round(3)
```
```output
150384 rows, 24 markets
        coef   t_OLS   t_DK
const  0.013   5.844  0.808
carry  0.171   8.683  1.596
mom    0.107  10.511  1.834
```

The coefficients recover the answer with error: 0.107 for momentum (truth 0.15) and 0.171 for carry, within the
expected mix and close to the README's 0.15. What changes completely is the confidence. With classic errors, the t's
are 8.68 and 10.51: apparent certainty. With Driscoll-Kraay, 1.60 and 1.83: neither passes the threshold, even though
both effects exist. There are 150,384 rows, but the information is much smaller: about 300 independent monthly
returns per market, and markets that move together in crises. That the honest test does not reject does not say there
is no effect (here you know there is one): it says that 26 years of 24 correlated markets are not enough to prove it
with a t of 2. It is the power of the previous section, now in a panel.

The check without sophisticated formulas is to remove the overlap: one date out of every 21 and errors clustered by
date (`cov_type="cluster"`), which correct the correlation between markets on the same day.

```python
fechas = np.sort(panel.index.get_level_values("fecha").unique())
mensual = panel[panel.index.get_level_values("fecha").isin(
    fechas[::h])]
Xm = sm.add_constant(mensual[["carry", "mom"]])
grupos = mensual.index.get_level_values("fecha").factorize()[0]
om = sm.OLS(mensual["y"], Xm).fit()
cl = sm.OLS(mensual["y"], Xm).fit(
    cov_type="cluster", cov_kwds={"groups": grupos})
print(len(mensual), "rows on", len(fechas[::h]), "dates")
pd.DataFrame({"coef": om.params, "t_OLS": om.tvalues,
              "t_cluster": cl.tvalues}).round(3)
```
```output
7176 rows on 299 dates
        coef  t_OLS  t_cluster
const  0.011  1.100      0.588
carry  0.209  2.288      1.536
mom    0.053  1.126      0.668
```

Without overlap, the classic t of carry drops from 8.68 to 2.29 and that of momentum, from 10.51 to 1.13; clustering
by date, to 1.54 and 0.67. The coefficients move more than with Driscoll-Kraay (0.209 and 0.053: keeping one date out
of every 21 throws away information and leaves a noisier estimate), but the conclusion is the same: the honest
uncertainty is several times the naive one.

What if the signal were useless? A persistent **placebo** regressor, with no relation at all to prices, is the acid
test. 20 AR(1) placebos are simulated with the same persistence as the generator's carry (0.997 daily), each one
independent per market, and you count how many times it comes out “significant”:

```python
rng = np.random.default_rng(2111)
rech = {"OLS": 0, "Driscoll-Kraay": 0, "DK L=63": 0}
for _ in range(20):
    e = rng.standard_normal(lp.shape) * np.sqrt(1 - 0.997**2)
    pl = np.empty(lp.shape)
    pl[0] = rng.standard_normal(lp.shape[1])
    for i in range(1, len(pl)):
        pl[i] = 0.997 * pl[i - 1] + e[i]
    pl = pd.DataFrame(pl, index=lp.index, columns=lp.columns)
    d = pd.concat({"y": fwd, "carry": c, "mom": mom, "pl": pl},
                  axis=1).stack(future_stack=True).dropna()
    Xp = sm.add_constant(d[["carry", "mom", "pl"]])
    tt = d.index.get_level_values("fecha").factorize()[0]
    m = sm.OLS(d["y"], Xp)
    rech["OLS"] += m.fit().pvalues["pl"] < 0.05
    for k, L in [("Driscoll-Kraay", h), ("DK L=63", 63)]:
        f = m.fit(cov_type="hac-groupsum",
                  cov_kwds={"time": tt, "maxlags": L})
        rech[k] += f.pvalues["pl"] < 0.05
print({k: f"{v} of 20" for k, v in rech.items()})
```
```output
{'OLS': '14 of 20', 'Driscoll-Kraay': '1 of 20', 'DK L=63': '1 of 20'}
```

With classic errors, the placebo is “significant” in 14 of 20 simulations. With Driscoll-Kraay, in 1 with 21 lags and
in 1 with 63. Twenty replications are few to measure a rate precisely, but the order of magnitude is clear: with
persistent signals and overlapping targets, classic errors are not an optimistic approximation, they are a discovery
generator.

## Which standard error to use

```mermaid
flowchart TB
  Q1{Dependence?} -- no --> Q2{Several assets?}
  Q1 -- yes --> Q3{Several assets?}
  Q2 -- no --> A[Classic t or iid]
  Q2 -- yes --> B[Cluster by date]
  Q3 -- no --> C[Newey-West L=2h]
  Q3 -- yes --> D[Driscoll-Kraay]
```

| Situation | Standard error | In statsmodels | Check |
|---|---|---|---|
| Daily returns of a strategy | Classic; i.i.d. bootstrap if there is no formula | `fit()` | Block bootstrap: should agree |
| Overlapping target of $h$ sessions, one asset | Newey-West with $L \ge h$ | `cov_type="HAC"` | Non-overlapping subsample |
| Several assets, no overlap | Clustered by date | `cov_type="cluster"` | Fama-MacBeth (lesson 2.3.5) |
| Several assets and overlap | Driscoll-Kraay | `cov_type="hac-groupsum"` | No overlap + cluster by date |
| Any statistic without a formula (Sharpe, drawdown) | Block bootstrap | `arch.bootstrap` | Several block lengths |

Three warnings that no row of the table resolves:

- **Selection.** If a hundred variants were tried and the best one is reported, all these p-values are too low
  (lessons 2.1.8 and 2.5.4).
- **Costs.** A significant coefficient on gross returns may not survive the bid-ask spread and commissions (lesson
  2.4.7). Statistical significance is not profitability.
- **Stability.** A t of 3 over 20 years may come from a single crisis. Look at the coefficient by subperiods before
  trusting it (lesson 2.5.6).

## Exercises

1. Repeat the false positive table for the 995 null strategies with a target overlapping over 5 sessions and over 63
   sessions, with classic errors and with Newey-West with $L = h$. How does the problem grow with the overlap?

**Solution.**

```python
for hh in [5, 63]:
    lr = np.log1p(r[nulas]).cumsum()
    Yh = (lr.shift(-hh) - lr).dropna()
    ols_fp = np.mean([p_media(Yh[e].to_numpy()) < 0.05
                      for e in nulas])
    nw_fp = np.mean([p_media(Yh[e].to_numpy(), hh) < 0.05
                     for e in nulas])
    print(f"h = {hh:2d}: OLS {ols_fp:.3f}; NW L=h {nw_fp:.3f}")
```
```output
h =  5: OLS 0.364; NW L=h 0.076
h = 63: OLS 0.805; NW L=h 0.138
```

With overlap 5, classic errors already give 36.4% false positives; with 63, 80.5%: the naive t grows with $\sqrt{h}$
and more and more null strategies exceed the threshold. Newey-West lowers the rate in both cases (7.6% and 13.8%), but
with 63 lags on about 940 observations it stays well above the nominal level: with such a long overlap there are few
independent observations (about 15) and any variance estimator is imprecise. With very long targets, the prudent thing
is to sample without overlap or to use more history.

2. Compute the 95% interval of the **annual Sharpe** of `e0518` and `e0258` with a stationary block bootstrap with a
   mean block length of 20. Does it include the true Sharpe? Does it include zero?

**Solution.**

```python
def sharpe(x):
    return np.sqrt(252) * x.mean() / x.std()

for e in ["e0518", "e0258"]:
    bs = StationaryBootstrap(20, r[e].to_numpy(), seed=11)
    inf, sup = bs.conf_int(sharpe, reps=2000,
                           method="percentile").ravel()
    print(f"{e}: measured Sharpe {sharpe(r[e].to_numpy()):.2f}, "
          f"95% CI [{inf:.2f}, {sup:.2f}], "
          f"true {verdad.loc[e, 'sharpe_real']}")
```
```output
e0518: measured Sharpe 1.58, 95% CI [0.43, 2.77], true 1.5
e0258: measured Sharpe 0.62, 95% CI [-0.43, 1.69], true 0.7
```

Both intervals include the true Sharpe and have a width of about two Sharpe points: it is the standard error of
≈ 0.5 given by the $1/\sqrt{T}$ rule with four years. The one for `e0518` excludes zero; the one for `e0258` includes
it. A measured Sharpe of 1.58 is compatible with a true one of 0.5 or 2.7: four years do not distinguish an excellent
strategy from a mediocre one.

3. A colleague presents a regression of the 12-month return of 50 stocks on a value signal, with monthly data over
   15 years (9,000 rows), and a t of 6 with classic errors. What two problems does the t have and how would you
   recompute it?

**Solution.**

First, overlap: with 12-month returns observed every month, each observation shares 11 months with the next one, and
the t is inflated by about $\sqrt{12} \approx 3.5$. Second, correlation between stocks in the same month: the 50
returns on a date move together, so they are not 50 independent data points. I would recompute it with Driscoll-Kraay
(`cov_type="hac-groupsum"` with the date and at least 12 lags) and check it with a non-overlapping annual subsample
and errors clustered by date. A naive t of 6 can end up at 1 or 2. And then I would ask how many signals were tried
before this one (lesson 2.1.8).

## Self-assessment

1. A strategy has a true annual Sharpe of 1.0 and you test it with four years of daily data. What is the probability that the t-test at 5% declares it significant?

   - A. Almost 100%: a Sharpe of 1 is very high
   - B. 5%
   - C. About 50%: with four years, the expected t-statistic is 1 × √4 = 2, right at the threshold
   - D. 95%

   *Answer C: About 50%: with four years, the expected t-statistic is 1 × √4 = 2, right at the threshold.* The t-statistic of the mean is approximately the annual Sharpe times the square root of the years. With an expected t of 2 and a threshold of 1.96, half of the samples fall below it. In the lesson's simulation, 52.8% of 4,000 samples reject the null.

2. With 21-session returns computed every day (overlapping), why does the t of the mean come out inflated?

   - A. Because 21-day returns are larger
   - B. Because the mean of overlapping returns is biased
   - C. Because the variance of 21-day returns is smaller
   - D. Because two consecutive observations share 20 of their 21 days: they are not independent, and the classic formula counts 21 times more information than there is

   *Answer D: Because two consecutive observations share 20 of their 21 days: they are not independent, and the classic formula counts 21 times more information than there is.* The mean is not biased; its standard error is. The formula s/√n assumes independent observations; with overlap h, the true variance of the mean is about h times what the formula gives. In the 995 strategies with no edge, the naive test gives 65.3% false positives.

3. Newey-West with 21 lags on returns overlapping over 21 sessions gives 9.4% false positives instead of 5%. What explains it best?

   - A. That the Bartlett weights (1 − j/(L+1)) underweight the autocovariances close to lag L; with L equal to the overlap, the standard error falls somewhat short. With L = 2h it drops to 6.8%
   - B. That Newey-West is badly implemented in statsmodels
   - C. That 0 lags should be used
   - D. That the strategies did have an edge

   *Answer A: That the Bartlett weights (1 − j/(L+1)) underweight the autocovariances close to lag L; with L equal to the overlap, the standard error falls somewhat short. With L = 2h it drops to 6.8%.* With overlap h, the autocovariances are positive up to lag h − 1 and Bartlett weights them less and less. It is a known finite-sample bias; more lags reduce it at the cost of more variance in the estimator. Sampling without overlap is the simple check.

4. In the panel of 24 futures, the momentum coefficient has t = 10.51 with OLS and t = 1.83 with Driscoll-Kraay. What changes between the two?

   - A. The estimated coefficient
   - B. Only the standard error: Driscoll-Kraay sums the regressor × residual products of all markets on each date and applies Newey-West to that series, so it accounts for the overlap and for the fact that the 24 markets move together
   - C. The number of observations
   - D. The dependent variable

   *Answer B: Only the standard error: Driscoll-Kraay sums the regressor × residual products of all markets on each date and applies Newey-West to that series, so it accounts for the overlap and for the fact that the 24 markets move together.* The estimator is the same (0.107). OLS treats the 150,384 rows as independent; in reality there are about 300 distinct monthly returns per market and the markets are correlated on the same day. With the honest error, the planted momentum does not reach t = 2.

5. Among 995 strategies with no edge, the best one has t = 3.0 in its daily test and its block bootstrap interval does not include zero. Which conclusion is correct?

   - A. Nothing: the test and the bootstrap are valid for a strategy chosen in advance, not for the best of hundreds. Selection has to be corrected separately (lesson 2.1.8)
   - B. It has an edge: the block bootstrap is robust
   - C. That a 1% level should be used
   - D. That the bootstrap has a bug

   *Answer A: Nothing: the test and the bootstrap are valid for a strategy chosen in advance, not for the best of hundreds. Selection has to be corrected separately (lesson 2.1.8).* No standard error method corrects for selection. With 995 nulls you expect about 50 with p < 0.05 and some with p < 0.005. The known answer of the data set confirms it: its true Sharpe is 0.

## Key points

- The t-statistic of the mean is approximately annual Sharpe × √years: with a true Sharpe of 0.5 you would need 31 years to have 80% power.
- Under the null, p-values are uniform; under the alternative they pile up near zero, but with low power many good strategies stay above 0.05.
- The i.i.d. and block bootstraps agree if there is no serial dependence; with overlapping returns, only the block bootstrap (with blocks longer than the overlap) gives the right error.
- Overlapping returns multiply the t by about √h: 65.3% false positives in 995 null strategies. Newey-West with L ≥ h or sampling without overlap corrects it.
- In a panel, same-day correlation between assets also inflates the t: Driscoll-Kraay lowers the momentum t from 10.51 to 1.83 without touching the coefficient, and an effect that exists stops being significant.
- No standard error corrects for selection among many tests: that is lesson 2.1.8.

## Resources

### Research

- [Seasonality in Python III - Statistics on Trading](https://pythonparatrading.com/en/seasonality-intraday-statistics/) · Research-0026 · 13 min

### Documentation

- [statsmodels: OLS.fit and its covariance types (HAC, cluster, hac-groupsum)](https://www.statsmodels.org/stable/generated/statsmodels.regression.linear_model.OLS.fit.html) · statsmodels.org
- [arch: i.i.d., stationary and block bootstraps (documentation)](https://arch.readthedocs.io/en/latest/bootstrap/timeseries-bootstraps.html) · arch.readthedocs.io

### Books

- [Newey, W. K. and West, K. D. (1987): A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica 55(3), 703-708](https://doi.org/10.2307/1913610) · doi.org
- [Politis, D. N. and Romano, J. P. (1994): The Stationary Bootstrap. Journal of the American Statistical Association 89(428), 1303-1313](https://doi.org/10.1080/01621459.1994.10476870) · doi.org
- [Driscoll, J. C. and Kraay, A. C. (1998): Consistent Covariance Matrix Estimation with Spatially Dependent Panel Data. Review of Economics and Statistics 80(4), 549-560](https://doi.org/10.1162/003465398557825) · doi.org

### Data

- [Level 2 data: 1,000 synthetic strategies with their ground truth (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/ruido2_estrategias.parquet) · Download · ruido2_estrategias.parquet

Next lesson: [2.1.2 · Stationarity: ADF, KPSS, differencing and fractional differencing](https://pythonparatrading.com/en/python-trading-course/level-2/stationarity-adf-kpss-fractional-differencing/)