---
title: "2.5.6 Parameter sensitivity and robustness"
description: "Plateaus versus peaks: parameter grids on futures with a planted effect and on noise, plateau center and robustness across assets and periods."
url: "https://pythonparatrading.com/en/python-trading-course/level-2/parameter-sensitivity-and-robustness/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/parameter-sensitivity-and-robustness.png"
---
# 2.5.6 · Parameter sensitivity: plateaus versus peaks, robustness across assets and periods
> Every strategy has parameters, and almost any parameter grid has a maximum. What separates an effect from an accident is what surrounds that maximum: a broad plateau that repeats in other markets and other eras, or an isolated peak that fades as soon as you look out of sample.
- Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/)
- Module: 2.5 · Biases and validation
- Duration: 60 min
- Prerequisites: [2.3.1 · Trend following and momentum](https://pythonparatrading.com/en/python-trading-course/level-2/trend-following-and-momentum/), [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/), [2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations](https://pythonparatrading.com/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests/)
## By the end of this lesson
- Build a parameter grid for a strategy and read its Sharpe surface as a map, not as a ranking.
- Measure the smoothness of a surface with the neighbors/peak ratio, the positive fraction, the roughness and the correlation between combinations.
- Apply the rule “choose the center of a plateau” and check, in and out of sample, what it gains over choosing the peak.
- Check the robustness of the chosen parameters across markets, sectors, another asset class and periods, and know what it means when each one fails.
## An optimum is not a result
Every strategy has parameters: the length of an average, a threshold, a volatility window, a rebalancing period.
Trying several combinations and keeping the best one is unavoidable and is not, in itself, a mistake. The mistake is
reading the grid as a ranking and believing that the number in the first row is what the strategy will deliver in the
future. Lesson 2.5.4 corrects that number for the number of tests; this one looks at the shape of the grid, which says
something the number alone does not.
The intuition is mechanical. If there is a real effect that the strategy captures, changing a parameter a little
changes little of what it captures: a 125-session average and a 150-session one are very similar. The result forms a
**plateau**. If there is no effect, the maximum of the grid is a lottery winner: a combination that happened to catch
the good days of this sample, and its neighbors, which caught other days or none, do not resemble it. The result is a
**peak**.
```mermaid
flowchart TB
G[Grid] --> S[Sharpe surface]
S --> M[Smoothness metrics]
M --> C[Plateau center]
C --> F[Out of sample]
F --> A[Other assets]
A --> P[Other periods]
P --> D[Candidate]
```
## The grid on the futures: a plateau
The strategy is a moving-average crossover on the 24 futures: in each market, long if the average of the last $f$
sessions is above that of the last $s$ and short if it is below, with the position scaled by inverse volatility and
executed in the next session (lesson 2.3.1). Eight fast averages by eight slow ones give 64 combinations:
```python
import numpy as np
import pandas as pd
def sharpe(x):
x = np.asarray(x, dtype=float)
x = x[~np.isnan(x)]
return np.mean(x) / np.std(x, ddof=1) * np.sqrt(252)
fut = pd.read_parquet("futuros2_continuos.parquet")
px = fut.pivot(index="fecha", columns="mercado",
values="cierre_ajustado")
r = px.pct_change()
vol = r.ewm(span=60, min_periods=60).std() * np.sqrt(252)
escala = 0.30 / vol / px.shape[1]
RAPIDAS = [5, 10, 20, 30, 40, 50, 60, 80]
LENTAS = [100, 125, 150, 175, 200, 250, 300, 350]
medias = {n: px.rolling(n).mean() for n in RAPIDAS + LENTAS}
def pnl_cruce(f, s):
"""Per market: long if the fast average is above the slow one."""
signo = np.sign(medias[f] - medias[s])
return ((signo * escala).shift(1) * r).loc["2001-01-02":]
por_mercado = {(f, s): pnl_cruce(f, s) for f in RAPIDAS for s in LENTAS}
carteras = pd.DataFrame({k: v.sum(axis=1)
for k, v in por_mercado.items()})
def rejilla(carteras, desde=None, hasta=None):
sr = carteras.loc[desde:hasta].apply(sharpe)
return sr.unstack().rename_axis(index="rápida", columns="lenta")
g_fut = rejilla(carteras)
g_fut.round(2)
```
```output
lenta 100 125 150 175 200 250 300 350
rápida
5 0.71 0.68 0.73 0.81 0.82 0.90 1.04 1.09
10 0.60 0.72 0.83 0.82 0.83 0.92 1.07 1.08
20 0.67 0.78 0.80 0.87 0.78 0.97 1.06 1.12
30 0.74 0.89 0.86 0.89 0.91 0.97 1.10 1.12
40 0.83 0.87 0.88 0.93 0.91 0.97 1.08 1.03
50 0.86 0.77 0.83 0.90 0.88 0.93 1.04 1.03
60 0.87 0.75 0.89 0.78 0.83 0.86 0.97 0.98
80 0.70 0.72 0.69 0.75 0.83 0.76 1.00 0.97
```
All 64 combinations make money, with a Sharpe between 0.60 and 1.12. There is structure: the surface rises toward the
long slow averages (the 300- and 350-session columns go from 0.97 to 1.12) and drops somewhat with the fast average of
80 and with the short slow ones. But no combination stands alone: it is the plateau in the left panel of the header
figure, where all the squares are blue and of maximum size. The planted effect is the 12-month sign (252 sessions);
none of these rules computes it exactly, all of them capture it in part because long trends move averages of any
length, and the ones that come closest are those with a long slow average: an $n$-session average lags by
$(n - 1)/2$, so the 300-session one looks, on average, about 150 days back.
## The same search on noise: peaks
To see a peak you need a case with no effect. The README's control series, `ruptura` after its break, is i.i.d.
normal noise with zero mean. On it a family of short-term reversal rules is tested, which is where an impatient
researcher's searches usually end up: go against the sign of the $n$-session return and revise the position every $m$
sessions. Nine by six, 54 combinations, all always in the market:
```python
reg = pd.read_parquet("regimenes2.parquet")
ru = (reg[reg["serie"] == "ruptura"].set_index("fecha")["precio"]
.loc["2013-05-22":]) # after the break: i.i.d., mean 0
r_ru = ru.pct_change()
DIAS = [1, 2, 3, 4, 5, 7, 10, 15, 20]
CADA = [1, 2, 3, 5, 7, 10]
def pnl_reversion(n, m):
"""Against the sign of the n-session return; the
position is revised every m sessions."""
signo = -np.sign(ru.pct_change(n))
signo = signo.where(np.arange(len(signo)) % m == 0).ffill()
return (signo.shift(1) * r_ru).loc["2014-01-02":]
carteras_ru = pd.DataFrame({(n, m): pnl_reversion(n, m)
for n in DIAS for m in CADA})
g_ru = rejilla(carteras_ru).rename_axis(index="días",
columns="cada")
g_ru.round(2)
```
```output
cada 1 2 3 5 7 10
días
1 -0.02 0.00 -0.02 0.14 -0.46 -0.05
2 -0.29 -0.26 -0.42 0.01 -0.07 -0.02
3 -0.24 -0.38 -0.26 -0.13 -0.27 0.28
4 -0.11 -0.10 0.12 0.11 -0.46 0.57
5 0.02 0.07 0.05 -0.23 0.13 0.38
7 -0.01 -0.03 -0.21 -0.28 0.00 0.16
10 -0.22 -0.36 -0.13 -0.02 -0.01 0.24
15 -0.01 -0.00 0.27 0.34 0.08 0.19
20 0.25 0.09 -0.21 -0.05 -0.11 0.23
```
The best combination, going against the 4-session return and revising every 10, gives a Sharpe of 0.57 over twelve
years: presentable in a report. It is on the edge of the grid, and its five neighbors are worth 0.28, −0.27, −0.46,
0.13 and 0.38: on average, 0.01. It is the right panel of the header figure: blue and orange squares of all sizes mixed
with no order. There is nothing to capture, and each combination has captured a different piece of chance.
## Measuring smoothness
Looking at the surface comes first, but it pays to summarize it in figures that can be compared across strategies and
stored in the experiment journal (lesson 2.2.4). Four simple ones:
- **Neighbors/peak**: mean of the cells surrounding the maximum (8, or fewer on an edge) divided by the maximum. Close
to 1, plateau; close to 0 or negative, isolated peak.
- **Positive fraction**: what share of the grid makes money. With no effect, around half.
- **Roughness**: mean jump between adjacent cells divided by the standard deviation of the grid. On a white-noise
surface (independent cells with the same dispersion), the mean jump between two cells is
$E|X - Y| = 2\sigma/\sqrt{\pi}$, so the roughness is $2/\sqrt{\pi} \approx 1.13$. A plateau gives quite a bit less.
- **Correlation between combinations**: how similar the return series of the grid are to each other.
```python
def vecinos(a, i, j):
filas, cols = a.shape
return [a[x, y] for x in range(i - 1, i + 2)
for y in range(j - 1, j + 2)
if (x, y) != (i, j) and 0 <= x < filas and 0 <= y < cols]
def suavidad(g):
a = g.to_numpy()
i, j = np.unravel_index(np.argmax(a), a.shape)
v = np.mean(vecinos(a, i, j))
saltos = np.r_[np.abs(np.diff(a, axis=0)).ravel(),
np.abs(np.diff(a, axis=1)).ravel()]
m = {"pico": a[i, j], "media_vecinos": v,
"vecinos/pico": v / a[i, j], "positivas": (a > 0).mean(),
"rugosidad": saltos.mean() / a.std()}
return {"en": f"{g.index[i]}/{g.columns[j]}",
**{k: float(round(x, 2)) for k, x in m.items()}}
pd.DataFrame({"futuros": suavidad(g_fut), "ruido": suavidad(g_ru)})
```
```output
futuros ruido
en 20/350 4/10
pico 1.12 0.57
media_vecinos 1.09 0.01
vecinos/pico 0.97 0.02
positivas 1.0 0.41
rugosidad 0.47 0.94
```
The figures say the same as the chart. In the futures, the peak's neighbors (five, because it is on the edge) keep
97% of its Sharpe, all cells are positive and the roughness is 0.47. In the noise, the neighbors keep 2%, only 41% of
the cells make money and the roughness, 0.94, is close to the 1.13 of white noise.
The fourth metric explains part of the difference, and it is a warning:
```python
def parecido(carteras):
c = carteras.dropna().corr().to_numpy()
n = len(c)
autovalores = np.linalg.eigvalsh(c)
return {"combinaciones": n,
"correlación_media": (c.sum() - n) / (n * (n - 1)),
"primer_componente": autovalores[-1] / n}
pd.DataFrame({"futuros": parecido(carteras),
"ruido": parecido(carteras_ru)}).round(2)
```
```output
futuros ruido
combinaciones 64.00 54.00
correlación_media 0.77 0.32
primer_componente 0.78 0.35
```
The 64 variants of the moving-average crossover have an average correlation of 0.77, and a single principal component
explains 78% of their variance: they are almost the same strategy with small differences. The reversal ones are much
less alike (0.32). Part of the smoothness of the futures comes from there, and not from the effect: with parameters
this correlated, a surface can come out smooth on noise too, especially with long averages (exercise 2 checks it).
Smoothness is **necessary** to trust a set of parameters, but not **sufficient**. This correlation is also what has to
be taken into account when counting the number of tests for the Deflated Sharpe (lesson 2.5.4): 64 nearly identical
variants are not 64 independent attempts.
## Choosing the center of the plateau
The rule of thumb that follows: **do not choose the maximum of the grid; choose the center of the plateau**. One
explicit way is to smooth the surface (each cell becomes the mean of its 3 × 3 neighborhood) and keep the maximum of
the smoothed one. The chosen cell is the one best surrounded, and its smoothed value is a more prudent estimate of what
can be expected.
$$
\overline{S}_{ij} = \frac{1}{9}\sum_{a=-1}^{1}\sum_{b=-1}^{1} S_{i+a,\,j+b}
$$
The edges of the grid do not have a full neighborhood, and a mean with fewer neighbors is noisier: they are left out.
If the smoothed maximum ended up stuck to the edge, the conclusion would be to widen the grid, not to choose the edge.
```python
def suavizar(g):
"""Mean of each cell with its 8 neighbors. The edges are left
empty: they do not have a full neighborhood."""
a = g.to_numpy()
f, c = a.shape
s = np.full((f, c), np.nan)
for i in range(1, f - 1):
for j in range(1, c - 1):
s[i, j] = a[i - 1:i + 2, j - 1:j + 2].mean()
return pd.DataFrame(s, index=g.index, columns=g.columns)
def elegir(g):
"""Raw peak and plateau center (maximum of the smoothed grid)."""
return g.stack().idxmax(), suavizar(g).stack().idxmax()
def txt(k):
return f"{k[0]}/{k[1]}"
for nombre, g in (("futuros", g_fut), ("ruido", g_ru)):
pico, centro = elegir(g)
print(f"{nombre}: peak {txt(pico)} ({g.loc[pico]:.2f}); "
f"center {txt(centro)} ({g.loc[centro]:.2f}, "
f"neighborhood {suavizar(g).loc[centro]:.2f})")
```
```output
futuros: peak 20/350 (1.12); center 30/300 (1.10, neighborhood 1.05)
ruido: peak 4/10 (0.57); center 15/7 (0.08, neighborhood 0.10)
```
In the futures, the raw peak (20/350) is on the edge, and the center of the plateau is the 30/300 crossover: Sharpe
1.10, with a neighborhood that averages 1.05. It is in the last interior column: the plateau keeps rising toward the
slow averages and the grid falls short on that side. It is the case in which the rule asks to widen the grid; exercise
1 does it, and the center barely moves. In the noise, the center is 15/7, with 0.08 and a neighborhood of 0.10: the
rule finds no plateau, and that is already a result. Had the noise researcher used this rule, they would have seen
that their 0.57 “best strategy” is surrounded by nothing.
The test of the rule is out of sample. Choose with the first half of the data and measure on the second: in the
futures, 2001-2012 against 2013-2025; in the noise, 2014-2019 against 2020-2025 (lesson 2.5.2).
```python
casos = {
"futuros": (carteras, ("2001", "2012"), ("2013", "2025")),
"ruido": (carteras_ru, ("2014", "2019"), ("2020", "2025")),
}
filas, dentro_fuera = {}, {}
for nombre, (c, ins, oos) in casos.items():
g_in, g_out = rejilla(c, *ins), rejilla(c, *oos)
dentro_fuera[nombre] = (g_in, g_out)
pico, centro = elegir(g_in)
for regla, k in (("pico", pico), ("centro", centro)):
filas[(nombre, regla, txt(k))] = {"dentro": g_in.loc[k],
"fuera": g_out.loc[k]}
filas[(nombre, "mediana", "todas")] = {
"dentro": g_in.stack().median(),
"fuera": g_out.stack().median()}
rho = np.corrcoef(g_in.to_numpy().ravel(),
g_out.to_numpy().ravel())[0, 1]
print(f"{nombre}: in-out correlation of the grid {rho:.2f}")
pd.DataFrame(filas).T.round(2)
```
```output
futuros: in-out correlation of the grid 0.27
ruido: in-out correlation of the grid 0.15
dentro fuera
futuros pico 20/175 0.96 0.79
centro 20/300 0.89 1.23
mediana todas 0.77 0.99
ruido pico 5/10 0.63 0.13
centro 15/7 0.21 -0.05
mediana todas -0.04 -0.05
```
*Fig. — In-sample and out-of-sample Sharpe of each combination: futures (2001-2012 against 2013-2025) and noise (2014-2019 against 2020-2025).* Scatter with 64 blue futures points top right, almost all above the diagonal (in-sample 0.39 to 0.96; out-of-sample 0.66 to 1.37), and 54 grey noise points around the origin. The diagonal marks in-sample equal to out-of-sample. The noise peak, highlighted, goes from 0.63 in-sample to 0.13 out-of-sample; the futures peak, from 0.96 to 0.79, and the futures centre, from 0.89 to 1.23.
In the futures, everything holds: the peak chosen in 2001-2012 (20/175, Sharpe 0.96) gives 0.79 in 2013-2025; the
center (20/300, 0.89), 1.23; the median of the 64 combinations goes from 0.77 to 0.99. The second half is more
favorable to trend than the first, and above all to the long slow averages, which is where the center falls. This
should not be read as the center always winning: on a real plateau, choosing the center instead of the peak costs
little or nothing out of sample, and its in-sample value is not the maximum of a draw.
In the noise, the 2014-2019 peak (5/10, 0.63) gives 0.13 in 2020-2025, and the median of the grid is −0.04 in sample
and −0.05 out of sample: the whole grid is worth zero and the peak was the lottery prize. The correlation between the
in-sample grid and the out-of-sample one, on the other hand, separates the two cases little: 0.27 in the futures and
0.15 in the noise. In the futures the shape changes (in sample the surface is flat, with the peak at 175; out of
sample the long slow averages dominate) even though the level holds. What separates the two cases is the level of the
whole grid out of sample (median of 0.99 against −0.05), not the order of its cells.
```mermaid
flowchart TB
S{Broad plateau?} -- no --> X[Discard]
S -- yes --> O{Holds out of sample?}
O -- no --> X
O -- yes --> A{Other markets?}
A -- no --> R[Revisit hypothesis]
A -- yes --> P{Other periods?}
P -- no --> R
P -- yes --> C[Candidate]
```
## Robustness across assets
A real effect should appear in every market where the hypothesis says it is, not only in the aggregate portfolio. A
24-market portfolio can have a good Sharpe because two or three extraordinary markets make up for twenty mediocre ones,
and then the result depends on those two or three staying the same. With the parameters of the plateau center:
```python
esp = pd.read_csv("futuros2_especificaciones.csv",
index_col="mercado")
CENTRO = elegir(g_fut)[1]
pm = por_mercado[CENTRO]
por_merc = pm.apply(sharpe)
tabla = pd.DataFrame({
"mercados": por_merc.groupby(esp["sector"]).size(),
"positivos": (por_merc > 0).groupby(esp["sector"]).sum(),
"sharpe_medio": por_merc.groupby(esp["sector"]).mean(),
"sharpe_sector": pm.T.groupby(esp["sector"]).sum().T.apply(sharpe),
})
print(f"Crossover {txt(CENTRO)}: {(por_merc > 0).sum()} of 24 markets "
f"with a positive Sharpe")
tabla.round(2)
```
```output
Crossover 30/300: 21 of 24 markets with a positive Sharpe
mercados positivos sharpe_medio sharpe_sector
sector
agricolas 4 4 0.46 0.85
bonos 5 5 0.38 0.53
divisas 3 2 0.15 0.23
energia 3 2 0.15 0.22
metales 4 3 0.17 0.29
renta_variable 5 5 0.46 0.59
```
Twenty-one of the 24 markets are positive (the three negative ones are in currencies, energy and metals), and so are
all the sectors, as portfolios: from 0.22 in energy to 0.85 in agriculturals. The average Sharpe per market (from
0.15 to 0.46) is much lower than that of the portfolio (1.10): diversification across weakly correlated markets does
most of the work. A single market giving a Sharpe of 0.1 or 0.2 is not a red flag; most of them coming out negative
would be.
The toughest check is a different asset class. Level 2 has ten spot currencies in `fx2_carry.parquet`, and its README
plants no momentum in them: their known response is a carry (spot depreciates by only 30% of the rate differential,
so carry earns 70%) and some carry crashes. The same rule, without touching a parameter:
```python
fx = pd.read_parquet("fx2_carry.parquet")
spot = fx.pivot(index="fecha", columns="divisa", values="spot")
r_fx = spot.pct_change()
vol_fx = r_fx.ewm(span=60, min_periods=60).std() * np.sqrt(252)
f, s = CENTRO
signo_fx = np.sign(spot.rolling(f).mean() - spot.rolling(s).mean())
pnl_fx = ((signo_fx * 0.30 / vol_fx / 10).shift(1) * r_fx)
pnl_fx = pnl_fx.loc["2006-01-02":]
print(f"Currencies, crossover {txt(CENTRO)}: "
f"Sharpe {sharpe(pnl_fx.sum(axis=1)):.2f}")
print(pnl_fx.apply(sharpe).round(2).sort_values().to_dict())
```
```output
Currencies, crossover 30/300: Sharpe -0.03
{'EST': -0.6, 'ISL': -0.16, 'JAR': -0.14, 'HUR': -0.11, 'GAL': -0.06, 'BOR': 0.03, 'ALV': 0.15, 'DUN': 0.23, 'FOR': 0.24, 'CAS': 0.29}
```
*Fig. — Cross-asset robustness of the 30/300 crossover: Sharpe of each market (ticks) and of each sector as a portfolio (dots), with the fx2 spot currencies, which do not carry the effect.* Strip chart by sector. In the six futures sectors almost every tick is right of zero (21 of 24 markets positive) and the sector portfolios range from 0.22 in energy to 0.85 in agriculture. In the ten spot currencies, with no planted effect, the ticks fall on both sides of zero, from −0.60 to 0.29, and their portfolio gives −0.03. The highlighted line is the full futures portfolio, 1.10.
On spot currencies the rule gives −0.03: five currencies with a negative Sharpe and five with a positive one, from
−0.60 to 0.29. The spot drift from the rate differential is persistent and a trend rule could follow it at times, but
here it picks up nothing: far from the 1.10 of the futures. This does **not** invalidate the strategy: it confirms
that the effect is where the hypothesis says and nowhere else. Robustness across assets is required within the domain
of the hypothesis (lesson 2.2.1). In real markets, the literature documents time-series momentum in index, bond,
commodity and currency futures[^1]; an effect that appeared in only one of those groups, or only in the aggregate
portfolio, would be a signal to revisit the hypothesis before trading.
## Robustness across periods
The last axis is time: parameters that are good over 25 years may be so only because one decade was extraordinary.
Split the sample into segments and look at the plateau center and the worst combination of the grid in each one:
```python
PICO_RU = elegir(dentro_fuera["ruido"][0])[0]
tramos = {"2001-05": ("2001", "2005"), "2006-10": ("2006", "2010"),
"2011-15": ("2011", "2015"), "2016-20": ("2016", "2020"),
"2021-25": ("2021", "2025")}
periodos = pd.DataFrame({
"futuros_centro": {k: sharpe(carteras[CENTRO].loc[a:b])
for k, (a, b) in tramos.items()},
"futuros_peor": {k: carteras.loc[a:b].apply(sharpe).min()
for k, (a, b) in tramos.items()},
})
bloques_ru = {"2014-16": ("2014", "2016"), "2017-19": ("2017", "2019"),
"2020-22": ("2020", "2022"), "2023-25": ("2023", "2025")}
print(periodos.round(2))
print(f"Noise, 2014-2019 peak ({txt(PICO_RU)}):",
{k: float(round(sharpe(carteras_ru[PICO_RU].loc[a:b]), 2))
for k, (a, b) in bloques_ru.items()})
```
```output
futuros_centro futuros_peor
2001-05 1.27 0.56
2006-10 0.62 -0.41
2011-15 1.25 0.28
2016-20 0.88 0.14
2021-25 1.47 0.72
Noise, 2014-2019 peak (5/10): {'2014-16': 0.33, '2017-19': 0.93, '2020-22': 0.13, '2023-25': 0.13}
```
The 30/300 crossover is positive in all five five-year periods, from 0.62 in 2006-2010 to 1.47 in 2021-2025. And the
worst of the 64 combinations is negative in only one (−0.41 in 2006-2010): the plateau is not an accident of one era.
The strongest five-year period contains the bond sell-off that the generator's script plants in 2022, and the first
(1.27), the 2000-2002 equity decline: stretches of long trend, which is exactly what a trend rule should capture.
The noise peak tells the opposite story: 0.33 and 0.93 in the two three-year periods it was chosen with, and 0.13 and
0.13 in the next two. The in-sample improvement was the selection criterion; out of sample, only noise remains.
Summary for the experiment journal of a candidate strategy:
| Test | 30/300 crossover on futures | 5/10 reversal on noise |
|---|---|---|
| Neighbors/peak | 0.97 | 0.02 |
| Positive cells | 100% | 41% |
| Roughness (white noise: 1.13) | 0.47 | 0.94 |
| In / out of sample (chosen in sample) | 0.89 / 1.23 (20/300) | 0.63 / 0.13 |
| Markets with a positive Sharpe | 21 of 24 | Not applicable (one series) |
| Positive periods | 5 of 5 | 4 of 4, falling from 0.93 to 0.13 |
## Exercises
1. Widen the futures grid with slow averages of 400 and 500 sessions and a fast one of 90. Does the center of the
plateau move? Do the smoothness metrics change?
**Solution.**
```python
RAPIDAS2 = RAPIDAS + [90]
LENTAS2 = LENTAS + [400, 500]
for n in RAPIDAS2 + LENTAS2:
if n not in medias:
medias[n] = px.rolling(n).mean()
carteras2 = pd.DataFrame({
(f, s): pnl_cruce(f, s).sum(axis=1)
for f in RAPIDAS2 for s in LENTAS2}).loc["2003-01-02":]
g2 = rejilla(carteras2)
pico2, centro2 = elegir(g2)
print(f"peak {txt(pico2)} ({g2.loc[pico2]:.2f}); "
f"center {txt(centro2)} ({g2.loc[centro2]:.2f})")
print(suavidad(g2))
```
```output
peak 20/350 (1.16); center 20/350 (1.16)
{'en': '20/350', 'pico': 1.16, 'media_vecinos': 1.12, 'vecinos/pico': 0.96, 'positivas': 1.0, 'rugosidad': 0.5}
```
When the grid is widened (and started in 2003 so that the 500-session average exists from the beginning), the maximum
stays at 20/350 with 1.16, now surrounded on all sides, and peak and center coincide: one cell away from the 30/300 of
the original grid. The 400- and 500-session averages do not improve on the 350 one: the plateau tops out there, at the
height of the year used by the planted effect. The metrics barely change: neighbors/peak 0.96, all cells positive and
roughness 0.50. That is what is expected of a plateau: adding parameters around it neither breaks it nor moves it far.
A peak, on the other hand, usually disappears or changes place when the grid or the period changes.
2. Compute the moving-average crossover grid on the memoryless twin of the futures from lesson 2.5.5 (the same
sessions in a different order) and its smoothness metrics. Is smoothness enough to tell it apart from the futures?
**Solution.**
```python
rng = np.random.default_rng(2028)
orden = rng.permutation(len(r) - 1)
r_g = pd.DataFrame(r.to_numpy()[1:][orden], index=r.index[1:],
columns=r.columns)
px_g = (1 + r_g).cumprod()
medias_g = {n: px_g.rolling(n).mean() for n in RAPIDAS + LENTAS}
vol_g = r_g.ewm(span=60, min_periods=60).std() * np.sqrt(252)
esc_g = 0.30 / vol_g / 24
carteras_g = pd.DataFrame({
(f, s): ((np.sign(medias_g[f] - medias_g[s]) * esc_g).shift(1)
* r_g).loc["2001-01-02":].sum(axis=1)
for f in RAPIDAS for s in LENTAS})
g_g = rejilla(carteras_g)
print(suavidad(g_g))
print({k: round(float(v), 2) for k, v in parecido(carteras_g).items()})
```
```output
{'en': '80/150', 'pico': 0.12, 'media_vecinos': -0.02, 'vecinos/pico': -0.18, 'positivas': 0.22, 'rugosidad': 0.68}
{'combinaciones': 64.0, 'correlación_media': 0.75, 'primer_componente': 0.76}
```
The twin gives a **smooth** surface (roughness 0.68, far from the 1.13 of white noise, and an average correlation of
0.75 between combinations), but flat and empty: only 22% of the cells are positive and the peak, 80/150, is worth 0.12
with neighbors that average −0.02. Smoothness alone does not tell an effect from its absence: with long, highly
correlated averages, any series gives a smooth surface. What sets the futures apart is that the plateau is smooth **and
high**: all cells positive and neighbors that keep 97% of the peak. That is why the metrics are read together.
3. A colleague presents a strategy with a Sharpe of 1.4 for the best of 300 parameter sets. Their grid has a smooth
shape, but it has only been tested on one country's index between 2010 and 2020. Which three checks would you ask
for before spending more time on it, and what result from each one would make you discard it?
**Solution.**
First, the center of the plateau and its smoothed value, with the edges left out, and the in- and out-of-sample
comparison choosing the parameters with the first part only: if the center drops well below 1.4 or the out-of-sample
grid does not resemble the in-sample one, the 1.4 was the maximum of a lottery. Second, robustness across assets where
their hypothesis predicts the effect (other indices, other countries): if it only works on the one that was
optimized, there is no general effect. Third, robustness across periods, with segments before 2010 and after 2020: if
the whole result comes from a few years, it depends on a regime. And, in parallel, the real number of tests (300,
corrected for their correlation) for the Deflated Sharpe of lesson 2.5.4. A smooth grid is not enough: neighboring
parameters are correlated and smoothness can also appear with no effect.
[^1]: Moskowitz, Ooi and Pedersen (2012), “Time series momentum”, Journal of Financial Economics 104(2), 228-250,
[doi:10.1016/j.jfineco.2011.11.003](https://doi.org/10.1016/j.jfineco.2011.11.003): according to its abstract,
significant momentum in every one of the 58 liquid futures studied, across indices, currencies, commodities and
bonds.
## Self-assessment
1. In the futures grid, the peak (20/350, on the edge) has a Sharpe of 1.12 and its five neighbors average 1.09. In the noise, the peak (4/10) has 0.57 and its neighbors, 0.01. What does that difference say?
- A. That in the futures the result does not depend on the exact value of the parameters (plateau) and in the noise it does: a peak surrounded by zeros is almost always chance
- B. That the noise has a narrower but equally good peak
- C. That finer grids are needed on the noise
- D. Nothing: only the Sharpe of the peak matters
*Answer A: That in the futures the result does not depend on the exact value of the parameters (plateau) and in the noise it does: a peak surrounded by zeros is almost always chance.* If the effect is real, nearby parameters capture almost the same thing and give similar results. A maximum whose neighbors earn nothing is what chance produces when you test a lot: the neighbors/peak ratio is 0.97 in the futures and 0.02 in the noise.
2. The 64 combinations of the futures grid have an average correlation of 0.77 with each other. What does that imply?
- A. That the grid is badly built
- B. That there are 64 independent tests
- C. That the strategy has no edge
- D. That they are, in practice, few distinct strategies: it is part of why the surface is smooth, and it is what has to be taken into account when counting the tests for the Deflated Sharpe
*Answer D: That they are, in practice, few distinct strategies: it is part of why the surface is smooth, and it is what has to be taken into account when counting the tests for the Deflated Sharpe.* A first component that explains 78% of the common variance shows that the 64 variants move almost together. The smoothness comes partly from there: it is necessary, but on its own it does not prove an effect. That is why the out-of-sample check and robustness across markets and periods are added.
3. Why does the lesson's `suavizar` function leave the edges of the grid empty?
- A. To keep the code shorter
- B. Because the edges are always worse
- C. Because an edge cell has no full neighborhood: its mean would use fewer neighbors and come out noisier. If the smoothed maximum falls on the edge, the grid has to be widened
- D. Because that way the geometric center is always chosen
*Answer C: Because an edge cell has no full neighborhood: its mean would use fewer neighbors and come out noisier. If the smoothed maximum falls on the edge, the grid has to be widened.* With fewer neighbors, the edge mean is less reliable and tends to win by chance. In the futures grid the raw peak (20/350) is on the edge and the surface rises toward the long slow averages: the signal is to widen the grid. Once widened (exercise 1), the maximum stays at 20/350, surrounded on all sides.
4. Chosen with the 2001-2012 data, the futures peak (20/175) goes from 0.96 to 0.79 out of sample and the plateau center (20/300), from 0.89 to 1.23. In the noise, the peak goes from 0.63 to 0.13. What does the center rule teach?
- A. That the center always earns more than the peak out of sample
- B. That it does not depend on the luck of one cell: on a real plateau it costs little or nothing against the peak, and on noise it reveals that there is no plateau (the center's neighborhood is worth 0.10)
- C. That the longest slow average should always be chosen
- D. That it avoids transaction costs
*Answer B: That it does not depend on the luck of one cell: on a real plateau it costs little or nothing against the peak, and on noise it reveals that there is no plateau (the center's neighborhood is worth 0.10).* On a real plateau, peak and center give similar results out of sample; here the center even wins, because out of sample the long slow averages dominate and the center falls in that area, but that is not a rule. What the rule always gives is an alarm: if the center is mediocre, the peak was a fluke.
5. The 30/300 crossover gives a positive Sharpe in 21 of the 24 futures, but −0.03 on the spot currencies of `fx2_carry`. How is it interpreted?
- A. The strategy is not robust: it has to be discarded
- B. The `fx2_carry` currencies do not carry the planted momentum: robustness across assets is required where the hypothesis says the effect is, and a null result outside that domain is consistent with it
- C. Other parameters have to be optimized for the currencies
- D. Spot currencies have higher costs
*Answer B: The `fx2_carry` currencies do not carry the planted momentum: robustness across assets is required where the hypothesis says the effect is, and a null result outside that domain is consistent with it.* A robustness test makes sense when the hypothesis predicts what should happen. The level 2 README plants momentum in the futures and not in spot FX; that the rule does not work there does not invalidate it. What would invalidate it is failing in most of the futures.
## Key points
- A parameter surface is read as a map: what matters is the shape around the maximum, not the maximum.
- In the futures with a planted effect, the 64 combinations give a positive Sharpe (0.60 to 1.12) and the peak's neighbors average 97% of its value; in the noise, the 0.57 peak has neighbors that average 0.01 and only 41% of the cells are positive.
- Roughness (mean jump between adjacent cells divided by the standard deviation of the grid) is 0.47 on the plateau and 0.94 on the noise, close to the 1.13 of a white-noise surface.
- Rule of thumb: choose the center of the plateau (the maximum of the smoothed surface, away from the edges) and widen the grid if it sticks to one. Out of sample, in the noise the peak falls from 0.63 to 0.13; in the futures, peak and center give 0.79 and 1.23.
- Smoothness is necessary but not sufficient: neighboring parameters are highly correlated (0.77 on average in the futures). The next test is robustness across markets, sectors and periods.
- Robustness across assets is required where the hypothesis predicts the effect: 21 of 24 futures positive; on spot currencies, with no planted momentum, the same rule gives −0.03.
## Resources
### Research
- [Intraday Seasonality IV - Field Search](https://pythonparatrading.com/en/seasonality-intraday-optimization/) · Research-0027 · 7 min
- [Your first system in Python: moving-average crossover on SPY](https://pythonparatrading.com/en/first-system-moving-average-crossover/) · Research-0018 · 7 min
### Documentation
- [pandas: DataFrame.rolling (moving averages of the grid)](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.rolling.html) · pandas.pydata.org
### Books
- [Bailey, D. H., Borwein, J., López de Prado, M. and Zhu, Q. J. (2016): The probability of backtest overfitting. The Journal of Computational Finance](https://doi.org/10.21314/JCF.2016.322) · doi.org
- [Harvey, C. R. and Liu, Y. (2015): Backtesting. The Journal of Portfolio Management 42(1), 13-28](https://doi.org/10.3905/jpm.2015.42.1.013) · doi.org
- [Moskowitz, T., Ooi, Y. H. and Pedersen, L. H. (2012): Time series momentum. Journal of Financial Economics 104(2), 228-250](https://doi.org/10.1016/j.jfineco.2011.11.003) · doi.org
### Data
- [Level 2 data: continuous futures, 24 markets, 2000-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/futuros2_continuos.parquet) · Download · futuros2_continuos.parquet
Previous lesson: [2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations](https://pythonparatrading.com/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests/)
Next lesson: [2.5.7 · Synthetic data for stress testing: GBM, GARCH and bootstrapping](https://pythonparatrading.com/en/python-trading-course/level-2/synthetic-data-for-stress-testing/)