Research journal and experiment tracking against data snooping.

Every look at the data is a test

Data snooping occurs when the same data set is used more than once for inference or to choose a model11Note 1 · White (2000), in the resources, defines data snooping this way and proposes a test (the reality check) of the null hypothesis that the best model of a search does not beat a benchmark.. In trading it is almost unavoidable: there is only one price history and everyone uses it. The mechanism is simple and requires no bad faith:

1. An idea is tested. The backtest comes out so-so.
2. A parameter, a filter, the universe or the period is changed. It comes out somewhat better.
3. This is repeated until it comes out well, and the last result is presented.

Each step, on its own, is reasonable. The problem is the whole: the final result is the best of N attempts, and the best of N attempts has an upward bias even if none of them has an edge. A Sharpe of 1.0 that comes out on the first try and one of 1.0 that comes out after 200 attempts do not mean the same thing, and in the final report they cannot be told apart unless someone has counted the attempts.

This problem has statistical tools (multiple-testing correction, lesson 2.1.8; Deflated Sharpe and PBO, lesson 2.5.4; out-of-sample validation, lessons 2.5.2 and 2.5.3), but all of them need a prior input that only exists if it has been recorded: how many tests were made and which ones. That is what this lesson is about.

The research loop

A disciplined research process separates what is decided before looking from what is looked at afterwards:

flowchart TB
  H[Hypothesis] --> D[Design]
  D --> R[Pre-registration]
  R --> E[Execution]
  E --> A[Record result]
  A --> X{Decision}
  X -- discard --> H
  X -- adjust --> D
  X -- candidate --> V[Validate outside]

The key point is pre-registration: the hypothesis, the data, the code and the parameters are recorded before seeing the result. Then the result is recorded, however it comes out, and the decision with its reason. Each turn of the loop (each “adjust”) is one more test in the count. What is recorded for each experiment:

A minimal log in code

Hashes turn large things (a data table, a function) into a short string that changes if any detail changes. SHA-256 from the standard library (hashlib) is enough; of the 64 hexadecimal characters, 16 are kept:

import hashlib
import inspect
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path

import numpy as np
import pandas as pd

def sha(texto):
    return hashlib.sha256(texto.encode()).hexdigest()

def huella_datos(obj):
    """Hash of the contents (values and index) of a table or array."""
    if isinstance(obj, np.ndarray):
        return sha(obj.tobytes().hex())[:16]
    h = pd.util.hash_pandas_object(obj, index=True).to_numpy()
    return sha(h.tobytes().hex())[:16]

def huella_codigo(fn):
    """Hash of a function's source code."""
    return sha(inspect.getsource(fn))[:16]

a = pd.DataFrame({"x": [1.0, 2.0, 3.0]})
b = a.copy()
b.loc[2, "x"] = 3.0000001
print(huella_datos(a), huella_datos(a.copy()), huella_datos(b))

1be10aae2e9da811 1be10aae2e9da811 42a5a0bcda12d1b6

The same table gives the same hash; a change in the seventh decimal place of one value gives a completely different one. huella_codigo does the same with the text of a function: changing a space changes the hash. In a real project, the code hash is the git commit (git rev-parse HEAD) plus a flag for whether there were uncommitted changes (lesson 0.2.5).

The log is a JSONL file (one JSON line per experiment) that is only opened for appending. Each line carries the hash of the previous one, like a chain: if someone edits or deletes an old line, the chain breaks. The clock is passed as an argument: in production it would be datetime.now(timezone.utc); here it is a fixed clock that advances one second per call, so that the lesson's output is always the same.

class Registro:
    def __init__(self, ruta, reloj):
        self.ruta, self.reloj = Path(ruta), reloj
        self.ruta.touch()

    def leer(self):
        with self.ruta.open(encoding="utf-8") as f:
            return [json.loads(linea) for linea in f if linea.strip()]

    def _ultimo(self):
        filas = self.leer()
        return filas[-1]["hash"] if filas else "0" * 16

    def anotar(self, **campos):
        reg = {"ts": self.reloj(), **campos, "previo": self._ultimo()}
        cuerpo = json.dumps(reg, sort_keys=True, ensure_ascii=False)
        reg["hash"] = sha(cuerpo)[:16]
        with self.ruta.open("a", encoding="utf-8") as f:
            f.write(json.dumps(reg, ensure_ascii=False) + "\n")
        return reg

    def verificar(self):
        """Index of the first broken record (None if everything matches)."""
        previo = "0" * 16
        for i, reg in enumerate(self.leer()):
            sin_hash = {k: v for k, v in reg.items() if k != "hash"}
            cuerpo = json.dumps(sin_hash, sort_keys=True,
                                ensure_ascii=False)
            roto = sha(cuerpo)[:16] != reg["hash"]
            if reg["previo"] != previo or roto:
                return i
            previo = reg["hash"]
        return None

INICIO = datetime(2026, 10, 21, 9, 0, tzinfo=timezone.utc)

def reloj_fijo(inicio=INICIO):
    t = [inicio]
    def ahora():
        t[0] += timedelta(seconds=1)
        return t[0].isoformat()
    return ahora

Before using it, a test: record three experiments, verify the chain, alter an old result and verify again.

ruta = Path("prueba_registro.jsonl")
ruta.unlink(missing_ok=True)
reg = Registro(ruta, reloj_fijo())
for k in range(3):
    reg.anotar(hipotesis="test", params={"k": k}, sharpe=0.1 * k)
print("valid chain:", reg.verificar() is None)

lineas = ruta.read_text(encoding="utf-8").splitlines()
alterada = json.loads(lineas[1])
alterada["sharpe"] = 2.5                    # “improve” a result
lineas[1] = json.dumps(alterada, ensure_ascii=False)
ruta.write_text("\n".join(lineas) + "\n", encoding="utf-8")
print("first broken record:", reg.verificar())
print(lineas[0])

valid chain: True
first broken record: 1
{"ts": "2026-10-21T09:00:01+00:00", "hipotesis": "test", "params": {"k": 0}, "sharpe": 0.0, "previo": "0000000000000000", "hash": "d610f6b1fe94e347"}

Record 1 now has contents that do not match its hash, and verificar flags it. If its hash were also recomputed, record 2 would no longer match it. To rewrite history without it showing, the whole chain would have to be recomputed from the edited point; publishing the last hash from time to time (in a commit, in a dated message) prevents that too.

Two hundred tests on noise

To see the bias of the best result, an experiment where the answer is known is needed: prices with no edge at all. Ten years of normal, independent daily returns are generated, with zero mean and 1% daily volatility. On them, a typical search: a moving average crossover with 10 fast averages (from 5 to 50 sessions), 10 slow ones (from 60 to 240) and both directions (follow the crossover or go against it), 200 combinations. The selection sample is sessions 260 to 1,259 (four years); the check sample, the next 1,260 (five).

RAPIDAS = list(range(5, 55, 5))
LENTAS = list(range(60, 260, 20))
SELECCION, FUERA = slice(260, 1260), slice(1260, None)

def sharpe(x):
    return x.mean() / x.std() * np.sqrt(252)

def buscar(r):
    """Selection and out-of-sample Sharpe of each combination."""
    p = np.exp(np.cumsum(r))
    cs = np.concatenate([[0.0], np.cumsum(p)])
    media = {w: np.r_[np.full(w - 1, np.nan), (cs[w:] - cs[:-w]) / w]
             for w in set(RAPIDAS) | set(LENTAS)}
    filas = []
    for f in RAPIDAS:
        for s in LENTAS:
            pos = np.sign(media[f] - media[s])
            pos = np.r_[np.nan, pos[:-1]]   # traded the next day
            for direccion in (1, -1):
                x = direccion * pos * r
                filas.append((f, s, direccion, sharpe(x[SELECCION]),
                              sharpe(x[FUERA])))
    return pd.DataFrame(filas, columns=["rapida", "lenta", "direccion",
                                        "sr_sel", "sr_fuera"])

rng = np.random.default_rng(2024)
ruido = rng.normal(0, 0.01, 2520)
prueba = buscar(ruido)
prueba[["sr_sel", "sr_fuera"]].describe().round(3)

        sr_sel  sr_fuera
count  200.000   200.000
mean     0.000     0.000
std      0.312     0.465
min     -0.846    -0.858
25%     -0.214    -0.427
50%      0.000     0.000
75%      0.214     0.427
max      0.846     0.858

The 200 tests are recorded in the log, one by one, with the hash of the code of buscar, that of the data and their parameters. That is what a disciplined researcher would do with every backtest:

ruta = Path("experimentos.jsonl")
ruta.unlink(missing_ok=True)
registro = Registro(ruta, reloj_fijo())
codigo, datos = huella_codigo(buscar), huella_datos(ruido)
for fila in prueba.itertuples():
    registro.anotar(
        hipotesis="H1 moving average crossover on a series with no edge",
        codigo=codigo, datos=datos,
        params={"rapida": fila.rapida, "lenta": fila.lenta,
                "direccion": fila.direccion},
        sr_seleccion=round(fila.sr_sel, 4))

filas = registro.leer()
mejor = max(filas, key=lambda f: f["sr_seleccion"])
print(len(filas), "tests; valid chain:",
      registro.verificar() is None)
print("best:", mejor["params"], mejor["sr_seleccion"])
fuera = prueba.loc[prueba["sr_sel"].idxmax(), "sr_fuera"]
print(f"the same combination out of sample: {fuera:.3f}")

200 tests; valid chain: True
best: {'rapida': 20, 'lenta': 80, 'direccion': 1} 0.8458
the same combination out of sample: 0.177

The summary in the previous cell has a symmetry that is no coincidence: each combination appears with both directions, so the 200 Sharpe ratios are 100 pairs of opposite sign, with a mean of exactly 0. The dispersion across combinations is 0.31 in the selection sample and 0.47 outside it.

The best combination, fast at 20, slow at 80 and following the crossover, has a selection Sharpe of 0.85: with four years, that is a t of 1.7, a result that would look promising in a report. In the next five years the same rule gives 0.18. On data in which, by construction, no rule has an edge.

How much the maximum rises with the number of tests

A single noise history is an anecdote. To measure the bias, the search is repeated on 50 independent histories and, in each one, the 200 tests are run through in a random order, as a researcher would do them, keeping the best Sharpe found up to each attempt:

mejores, fuera_mejor, caminos = [], [], []
for _ in range(50):
    res = buscar(rng.normal(0, 0.01, 2520))
    orden = rng.permutation(len(res))
    sr = res["sr_sel"].to_numpy()[orden]
    caminos.append(np.maximum.accumulate(sr))
    i = res["sr_sel"].idxmax()
    mejores.append(res.loc[i, "sr_sel"])
    fuera_mejor.append(res.loc[i, "sr_fuera"])
medio = np.mean(caminos, axis=0)
print(f"best of 200 in selection: {np.mean(mejores):.2f} on average")
print(f"that combination out of sample: {np.mean(fuera_mejor):.2f} "
      f"(std. dev. {np.std(fuera_mejor):.2f})")
pd.Series(medio[[0, 4, 9, 19, 49, 99, 199]],
          index=[1, 5, 10, 20, 50, 100, 200]).round(3)

best of 200 in selection: 0.93 on average
that combination out of sample: 0.02 (std. dev. 0.47)
1      0.014
5      0.477
10     0.584
20     0.709
50     0.826
100    0.884
200    0.930
dtype: float64

How much should it rise? If the N tests were independent and had no edge, each estimated Sharpe would be approximately normal with mean 0 and standard deviation $\sigma_{SR} \approx 1/\sqrt{T}$, with $T$ the number of years (0.5 with four years of selection). The expected maximum of N independent standard normals is

with $\varphi$ and $\Phi$ the normal density and distribution function; the square root is the extreme value approximation for large N, which overestimates for small N. The integral is computed numerically:

from scipy import integrate, optimize, stats

def max_esperado(n):
    def f(z):
        return z * n * stats.norm.pdf(z) * stats.norm.cdf(z) ** (n - 1)
    return integrate.quad(f, -10, 10)[0]

sigma_sr = np.sqrt(252 / 1000)      # 1000 selection sessions
tabla = pd.DataFrame({
    "independientes": [max_esperado(n) * sigma_sr
                       for n in (1, 5, 10, 20, 50, 100, 200)],
    "raiz_2lnN": [np.sqrt(2 * np.log(n)) * sigma_sr
                  for n in (1, 5, 10, 20, 50, 100, 200)],
    "medias_moviles": medio[[0, 4, 9, 19, 49, 99, 199]],
}, index=[1, 5, 10, 20, 50, 100, 200])
n_efectivo = optimize.brentq(
    lambda n: max_esperado(n) * sigma_sr - np.mean(mejores), 1.5, 200)
print(f"effective number of tests: {n_efectivo:.0f} of 200")
tabla.round(3)

effective number of tests: 19 of 200
     independientes  raiz_2lnN  medias_moviles
1             0.000      0.000           0.014
5             0.584      0.901           0.477
10            0.772      1.077           0.584
20            0.937      1.229           0.709
50            1.129      1.404           0.826
100           1.259      1.523           0.884
200           1.379      1.634           0.930

The figure at the top draws both curves. Three readings:

• The bias is large and comes early. With 10 tests the best selection Sharpe is already 0.58 on average; with 200, 0.93. All of it on data in which the true Sharpe of any rule is exactly 0.
• Out of sample nothing is left. The winning combination of each history has an average out-of-sample Sharpe of 0.02, with a standard deviation of 0.47: what any random rule gives over five years.
• Correlated tests count for less than N. The 200 moving average combinations are very similar to each other, and their maximum grows like that of 19 independent tests (the integral gives 0.94 for 20). That effective number is the one to give to the multiple-testing correction, and estimating it requires having logged all the tests, including the ones that looked alike.

With an edge: the same search on the futures

The same procedure on data with a planted edge: the portfolio of the level's 24 futures, with the same risk per market, and the same 200 combinations. Selection is 2001-2012; the check, 2013-2025. The planted time series momentum (lesson 2.2.2) favors following the crossover, not going against it:

fut = pd.read_parquet("futuros2_continuos.parquet")
aj = fut.pivot(index="fecha", columns="mercado",
               values="cierre_ajustado")
r_f = aj.pct_change()
inv_vol = 1 / r_f.ewm(span=60, min_periods=60).std().shift(1)
medias = {w: aj.rolling(w).mean() for w in set(RAPIDAS) | set(LENTAS)}

def buscar_futuros():
    filas = []
    for f in RAPIDAS:
        for s in LENTAS:
            pos = np.sign(medias[f] - medias[s]).shift(1)
            x = (pos * inv_vol * r_f).mean(axis=1)
            for direccion in (1, -1):
                y = direccion * x
                filas.append((f, s, direccion,
                              sharpe(y.loc["2001":"2012"]),
                              sharpe(y.loc["2013":])))
    return pd.DataFrame(filas, columns=["rapida", "lenta", "direccion",
                                        "sr_sel", "sr_fuera"])

futr = buscar_futuros()
for fila in futr.itertuples():
    registro.anotar(
        hipotesis="H2 moving average crossover on 24 futures",
        codigo=huella_codigo(buscar_futuros), datos=huella_datos(aj),
        params={"rapida": fila.rapida, "lenta": fila.lenta,
                "direccion": fila.direccion},
        sr_seleccion=round(fila.sr_sel, 4))
i = futr["sr_sel"].idxmax()
print("best:", futr.loc[i, ["rapida", "lenta", "direccion"]].tolist(),
      f"selection {futr.loc[i, 'sr_sel']:.2f}, "
      f"outside {futr.loc[i, 'sr_fuera']:.2f}")
futr.groupby("direccion")[["sr_sel", "sr_fuera"]].mean().round(2)

best: [25, 160, 1] selection 0.97, outside 0.83
           sr_sel  sr_fuera
direccion
-1          -0.75     -0.84
 1           0.75      0.84

With an edge, the pattern changes. The best combination (fast at 25, slow at 160, following the crossover) has a Sharpe of 0.97 in selection and 0.83 outside: it drops, but stays far from zero. The average of the 100 combinations that follow the crossover goes the opposite way: 0.75 in selection and 0.84 outside. In these data, 2013-2025 is a better period for trend than 2001-2012 (the level 2 README gives, for a momentum portfolio of the 24 markets, a Sharpe of 0.75 in 2000-2009 and 1.20 in 2010-2019), so the family rises out of sample while the winner drops and ends up below the average. The practical lesson is twofold: the best result is inflated even when there is an edge, and the family's estimate (0.75, which falls short) is less wrong than the maximum's (0.97, which overshoots).

The figure compares the two searches by group: the 200 combinations sorted by their selection Sharpe into ten groups of 20, with each group's average out-of-sample Sharpe. With noise, the groups swing between −0.36 and +0.36 out of sample with no direction at all: the group with the best selection gives +0.07 and the second, −0.36. Having come out better in selection says nothing about the future. With an edge, the line jumps from −0.72 to +0.72 between the fifth and the sixth group: selection gets the direction right, which is the planted edge. Within the combinations that follow the crossover, the out-of-sample Sharpe rises from 0.72 to 0.91 in the first three groups and then stops rising: the group with the best selection (0.88) gives 0.87 outside, less than the third (0.91 with 0.75 in selection). Selection distinguishes something within the family, but at the very top it only ranks noise: choosing the winner by its selection Sharpe adds nothing to choosing among the best of the family.

From the log to the correction

The log now holds 400 experiments from two hypotheses. What it gives the following lessons is the denominator:

todo = pd.DataFrame(registro.leer())
resumen = todo.groupby("hipotesis").agg(
    pruebas=("hash", "size"),
    combinaciones=("params", lambda p: len(
        {json.dumps(x, sort_keys=True) for x in p})),
    datos=("datos", "nunique"),
    mejor_sr=("sr_seleccion", "max"))
print(resumen)
print("valid chain:", registro.verificar() is None)

# p-value of the best H1 test on its own and corrected for the number
# of tests (Bonferroni, lesson 2.1.8), with N = 200 and with effective N
p_sola = stats.norm.sf(mejor["sr_seleccion"] / sigma_sr)
for n in (1, round(n_efectivo), 200):
    print(f"N = {n:>3}: p = {min(1, n * p_sola):.3f}")

                                                    pruebas  combinaciones  datos  mejor_sr
hipotesis
H1 moving average crossover on a series with no...      200            200      1    0.8458
H2 moving average crossover on 24 futures               200            200      1    0.9687
valid chain: True
N =   1: p = 0.046
N =  19: p = 0.874
N = 200: p = 1.000

The table is the summary that should accompany any result: how many tests, of how many distinct combinations and on how many versions of the data (one per hypothesis: if the hash had changed between tests, someone would have touched the data halfway through the search). The best H1 test, looked at on its own, has a one-sided p-value of 0.046: “significant” at 5%. Corrected for the 19 effective tests, 0.874; for the 200 nominal ones, 1. The first figure is the one that appears in a report without a log. The finer corrections (FDR, Deflated Sharpe, PBO) are those of lessons 2.1.8 and 2.5.4; all of them start with this count.

Each experiment goes through a few states, and the log stores the state each one ended in. None disappears:

stateDiagram-v2
  [*] --> Registered
  Registered --> Executed
  Executed --> Discarded
  Executed --> Candidate
  Candidate --> Validated
  Candidate --> Discarded
  Validated --> Paper

Hygiene rules that the log makes possible and that are worth writing in the journal itself:

• The hypothesis comes before the backtest. If a test has no hypothesis, it is recorded as exploration and its result does not count as evidence.
• The metrics and the check period are fixed beforehand. Choosing afterwards the metric that comes out best is another form of multiple testing.
• The check sample is used once. If a candidate fails out of sample and is adjusted so that it passes, the check sample has become a selection sample (lesson 2.5.2).
• Nothing is deleted. The tests that go badly are the denominator.

Exercises

1. Add to the log a method ya_probado(codigo, datos, params) that says whether an identical experiment has already been run. Why is it useful and what does it not detect?

def ya_probado(registro, codigo, datos, params):
    clave = json.dumps(params, sort_keys=True)
    return any(f.get("codigo") == codigo and f.get("datos") == datos
               and json.dumps(f.get("params"), sort_keys=True) == clave
               for f in registro.leer())

print(ya_probado(registro, codigo, datos,
                 {"rapida": 20, "lenta": 60, "direccion": 1}))
print(ya_probado(registro, codigo, datos,
                 {"rapida": 21, "lenta": 60, "direccion": 1}))

True
False

2. Repeat the search on noise with only the “follow the crossover” direction (100 combinations). How do the best selection Sharpe and the effective number change? Why?

mejores_1 = []
for _ in range(50):
    res = buscar(rng.normal(0, 0.01, 2520))
    mejores_1.append(res.loc[res["direccion"] == 1, "sr_sel"].max())
n_ef_1 = optimize.brentq(
    lambda n: max_esperado(n) * sigma_sr - np.mean(mejores_1),
    1.01, 200)
print(f"best of 100: {np.mean(mejores_1):.2f}; "
      f"effective N {n_ef_1:.1f}")

best of 100: 0.59; effective N 5.2

3. A colleague shows you a backtest with a Sharpe of 1.4 over five years and tells you they “only tested three things”. What would you ask for before believing it?