Optuna and the risk of overfitting the validation.

Choosing hyperparameters is fitting too

A boosting model (lesson 5.5.3) has half a dozen hyperparameters that are not learned with the gradient: learning rate, number of leaves, minimum leaf size, fraction of features and of rows, regularization, number of rounds. They are chosen by trying configurations and keeping the one that scores best on a validation. That choice is one more fit, and its training set is the validation: with enough trials, the best configuration is not the best, it is the one that best matched the noise of those particular weeks. It is the same phenomenon as the Sharpe of the best of N backtests (lessons 2.5.1 and 2.5.4) in another disguise.

In a problem with abundant signal (classifying images, for example), the bias is small next to the real differences between configurations. On the level's panel, where the oracle has an IC of 0.03 (lesson 5.1.1), the real differences between reasonable configurations are of the order of the validation noise, and the selection bias can be everything that is measured. The discipline is the usual one: three roles for the data, and the last one is looked at once.

flowchart TB
  A[Search data] --> B[Purged validation]
  B --> C[One IC per trial]
  C --> D[Choose the best]
  D --> E[Retrain]
  E --> F[Holdout period]
  F --> G[A single measurement]

Optuna in five pieces

Optuna (MIT license; version 5.0.0, from September 2026) organizes the search into five pieces:

The default sampler is TPE (Tree-structured Parzen Estimator): after a few random trials (n_startup_trials=10), it splits the trials into good and bad, fits a density to each group and proposes where the ratio between the two is largest. Since 5.0, for a single objective it uses the multivariate version by default (it models the hyperparameters jointly, not one by one) and the constant liar for parallel searches; in addition, create_study no longer accepts positional arguments and importances use PED-ANOVA instead of f-ANOVA (release notes in the resources).

For a study to be reproducible three things are needed: a seed in the sampler, a single process (n_jobs=1, the default of optimize; with several threads the order in which results arrive changes what TPE proposes) and a deterministic model. The first cell checks it with a known response function, a bowl with its minimum at $(2, -1)$:

import optuna

optuna.logging.set_verbosity(optuna.logging.WARNING)

def cuenco(trial):
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -10, 10)
    return (x - 2) ** 2 + (y + 1) ** 2

for nombre, muestreador in [
        ("TPE, seed 0", optuna.samplers.TPESampler(seed=0)),
        ("TPE, seed 0", optuna.samplers.TPESampler(seed=0)),
        ("random, seed 0",
         optuna.samplers.RandomSampler(seed=0))]:
    estudio = optuna.create_study(direction="minimize",
                                  sampler=muestreador)
    estudio.optimize(cuenco, n_trials=40)
    p = estudio.best_params
    print(f"{nombre:<21} best {estudio.best_value:.4f}  "
          f"x={p['x']:.3f} y={p['y']:.3f}")

TPE, seed 0           best 0.0097  x=2.049 y=-0.915
TPE, seed 0           best 0.0097  x=2.049 y=-0.915
random, seed 0        best 0.4073  x=1.404 y=-1.228

The two runs with the same seed give exactly the same result, and in 40 trials TPE gets much closer to the minimum (0.0097) than random search (0.4073). This is the favorable situation: the function has no noise and the difference between two points is always real. In a trading model, the objective function is an IC measured over a few hundred weeks, and each evaluation carries its error.

The test bench

The data are those of the level's synthetic panel (README of public/curso/datos/nivel-5/): weekly features with planted effects of about 5 bp per week and a signal-to-noise ratio of one to seventy. So that the search runs quickly in the runner, 100 stocks chosen with a fixed seed are used (in a project you would use all 400 and more trials). The features and the label are turned into normal ranks within each week (lesson 5.2.4) and the leak revision_analistas is left out.

• Search: 2012-2018, 364 weeks. The validation is a k-fold of 4 blocks of weeks with a one-week purge on each side (the label ret_sig_semana ends at the close of the following week) and a 4-week embargo after each block, as in lesson 5.4.2.
• Holdout: 2019-2025, 365 weeks. In a real project it would not be touched until the end; here it is measured in each trial only to show what happens outside.
• References: an untuned ridge (α = 1, practically OLS with 27,000 rows) and the oracle alfa_verdadero.

import zlib
import numpy as np
import pandas as pd
from scipy.stats import norm
from sklearn.linear_model import Ridge

f = pd.read_parquet("panel5_features.parquet")
e = pd.read_parquet("panel5_etiquetas.parquet",
                    columns=["fecha", "ticker", "ret_sig_semana"])
v = pd.read_parquet("panel5_verdad.parquet",
                    columns=["fecha", "ticker", "alfa_verdadero"])
tickers = sorted(f["ticker"].unique())
rng = np.random.default_rng([2029, zlib.crc32(b"n5_optuna")])
elegidas = [tickers[i] for i in
            np.sort(rng.choice(400, 100, replace=False))]
d = (f[f["ticker"].isin(elegidas)]
     .merge(e, on=["fecha", "ticker"])
     .merge(v, on=["fecha", "ticker"])
     .dropna(subset=["ret_sig_semana"])
     .sort_values(["fecha", "ticker"]).reset_index(drop=True))
FE = [c for c in f.columns[3:] if c != "revision_analistas"]

def g(s):
    """Normal rank within each week (lesson 5.2.4); missing, 0."""
    por = s.groupby(d["fecha"])
    u = (por.rank() - 0.5) / por.transform("count")
    return np.nan_to_num(norm.ppf(u.to_numpy()))

X = np.column_stack([g(d[k]) for k in FE]).astype(np.float32)
y = g(d["ret_sig_semana"])                 # label in ranks
r = d["ret_sig_semana"].to_numpy()
alfa = d["alfa_verdadero"].to_numpy()
fechas = np.unique(d["fecha"])
semana = np.searchsorted(fechas, d["fecha"].to_numpy())
NB = int((fechas < np.datetime64("2019-01-01")).sum())
BUSQ = np.flatnonzero(semana < NB)         # 2012-2018
RES = np.flatnonzero(semana >= NB)         # 2019-2025, holdout
ENT = np.flatnonzero(semana < NB - 1)      # without the last week

def ic_sem(p, filas):
    """Rank IC of each week (100 stocks per week)."""
    P = p.reshape(-1, 100).argsort(1).argsort(1).astype(float)
    R = r[filas].reshape(-1, 100).argsort(1).argsort(1)
    R = R.astype(float)
    P -= P.mean(1, keepdims=True)
    R -= R.mean(1, keepdims=True)
    return (P * R).sum(1) / np.sqrt((P**2).sum(1) * (R**2).sum(1))

def pliegues(s0, s1, k, embargo=4):
    """k blocks of weeks in [s0, s1) with a one-week purge on
    each side and an embargo after the block (lesson 5.4.2)."""
    out = []
    for b in np.array_split(np.arange(s0, s1), k):
        fuera = (semana >= b[0] - 1) & (semana <= b[-1] + 1 + embargo)
        dentro = (semana >= s0) & (semana < s1 - 1)
        out.append((np.flatnonzero(~fuera & dentro),
                    np.flatnonzero((semana >= b[0])
                                   & (semana <= b[-1]))))
    return out

PL = pliegues(0, NB, 4)

def ridge_ic(ent, pru):
    m = Ridge(alpha=1.0).fit(X[ent], y[ent])
    return ic_sem(m.predict(X[pru]), pru)

def resumen(serie):
    se = serie.std() / np.sqrt(len(serie))
    return f"{serie.mean():.4f} ± {se:.4f}"

ridge_val = np.concatenate([ridge_ic(a, b) for a, b in PL])
print(f"rows {len(d)}, search weeks {NB}, "
      f"holdout {len(fechas) - NB}")
print("sizes (train, validation):",
      [(len(a), len(b)) for a, b in PL])
print("IC ridge   search (CV)", resumen(ridge_val),
      " holdout", resumen(ridge_ic(ENT, RES)))
print("IC oracle  search     ", resumen(ic_sem(alfa[BUSQ], BUSQ)),
      " holdout", resumen(ic_sem(alfa[RES], RES)))

rows 72900, search weeks 364, holdout 365
sizes (train, validation): [(26700, 9100), (26600, 9100), (26600, 9100), (27200, 9100)]
IC ridge   search (CV) 0.0138 ± 0.0065  holdout 0.0172 ± 0.0062
IC oracle  search      0.0322 ± 0.0063  holdout 0.0248 ± 0.0069

Each fold trains on about 26,700 rows and validates on 9,100 (91 weeks). The ridge gives an IC of 0.0138 in the cross-validation and 0.0172 in the holdout; the oracle, 0.0322 and 0.0248. Note the standard error: ± 0.0065 for an IC measured over 364 weeks of 100 stocks. Any difference between configurations smaller than that cannot be told apart from chance with these data, and the levels change between periods (the oracle loses a quarter of its IC from one period to the other) even though the model is the same.

A 50-trial study

The objective function trains a LightGBM on each fold, stores the validation predictions and returns the mean IC over all the validation weeks. Three implementation details:

• lgb.train with a Dataset built once per fold: the feature binning does not depend on the hyperparameters being searched, and redoing it in each trial would cost almost as much as training. feature_pre_filter=False is needed to be able to change min_data_in_leaf on the same Dataset.
• trial.set_user_attr stores extra information per trial. Here, the holdout-period IC with the mean of the four fold models; the variable oof stores the validation predictions for the Deflated Sharpe.
• LightGBM determinism: seed=0, num_threads=4, deterministic=True, force_row_wise=True.

import lightgbm as lgb

BASE = dict(objective="regression", bagging_freq=1, max_bin=63,
            seed=0, num_threads=4, deterministic=True,
            force_row_wise=True, verbose=-1)

def espacio(trial):
    p = dict(
        learning_rate=trial.suggest_float("learning_rate", 0.01, 0.3,
                                          log=True),
        num_leaves=trial.suggest_int("num_leaves", 4, 48, log=True),
        min_data_in_leaf=trial.suggest_int("min_data_in_leaf", 20,
                                           2000, log=True),
        feature_fraction=trial.suggest_float("feature_fraction",
                                             0.3, 1.0),
        bagging_fraction=trial.suggest_float("bagging_fraction",
                                             0.3, 1.0),
        lambda_l2=trial.suggest_float("lambda_l2", 1e-3, 100,
                                      log=True))
    return p, trial.suggest_int("rondas", 50, 200, step=50)

def datasets(pl):
    """One LightGBM Dataset per fold, built once."""
    return [lgb.Dataset(X[a], y[a], free_raw_data=False,
                        params=dict(max_bin=63, verbose=-1,
                                    feature_pre_filter=False))
            .construct() for a, _ in pl]

DS = datasets(PL)
oof = {}   # validation predictions of each trial

def objetivo(trial):
    p, n = espacio(trial)
    pred = np.empty(len(BUSQ))
    res = np.zeros(len(RES))
    for (ent, pru), ds in zip(PL, DS):
        b = lgb.train({**BASE, **p}, ds, num_boost_round=n)
        pred[pru] = b.predict(X[pru])
        res += b.predict(X[RES]) / len(PL)
    oof[trial.number] = pred
    # only for the lesson: a project would not look at this
    trial.set_user_attr("ic_reservado",
                        float(ic_sem(res, RES).mean()))
    return float(ic_sem(pred, BUSQ).mean())

estudio = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=0))
estudio.optimize(objetivo, n_trials=50)
mejor = estudio.best_trial
print(f"best trial: no. {mejor.number}, validation IC "
      f"{mejor.value:.4f}")
print({k: (round(v, 4) if isinstance(v, float) else v)
       for k, v in mejor.params.items()})

best trial: no. 49, validation IC 0.0109
{'learning_rate': 0.0119, 'num_leaves': 6, 'min_data_in_leaf': 92, 'feature_fraction': 0.7374, 'bagging_fraction': 0.5103, 'lambda_l2': 0.0024, 'rondas': 50}

The winner is the last trial: small leaves (6), a low learning rate (0.012) and only 50 rounds, that is, a very smooth model. Its validation IC is 0.0109, below the untuned ridge (0.0138) on the same folds. Fifty configurations of a flexible model do not reach the linear baseline; lesson 5.5.1 had already seen that on this panel almost all the signal is linear in the ranks.

How much the best of N rises

The next cell follows the search trial by trial: the best validation IC up to trial N, the median of the trials run so far and, for the leader at each moment, its IC in the holdout period.

t = pd.DataFrame({
    "validacion": [x.value for x in estudio.trials],
    "reservado": [x.user_attrs["ic_reservado"]
                  for x in estudio.trials]})
lider = t["validacion"].expanding().apply(
    lambda s: s.idxmax()).astype(int)
filas = []
for n in (1, 5, 10, 20, 30, 40, 50):
    k = lider.iloc[n - 1]
    filas.append({
        "N": n, "mejor_validacion": t["validacion"].iloc[k],
        "mediana_validacion": t["validacion"].iloc[:n].median(),
        "elegido_reservado": t["reservado"].iloc[k],
        "mediana_reservado": t["reservado"].iloc[:n].median()})
print(pd.DataFrame(filas).set_index("N").round(4))
rho = t[["validacion", "reservado"]].corr(method="spearman").iloc[0, 1]
pos = (t["reservado"] > t["reservado"].iloc[mejor.number]).sum() + 1
print(f"validation-holdout rank correlation: {rho:.2f}")
print(f"the chosen trial ranks {pos} of 50 in the holdout period")

    mejor_validacion  mediana_validacion  elegido_reservado  mediana_reservado
N
1            -0.0081             -0.0081             0.0120             0.0120
5             0.0021             -0.0038             0.0102             0.0120
10            0.0064              0.0010             0.0153             0.0124
20            0.0092              0.0033             0.0086             0.0128
30            0.0092              0.0044             0.0086             0.0126
40            0.0092              0.0045             0.0086             0.0128
50            0.0109              0.0046             0.0131             0.0129
validation-holdout rank correlation: 0.07
the chosen trial ranks 22 of 50 in the holdout period

The figure at the top draws the full table. Above, the staircase of the best rises from −0.0081 to 0.0109: each step is a trial that was luckier than the previous ones with the validation weeks. Below, the same chosen trial in the holdout period goes up and down with no trend (from 0.0086 to 0.0153). Three measures summarize it:

• Advantage of the chosen trial over the typical trial: in validation, 0.0109 versus a median of 0.0046, that is, 0.0063. In the holdout, 0.0131 versus 0.0129: 0.0002.
• Rank correlation between the validation IC and the holdout IC across the 50 trials: 0.07. Ranking the configurations by their validation says almost nothing about how they rank outside.
• Rank of the chosen trial in the holdout: 22nd of 50.

The absolute level, on the other hand, rises from the search period to the holdout (every trial gives a higher IC outside than in validation), for the same reason that the ridge and the oracle change between periods. That is why the selection bias is measured against the other trials of the same period and not by subtracting two different periods.

The expected maximum of N trials

How much should the best of N rise if no configuration were better than another? Lesson 2.5.4 gave the approximation for the maximum of N independent normals with the Euler-Mascheroni constant $\gamma$:

The scale that multiplies that factor is the interesting question. There are two candidates: the standard error of the IC of one trial (how much it would move if it were repeated with other weeks) and the dispersion of the ICs across trials.

def max_esperado(n):
    """E[max] of n standard normals (approximation from 2.5.4)."""
    if n == 1:
        return 0.0
    gm = np.euler_gamma
    return ((1 - gm) * norm.ppf(1 - 1 / n)
            + gm * norm.ppf(1 - 1 / (n * np.e)))

se_val = np.mean([ic_sem(oof[k], BUSQ).std() for k in oof]) \
    / np.sqrt(NB)
disp = t["validacion"].std()
print(f"standard error of one trial's validation IC: {se_val:.4f}")
print(f"dispersion of the validation IC across trials: {disp:.4f}")
for n in (5, 10, 50, 200, 1000):
    print(f"N = {n:>4}: the best rises {max_esperado(n):.2f} "
          f"dispersions = {max_esperado(n) * disp:.4f} of IC")
print(f"measured with N = 50: best - mean = "
      f"{t['validacion'].max() - t['validacion'].mean():.4f}")

standard error of one trial's validation IC: 0.0063
dispersion of the validation IC across trials: 0.0038
N =    5: the best rises 1.19 dispersions = 0.0046 of IC
N =   10: the best rises 1.57 dispersions = 0.0060 of IC
N =   50: the best rises 2.28 dispersions = 0.0087 of IC
N =  200: the best rises 2.77 dispersions = 0.0106 of IC
N = 1000: the best rises 3.26 dispersions = 0.0125 of IC
measured with N = 50: best - mean = 0.0071

The standard error of one trial is 0.0063, but the 50 trials only differ by 0.0038 among themselves. The difference is that they are all measured on the same weeks: if a stretch of the validation favors momentum models, it favors almost all the trials at once, and that shared part does not change which one wins. What the maximum exploits is the part that tells them apart. With the dispersion, the formula predicts a rise of 0.0087 with 50 trials; the measurement (best minus mean) is 0.0071. TPE does not propose independent configurations, but ones increasingly similar to the good ones, which somewhat reduces the maximum compared with that of 50 independent ones.

The table gives the practical rule: the bias grows like $\sqrt{2 \ln N}$, quickly at first and slowly afterwards. Going from 5 to 50 trials adds 1.1 dispersions; from 50 to 1,000, 1.0. A budget of thousands of trials on a problem with this signal-to-noise ratio mostly buys bias.

Hyperparameter importance

Optuna estimates which hyperparameters explain the value of the objective function. In 5.0, the default evaluator is PED-ANOVA, which compares the distribution of each hyperparameter in the best trials (the top 10%, 5 of 50) with that of all of them. The target argument lets you compute it with another measure, here the holdout IC; since PED-ANOVA always looks for low values with target, it is passed with the sign flipped, and the warning that reminds you of this is silenced by its message.

import warnings

# with `target`, PED-ANOVA warns that it looks for low values: the minus
# sign already takes that into account
warnings.filterwarnings("ignore", message=".*importances of params")
imp_val = optuna.importance.get_param_importances(estudio)
imp_res = optuna.importance.get_param_importances(
    estudio, target=lambda x: -x.user_attrs["ic_reservado"])
pd.DataFrame({"validación": imp_val, "reservado": imp_res}).round(2)

                  validación  reservado
lambda_l2               0.47       0.62
min_data_in_leaf        0.16       0.06
feature_fraction        0.10       0.01
rondas                  0.08       0.01
num_leaves              0.08       0.03
learning_rate           0.07       0.04
bagging_fraction        0.05       0.23

The two columns agree on the first one (lambda_l2) and diverge on the rest: min_data_in_leaf explains 16% with the validation and 6% with the holdout; bagging_fraction, 5% and 23%. With 50 trials and an objective this noisy, the importance of a hyperparameter is another noisy estimate, computed with the same data that are being overfitted. It helps narrow the space of a follow-up search; it is not a finding about the market.

Nested validation

The clean defense against selection bias is not to use the winner's value as an estimate. In nested validation, the whole search is repeated inside each test period, using only the earlier data:

flowchart TB
  A[Test period] --> B[Earlier data]
  B --> C[Purged study]
  C --> D[Best parameters]
  D --> E[Retrain]
  E --> F[IC in period k]
  F --> G[Join periods]

The IC of the inner winner only serves to choose; what is reported is the IC of period k, which never took part in the choice. The resulting figure measures the procedure (search and then train), not a specific configuration, and it is the one that is compared with the baseline. Here, three periods (2019-2020, 2021-2022 and 2023-2025), each with a study of 10 trials and 3 purged folds over everything before it (in production there would be more trials and annual periods).

cortes = [np.datetime64(x) for x in
          ("2019-01-01", "2021-01-01", "2023-01-01", "2026-01-01")]
filas, ext, rid = [], [], []
for i in range(3):
    s0 = int((fechas < cortes[i]).sum())
    s1 = int((fechas < cortes[i + 1]).sum())
    pl = pliegues(0, s0, 3)
    ds = datasets(pl)

    def interno(trial):
        p, n = espacio(trial)
        ics = [ic_sem(lgb.train({**BASE, **p}, dk, num_boost_round=n)
                      .predict(X[pru]), pru)
               for (_, pru), dk in zip(pl, ds)]
        return float(np.concatenate(ics).mean())

    est_i = optuna.create_study(
        direction="maximize",
        sampler=optuna.samplers.TPESampler(seed=0))
    est_i.optimize(interno, n_trials=10)
    ent = np.flatnonzero(semana < s0 - 1)
    pru = np.flatnonzero((semana >= s0) & (semana < s1))
    p = dict(est_i.best_params)
    n = p.pop("rondas")
    b = lgb.train({**BASE, **p}, lgb.Dataset(X[ent], y[ent]),
                  num_boost_round=n)
    ext.append(ic_sem(b.predict(X[pru]), pru))
    rid.append(ridge_ic(ent, pru))
    anios = pd.DatetimeIndex(fechas[[s0, s1 - 1]]).year
    filas.append({"periodo": f"{anios[0]}-{anios[1]}",
                  "mejor_interno": est_i.best_value,
                  "lightgbm_fuera": ext[-1].mean(),
                  "ridge_fuera": rid[-1].mean(),
                  "oraculo": ic_sem(alfa[pru], pru).mean()})
print(pd.DataFrame(filas).set_index("periodo").round(4))
ext, rid = np.concatenate(ext), np.concatenate(rid)
print("nested 2019-2025: LightGBM", resumen(ext),
      " ridge", resumen(rid), " difference", resumen(ext - rid))

           mejor_interno  lightgbm_fuera  ridge_fuera  oraculo
periodo
2019-2020         0.0068          0.0040       0.0012   0.0251
2021-2022         0.0071          0.0132       0.0138   0.0222
2023-2025         0.0152          0.0122       0.0241   0.0262
nested 2019-2025: LightGBM 0.0101 ± 0.0063  ridge 0.0146 ± 0.0065  difference -0.0045 ± 0.0042

In two of the three periods, the chosen LightGBM gives less outside than its inner validation promised (0.0040 versus 0.0068; 0.0122 versus 0.0152), and in the other, more (0.0132 versus 0.0071): the best inner value does not predict the result of its period. Overall, the nested search gives an IC of 0.0101 ± 0.0063 over 2019-2025; the mean of the three best inner values, weighted by weeks, is 0.0105, so here the errors offset each other. With 10 trials per period (TPE's 10 random startup trials) the selection bias is smaller than with 50, and the nested validation has its own noise: what matters is that its figure is honest by construction. The untuned ridge gives 0.0146 ± 0.0065 over the same weeks: the paired difference, −0.0045 ± 0.0042, is not significant, but it goes against the model with search. Nobody goes beyond the oracle (0.022-0.026).

Counting the trials: the Deflated Sharpe

If the best of a search is presented, its metric has to discount how many configurations were tried. The level 2 tool is the Deflated Sharpe (lesson 2.5.4); lesson 5.4.4 applies it to ML signals. Each trial of the study left its validation predictions in oof: with them, for each one, the weekly long-short portfolio (10 stocks long and 10 short out of the 100, weights of ±0.1, simple returns) for 2012-2018 is built.

def cartera(p, filas, coste_pb=10):
    """Long 10, short 10 each week, weights ±0.1 (gross 2).
    Returns weekly returns gross and net of 10 bp per side."""
    P, R = p.reshape(-1, 100), r[filas].reshape(-1, 100)
    o = P.argsort(1)
    w = np.zeros_like(R)
    np.put_along_axis(w, o[:, -10:], 0.1, 1)
    np.put_along_axis(w, o[:, :10], -0.1, 1)
    bruto = (w * R).sum(1)
    rot = np.abs(np.diff(w, axis=0, prepend=0)).sum(1)
    return bruto, bruto - coste_pb / 1e4 * rot

def psr(sr, umbral, t, asim, curt):
    """Probabilistic Sharpe Ratio (lesson 2.5.4)."""
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr**2)
    return norm.cdf((sr - umbral) * np.sqrt(t - 1) / den)

S = pd.DataFrame({k: cartera(p, BUSQ)[0] for k, p in oof.items()})
sr = S.mean() / S.std()                    # weekly Sharpe
k = sr.idxmax()
T, asim, curt = len(S), S[k].skew(), S[k].kurt() + 3
print(f"Gross annual Sharpe of the 50 trials: median "
      f"{sr.median() * np.sqrt(52):.2f}, maximum "
      f"{sr[k] * np.sqrt(52):.2f} (trial {k})")
print(f"PSR(0) of the maximum: {psr(sr[k], 0, T, asim, curt):.3f}")
autoval = np.linalg.eigvalsh(S.corr().to_numpy())
n_ef = autoval.sum() ** 2 / (autoval**2).sum()
for n in (50, round(n_ef)):
    sr0 = np.sqrt(sr.var()) * max_esperado(n)
    print(f"N = {n:>2}: SR_0 = {sr0 * np.sqrt(52):.2f} annual, "
          f"DSR = {psr(sr[k], sr0, T, asim, curt):.3f}")

Gross annual Sharpe of the 50 trials: median -0.06, maximum 0.55 (trial 11)
PSR(0) of the maximum: 0.927
N = 50: SR_0 = 0.55 annual, DSR = 0.491
N =  3: SR_0 = 0.21 annual, DSR = 0.816

The 50 trials have a median gross annual Sharpe of −0.06; the best, trial 11, of 0.55. In isolation, its PSR(0) is 0.927: it does not reach the usual 0.95 even without discounting anything. With N = 50, the threshold $SR_0$ is exactly 0.55 annual and the Deflated Sharpe falls to 0.491: the best Sharpe of the search is exactly what is expected from the best of 50 trials with no edge.

Are they 50 trials? The 50 portfolios are highly correlated with each other (they are variants of the same model with the same features), so they are equivalent to fewer independent trials. A simple approximation is the effective number from the participation ratio of the eigenvalues of their correlation matrix, $N_{ef} = (\sum \lambda_i)^2 / \sum \lambda_i^2$, which here gives about 3. With N = 3 the DSR rises to 0.816, and it stays below 0.95. The figure shows the full path. López de Prado proposes a more careful estimate, grouping the trials into clusters and counting the clusters; the rule that does not change is that all the trials run are counted, including those of previous studies on the same data and the pruned ones.

The next cell closes the loop in the holdout period: the trial chosen by IC, the one chosen by Sharpe, the ridge and the oracle, each retrained on the whole search (without the last week, whose label reaches 2019) and measured once, with 10 bp per side on turnover.

def modelo_final(params):
    p = dict(params)
    n = p.pop("rondas")
    return lgb.train({**BASE, **p}, lgb.Dataset(X[ENT], y[ENT]),
                     num_boost_round=n)

ridge_fin = Ridge(alpha=1.0).fit(X[ENT], y[ENT])
candidatos = {
    f"chosen by IC (no. {mejor.number})":
        modelo_final(mejor.params).predict(X[RES]),
    f"chosen by Sharpe (no. {k})":
        modelo_final(estudio.trials[k].params).predict(X[RES]),
    "untuned ridge": ridge_fin.predict(X[RES]),
    "oracle": alfa[RES]}
filas = {}
for nombre, p in candidatos.items():
    bruto, neto = cartera(p, RES)
    filas[nombre] = {"IC": ic_sem(p, RES).mean(),
                     "sharpe_bruto": bruto.mean() / bruto.std()
                     * np.sqrt(52),
                     "sharpe_neto": neto.mean() / neto.std()
                     * np.sqrt(52)}
pd.DataFrame(filas).T.round(3)

                              IC  sharpe_bruto  sharpe_neto
chosen by IC (no. 49)      0.017         0.830       -0.087
chosen by Sharpe (no. 11)  0.017         0.502       -0.463
untuned ridge              0.017         0.828       -0.010
oracle                     0.025         1.216        0.220

The trial chosen by IC, retrained on all of 2012-2018, gives an IC of 0.017 in the holdout (more than the mean of its four fold models, 0.0131, because it trains on more data) and a gross Sharpe of 0.83, the same as the untuned ridge. The one chosen by Sharpe, trial 11, gives 0.50 gross. Net of costs, all three lose (−0.09, −0.46 and −0.01): a portfolio of 10 and 10 stocks rebuilt every week turns over too much for a signal of this size (lesson 5.1.3 measures it on the full panel). Not even the oracle goes beyond 0.22 net.

Pruning: less compute, the same trials

A pruner abandons the trials that are doing worse than the others before finishing them. With cross-validation, the natural step is the fold: after each one, trial.report reports the cumulative IC and trial.should_prune decides. The MedianPruner prunes if the intermediate value falls below the median of the previous trials at that step (n_warmup_steps=1: never after the first fold). With boosting you can also report by rounds, but then the decision is made by looking at the validation during training, which is another way of using it to choose.

def objetivo_poda(trial):
    p, n = espacio(trial)
    ics = []
    for i, ((_, pru), ds) in enumerate(zip(PL, DS)):
        b = lgb.train({**BASE, **p}, ds, num_boost_round=n)
        ics.append(ic_sem(b.predict(X[pru]), pru))
        trial.report(float(np.concatenate(ics).mean()), i)
        if trial.should_prune():
            raise optuna.TrialPruned()
    return float(np.concatenate(ics).mean())

est_p = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=1),
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5,
                                       n_warmup_steps=1))
est_p.optimize(objetivo_poda, n_trials=20)
estados = pd.Series([x.state.name for x in est_p.trials])
print(estados.value_counts().to_string())
print(f"folds trained: "
      f"{sum(len(x.intermediate_values) for x in est_p.trials)}"
      f" of {20 * len(PL)}")
print(f"best validation IC: {est_p.best_value:.4f}")

COMPLETE    14
PRUNED       6
folds trained: 69 of 80
best validation IC: 0.0097

6 of 20 trials were pruned and 69 of 80 folds were trained: a modest saving, because with four folds there is little to cut (the saving is larger with slow models and steps by epochs or rounds). Two more readings. Pruned trials are trials: they were looked at, and they count for the Deflated Sharpe. And this second study, with another sampler seed and 20 trials, ends with a best validation IC of 0.0097, versus 0.0109 for the first one with 50: the winner's value depends on the budget and even on the seed the search was run with.

For long studies, Optuna stores the history in a database and lets you resume it or spread it across processes (with n_jobs > 1 or several processes, the result stops being reproducible trial by trial). The code, not run because it writes a file:

estudio = optuna.create_study(
    study_name="lgbm-panel5",
    storage="sqlite:///busquedas.db",   # or JournalStorage
    direction="maximize",
    sampler=optuna.samplers.TPESampler(seed=0),
    load_if_exists=True)                # resumes if it already exists
estudio.optimize(objetivo, n_trials=200, timeout=3600)
# in another terminal: optuna-dashboard sqlite:///busquedas.db

The database has an advantage that is not technical: it is the trial log that the Deflated Sharpe needs. If a study is deleted and another is started with a corrected space, the trials of the first one still count.

Common mistakes

• Reporting the best value of the study as expected performance. It is the maximum of N noisy estimates: biased upward by about 2.3 dispersions with 50 trials.
• Validation without purging inside the objective function. A shuffled k-fold (lesson 5.4.1) does not only inflate the validation: it rewards the configurations that memorize best, such as small leaves and many rounds.
• Early stopping with the validation fold. Choosing the number of rounds by looking at the validation is one more hyperparameter chosen with the same data; it needs an inner split or to be counted as a trial.
• Parallelizing without knowing it. n_jobs=-1 makes the study irreproducible. With a single process and a seed, two runs give the same result.
• Tuning without a baseline. 50 LightGBM configurations do not reach an untuned ridge on this panel; without the ridge alongside, the 0.0109 would look like a result.
• Forgetting the trials that do not appear in the report. Deleted studies, changed spaces, pruned trials: all count toward discounting the selection.

Exercises

1. Search only the LightGBM seed (from 0 to 10,000) with 20 trials of a RandomSampler, with the other hyperparameters fixed and bagging and feature fraction at 50%. How much does the validation IC move? What does the best seed give in the holdout?

fijos = dict(learning_rate=0.05, num_leaves=8, min_data_in_leaf=200,
             feature_fraction=0.5, bagging_fraction=0.5,
             lambda_l2=1.0)

def objetivo_semilla(trial):
    s = trial.suggest_int("semilla", 0, 10_000)
    pred = np.empty(len(BUSQ))
    res = np.zeros(len(RES))
    for (_, pru), ds in zip(PL, DS):
        b = lgb.train({**BASE, **fijos, "seed": s}, ds,
                      num_boost_round=100)
        pred[pru] = b.predict(X[pru])
        res += b.predict(X[RES]) / len(PL)
    trial.set_user_attr("res", float(ic_sem(res, RES).mean()))
    return float(ic_sem(pred, BUSQ).mean())

est_s = optuna.create_study(
    direction="maximize",
    sampler=optuna.samplers.RandomSampler(seed=0))
est_s.optimize(objetivo_semilla, n_trials=20)
val = pd.Series([x.value for x in est_s.trials])
res = pd.Series([x.user_attrs["res"] for x in est_s.trials])
print(f"validation: from {val.min():.4f} to {val.max():.4f}; "
      f"the best, in the holdout: {res[val.idxmax()]:.4f} "
      f"(median {res.median():.4f})")

validation: from -0.0091 to 0.0033; the best, in the holdout: 0.0131 (median 0.0131)

2. Compute the Deflated Sharpe of the best trial for N = 2, 3, 5, 10, 20 and 50. With how many trials would it stop passing a threshold of 0.95?

for n in (2, 3, 5, 10, 20, 50):
    sr0 = np.sqrt(sr.var()) * max_esperado(n)
    print(f"N = {n:>2}: DSR = {psr(sr[k], sr0, T, asim, curt):.3f}")

N =  2: DSR = 0.868
N =  3: DSR = 0.816
N =  5: DSR = 0.752
N = 10: DSR = 0.667
N = 20: DSR = 0.587
N = 50: DSR = 0.491

3. A colleague tunes the hyperparameters with Optuna over 2012-2025 with a 5-fold purged k-fold, obtains an IC of 0.021 with the best trial and presents it as what the model will deliver in 2026. What is wrong and how would you fix it without new data?