Python Para Trading

enes

Search

Contact
Contact

LanguageEspañol

Lesson 5.3.1 · 60 min

Fixed-horizon labeling and its problems

Before training a model you have to decide what it has to predict. The default answer, the sign of the return over a fixed horizon, looks neutral and is not: the horizon decides what the label measures, a fixed threshold fills it with volatility and overlap multiplies the apparent evidence. This lesson measures it on the level's stock panel and intraday future, where the truth is known.

Fig. — FNX, minutes in the trend regime: share of fixed-horizon labels whose sign matches the true trend direction, by horizon
01/05
Module
5.3 · Labelling and sampling
Duration
60 min
Prerequisites
2.1.1 · Hypothesis tests, p-values, bootstrap and regression with robust standard errors, 5.2.1 · Price, volume and microstructure features, 5.2.4 · Leakage-free normalization and cross-sectional rank
Author
Updated
Level 5 lessons

  1. 5.1 · ML fundamentals applied to finance

    1. 5.1.1 Why financial machine learning is different
    2. 5.1.2 Supervised, unsupervised and reinforcement learning; scikit-learn pipelines
    3. 5.1.3 Economic metrics for trading models: beyond accuracy

  2. 5.2 · Feature engineering

    1. 5.2.1 Price, volume and microstructure features
    2. 5.2.2 Point-in-time fundamentals, macro and alternative data
    3. 5.2.3 Fractional differentiation for features
    4. 5.2.4 Leakage-free normalization and cross-sectional rank
    5. 5.2.5 Feature importance: MDI, MDA, SFI and SHAP

  3. 5.3 · Labelling and sampling

    1. 5.3.1 Fixed-horizon labeling and its problems
    2. 5.3.2 Triple-barrier method: labels that mimic a trade
    3. 5.3.3 Meta-labeling: one model for the side and another for the size
    4. 5.3.4 Label uniqueness, overlap weights and the sequential bootstrap
    5. 5.3.5 Event-based sampling: the symmetric CUSUM filter

  4. 5.4 · Validation for financial series

    1. 5.4.1 Why random k-fold fails on financial data
    2. 5.4.2 Purged k-fold with embargo and CPCV for ML models
    3. 5.4.3 Walk-forward with retraining: windows and frequency
    4. 5.4.4 Backtesting ML signals with costs and Deflated Sharpe

  5. 5.5 · Classical models

    1. 5.5.1 Regularized regressions: Ridge, Lasso, Elastic Net and logistic
    2. 5.5.2 Decision trees, Random Forest and bagging
    3. 5.5.3 Gradient boosting: XGBoost, LightGBM and CatBoost
    4. 5.5.4 Optuna and the risk of overfitting the validation
    5. 5.5.5 Unsupervised learning: PCA, clustering and autoencoders
    6. 5.5.6 Learning to rank for stock selection

  6. 5.6 · Deep learning with Keras 3

    1. 5.6.1 Keras 3: one model, several compute engines
    2. 5.6.2 An MLP on a stock panel: regularize and compare
    3. 5.6.3 1D CNN, LSTM and GRU: windows, scaling and leaks
    4. 5.6.4 Callbacks, a Sharpe loss and the model in production

  7. 5.7 · Deep learning with PyTorch

    1. 5.7.1 PyTorch: tensors, autograd and the training loop
    2. 5.7.2 Time series architectures: TCN, N-BEATS, PatchTST and iTransformer
    3. 5.7.3 Probabilistic models: quantile loss, DeepAR and intervals
    4. 5.7.4 Graph neural networks for asset relationships
    5. 5.7.5 End-to-end portfolio optimization: differentiable layers and Sharpe loss

  8. 5.8 · Time series foundation models

    1. 5.8.1 Time series foundation models: pretrain once, forecast without training
    2. 5.8.2 Foundation model families: architecture, data and license
    3. 5.8.3 In practice: context, quantiles, covariates and fine-tuning
    4. 5.8.4 Foundation models in finance: what the evidence says
    5. 5.8.5 Lab: foundation models vs GARCH, HAR and LightGBM

  9. 5.9 · Reinforcement learning

    1. 5.9.1 MDP, Q-learning, PPO and Gymnasium environments
    2. 5.9.2 RL for execution, market making and dynamic hedging
    3. 5.9.3 Why reinforcement learning overfits in directional trading

  10. 5.10 · Interpretability, robustness and model risk

    1. 5.10.1 SHAP, partial dependence and the stability of feature importances
    2. 5.10.2 Data drift and concept drift: detecting them and what to do
    3. 5.10.3 Ensembles, signal combination and synthetic data

What the model has to predict

Supervised learning needs an answer for each row. In tutorials on ML applied to markets the answer is almost always the same: the sign of the return from the close of bar tt to that of bar t+ht + h. With prices PP and log return rt,t+h=ln⁡(Pt+h/Pt)r_{t,t+h} = \ln(P_{t+h}/P_t), there are three variants:

y_t = \begin{cases} +1 & r_{t,t+h} > \tau_t \\ 0 & |r_{t,t+h}| \le \tau_t \\ -1 & r_{t,t+h} < -\tau_t \end{cases}, \qquad y_t = r_{t,t+h} - \operatorname{median}_j\, r_{j,t,t+h}.$$ The first is a binary classification; the second adds a “no move” class with a threshold $\tau_t$; the third is the one for a stock panel, where what matters for a long-short portfolio is the return **relative** to the other stocks on the same date. This family is called **fixed-horizon** because $h$ is the same for all rows, whatever happens between $t$ and $t + h$. ```mermaid flowchart LR T[Bar t: features] --> T1[End t1 = t + h] T1 --> R[r from t to t1] R --> Y[Label y] ``` It is the simplest label and the one that this level's validation and model lessons use with the panel. It has five problems, and this lesson measures four of them with data where the truth is known (the data are synthetic: the effects are planted and the generating model is published in `public/curso/datos/nivel-5/README.md`). The fifth, that it ignores the path between $t$ and $t + h$, is the starting point of the triple barrier (lesson 5.3.2). | Problem | What happens | Where it is measured | |---|---|---| | Noise | The label barely resembles the predictable part | Panel: sign versus the true alpha | | Fixed threshold | Classes are distributed by volatility, not by signal | Panel by year and FNX by hour | | Horizon | It decides which phenomenon the label measures | FNX: sign versus the latent trend | | Overlap | If $h$ is larger than the sampling step, labels share returns | Panel: 21-session label | | Path | A stop hit along the way does not count | Lesson 5.3.2 | ## The panel's labels The panel of 400 fictitious stocks comes with two labels already computed, each with its end: `ret_sig_semana` runs from the close of `fecha` (the last day of the week) to the close of the following week (`fin_sig_semana`), and `ret_sig_21d`, over 21 sessions (`fin_sig_21d`). The classification label is built by subtracting the weekly median: :::cell{title="The panel's two labels"} ```python import numpy as np import pandas as pd f = pd.read_parquet("panel5_features.parquet") e = pd.read_parquet("panel5_etiquetas.parquet") v = pd.read_parquet("panel5_verdad.parquet") d = (f.merge(e, on=["fecha", "ticker"]) .merge(v[["fecha", "ticker", "alfa_verdadero"]], on=["fecha", "ticker"]) .dropna(subset=["ret_sig_21d"]).reset_index(drop=True)) sem = d.groupby("fecha") d["y"] = d["ret_sig_semana"] - sem["ret_sig_semana"].transform("median") d["alfa"] = (d["alfa_verdadero"] - sem["alfa_verdadero"].transform("median")) d["etiqueta"] = np.sign(d["y"]).astype(int) cols = ["fecha", "fin_sig_semana", "ret_sig_semana", "etiqueta", "fin_sig_21d", "ret_sig_21d"] d.loc[d["ticker"] == "BAKU", cols].head(3).round( {"ret_sig_semana": 4, "ret_sig_21d": 4}) ``` ```output fecha fin_sig_semana ret_sig_semana etiqueta fin_sig_21d ret_sig_21d 2 2012-01-13 2012-01-20 -0.0723 -1 2012-02-14 0.0902 402 2012-01-20 2012-01-27 0.0176 1 2012-02-21 0.2986 802 2012-01-27 2012-02-03 0.0688 1 2012-02-28 0.3393 ``` ::: The end dates are not `fecha` plus seven days or plus 21 calendar days: they are counted in NYSE sessions, so holidays shift them. That is why the labeling function has to return the end and not let the rest of the code recompute it. ## How much of the label is signal The panel publishes `alfa_verdadero`: the extra return for the following week that the generator plants in each stock. It is the predictable part of the label; the rest is noise. If a model knew the exact alpha (the **oracle**), its hit rate when predicting the sign of the label would be the proportion of rows in which the sign of the alpha and that of the label match: :::cell{title="How many labels have the sign of the alpha?"} ```python acierta = np.sign(d["y"]) == np.sign(d["alfa"]) print(f"all: {acierta.mean():.2%}") pct = sem["y"].transform(lambda s: s.abs().rank(pct=True)) cortes = [0.25, 0.5, 0.75, 0.9, 0.97] pd.Series({f"|y| above percentile {int(q * 100)}": acierta[pct > q].mean() for q in cortes} ).map("{:.2%}".format) ``` ```output all: 50.93% |y| above percentile 25 51.20% |y| above percentile 50 51.41% |y| above percentile 75 51.62% |y| above percentile 90 51.63% |y| above percentile 97 51.39% dtype: str ``` ::: 50.93%. The cross-sectional standard deviation of the alpha is about 12 bp per week and that of the label, about 360 bp: the signal is one thirtieth of the noise, a realistic ratio for equities (lesson 5.1.1). This is the ceiling of any classifier trained on this label: an out-of-sample hit rate of 55% is not a good model, it is a leak. The second part of the cell answers a very widespread idea: removing small returns, “which are noise”, so that the model learns only from clear moves. The agreement with the alpha rises a little when discarding the smallest quartile (51.20%) and the bottom half (51.41%), plateaus at 51.6% and falls again with the 3% most extreme returns (51.39%). A stock's large returns are not more informative: with fat tails, the most extreme ones are mainly jumps (earnings, sector scares) that the alpha does not explain. And each filter leaves the model with fewer rows. A moderate threshold may help a little; it does not turn noise into signal. ## Fixed threshold: the classes follow volatility The second variant, with class 0, needs a threshold. The most common is a fixed number (“more than 3% in a week”). The problem is that volatility changes, and with it the proportion of returns that pass the threshold: :::cell{title="Fixed threshold versus volatility threshold"} ```python r = d["ret_sig_semana"] sigma_sem = d["vol_63d"] / np.sqrt(52) anio = d["fecha"].dt.year pd.DataFrame({ "|r| > 3%": (r.abs() > 0.03).groupby(anio).mean(), "|r| > 0.5·σ": (r.abs() > 0.5 * sigma_sem).groupby(anio).mean(), }).loc[[2013, 2017, 2019, 2020, 2022, 2023]].round(3) ``` ```output |r| > 3% |r| > 0.5·σ fecha 2013 0.362 0.574 2017 0.356 0.586 2019 0.392 0.578 2020 0.503 0.591 2022 0.525 0.599 2023 0.349 0.552 ``` ::: With a fixed 3%, the ±1 class is 35% of the rows in 2017 and 2023 and more than half in 2020 and 2022. A classifier trained this way learns, before anything else, which volatility regime it is in, and the classes change proportion between training and test. With a threshold proportional to the stock's volatility, $\tau_{i,t} = 0.5\cdot \sigma_{i,t}/\sqrt{52}$ (the panel's `vol_63d`, known at `fecha`, scaled to one week), the proportion stays between 55% and 60% every year. In intraday data the effect is stronger, because volatility has a U-shaped curve within each session. FNX, the level's fictitious future, has one year of one-minute bars (247 sessions of 390 minutes). To scale without the future you need a **forecast** volatility for the next 30 minutes: here, the variance of each minute of the day estimated with the previous 20 sessions, smoothed across neighboring minutes and summed over the horizon. :::cell{title="FNX: prices by session and seasonal volatility"} ```python b = pd.read_parquet("hf5_barras_1min.parquet") ver = pd.read_parquet("hf5_verdad.parquet") C = np.log(b["cierre"].to_numpy()).reshape(-1, 390) D = ver["direccion"].to_numpy().reshape(-1, 390) r2 = np.diff(C, axis=1) ** 2 # 389 returns per session # variance of each minute: mean of the PREVIOUS 20 sessions s2 = pd.DataFrame(r2).rolling(20).mean().shift(1) s2 = s2.T.rolling(9, center=True, min_periods=1).mean().T.to_numpy() acum = np.c_[np.zeros(len(s2)), np.cumsum(s2, axis=1)] print(C.shape, "sessions x minutes") ``` ```output (247, 390) sessions x minutes ``` ::: :::cell{title="30-minute labels with fixed and scaled threshold"} ```python h = 30 fut = C[:, h:] - C[:, :-h] # r from t to t+h, t = 0…359 sd = np.sqrt(acum[:, h:390] - acum[:, :390 - h]) ok = ~np.isnan(sd) media_hora = np.tile(np.arange(390 - h) // 30, (len(C), 1)) fijo = np.abs(fut) > 0.0020 escalado = np.abs(fut) > sd tabla = pd.DataFrame({ "forecast σ (bp)": [np.nanmean(sd[ok & (media_hora == k)]) * 1e4 for k in range(12)], "|r| > 20 bp": [fijo[ok & (media_hora == k)].mean() for k in range(12)], "|r| > forecast σ": [escalado[ok & (media_hora == k)].mean() for k in range(12)], }, index=[f"{9 + (30 + 30 * k) // 60:02d}:{(30 + 30 * k) % 60:02d}" for k in range(12)]) tabla.round(3) ``` ```output forecast σ (bp) |r| > 20 bp |r| > forecast σ 09:30 34.713 0.519 0.285 10:00 25.928 0.405 0.299 10:30 22.832 0.341 0.281 11:00 21.717 0.312 0.280 11:30 21.258 0.306 0.284 12:00 21.072 0.287 0.264 12:30 21.109 0.340 0.321 13:00 21.044 0.301 0.287 13:30 21.283 0.317 0.292 14:00 21.638 0.324 0.290 14:30 22.556 0.373 0.315 15:00 23.508 0.374 0.315 ``` ::: ::figure{vega="n5-m3-horizonte-umbral-dia"} With a fixed 20 bp threshold, 52% of the labels in the first half hour are ±1 and at midday, 29%: half of the model's work would be learning the time of day. With the threshold equal to the forecast volatility of the 30 minutes (from 35 bp at the open to 21 bp at midday), the proportion of ±1 stays between 26% and 32% in all time slots. This is the idea that the triple barrier takes further, with barriers at $\pm k\cdot\sigma_t\sqrt{h}$ (lesson 5.3.2). <aside class="callout" data-kind="aviso"> The scaled threshold has to use a volatility **known at t**. Dividing the future return by the realized volatility between $t$ and $t + h$ is a subtle leak: that volatility depends on the very move being labeled. The one in the cell uses only the previous sessions; the panel's `vol_63d`, only the 63 sessions up to `fecha`. </aside> ## The horizon decides what the label measures FNX alternates two latent regimes: **trend** (informed flow with a direction $D = \pm 1$, episodes of 60 minutes on average) and **reversion** to a 30-minute exponential mean. The minute-by-minute truth is in `hf5_verdad.parquet`. That makes it possible to ask, for each horizon $h$, how often the sign of the label matches the true direction of the trend at minute $t$, which is what a model should learn to recognize: :::cell{title="What the label measures depending on the horizon"} ```python def acierto_tendencia(h): fut = C[:, h:] - C[:, :-h] dd = D[:, :-h] m = dd != 0 return (np.sign(fut[m]) == dd[m]).mean() horizontes = [1, 5, 10, 15, 30, 45, 60, 90, 120, 180] pd.Series({h: acierto_tendencia(h) for h in horizontes}, name="sign = direction").rename_axis("h (min)").round(3) ``` ```output h (min) 1 0.540 5 0.621 10 0.662 15 0.687 30 0.714 45 0.735 60 0.732 90 0.719 120 0.711 180 0.676 Name: sign = direction, dtype: float64 ``` ::: The figure at the top draws the series. With $h = 1$ minute, the label matches the true direction 54% of the time: one minute's noise masks the drift. As the horizon is lengthened the drift accumulates faster than the noise (the drift grows with $h$; the noise, with $\sqrt{h}$) and the agreement rises to 73.5% at 45 minutes. Then it falls: the regime lasts about 60 minutes on average, and a 180-minute label mixes the trend episode with whatever comes after, which does not depend on the current direction. There is no correct horizon in general. There is one that corresponds to the phenomenon you want to capture and to how you are going to trade: a strategy that holds the position for half an hour needs a label of the order of half an hour, not of one minute (which rewards predicting noise) or of a day. And if the phenomenon lasts a variable time, no fixed $h$ fits all events: the triple barrier puts the end of each label where the price decides. ## Overlap: the same evidence counted several times In the panel, the features are weekly and the label `ret_sig_21d` covers 21 sessions, about four and a half weeks. One week's label shares four of its five weeks of returns with the following week's. Lesson 5.4.1 measures what that does to a shuffled cross-validation; here the interest is in what it does to inference. A feature's weekly IC with that label forms an autocorrelated series, and its naive $t$ statistic, $\overline{IC}/s_{IC}\cdot\sqrt{n}$, treats each week as an independent observation. There are two usual corrections: **Newey-West** errors (lesson 2.1.1), which estimate the variance of the mean with the autocovariances up to a lag, and **non-overlapping subsamples** (one week out of every five, averaging the five possible ones): :::cell{title="Overlapping labels and the t statistic"} ```python def ic_por_semana(col, lab): x = d[["fecha", col, lab]].dropna() rk = x.groupby("fecha")[[col, lab]].rank() return rk[col].groupby(x["fecha"]).corr(rk[lab]).to_numpy() def t_ingenuo(s): return s.mean() / s.std() * np.sqrt(len(s)) def t_newey_west(s, retardos=6): x = s - s.mean() var = x @ x / len(x) for k in range(1, retardos + 1): var += 2 * (1 - k / (retardos + 1)) * (x[k:] @ x[:-k]) / len(x) return s.mean() / np.sqrt(var / len(x)) def t_sin_solape(s, paso=5): return np.mean([t_ingenuo(s[i::paso]) for i in range(paso)]) filas = {} for c in ["alfa_verdadero", "valor_bp", "mom_12_1", "ey", "ruido_4", "ruido_3", "ruido_1"]: s21 = ic_por_semana(c, "ret_sig_21d") s5 = ic_por_semana(c, "ret_sig_semana") filas[c] = {"IC 21 sess.": s21.mean(), "naive t": t_ingenuo(s21), "t Newey-West": t_newey_west(s21), "non-overlap t": t_sin_solape(s21), "IC 1 wk.": s5.mean(), "t 1 wk.": t_ingenuo(s5)} pd.DataFrame(filas).T.round(3) ``` ```output IC 21 sess. naive t t Newey-West non-overlap t IC 1 wk. t 1 wk. alfa_verdadero 0.030 9.229 5.713 4.135 0.029 8.390 valor_bp 0.020 7.760 3.911 3.477 0.009 3.920 mom_12_1 0.017 5.542 2.951 2.477 0.008 2.445 ey 0.018 7.874 4.061 3.536 0.009 4.059 ruido_4 0.010 4.366 2.722 1.962 0.005 1.934 ruido_3 -0.001 -0.356 -0.200 -0.151 -0.001 -0.641 ruido_1 -0.001 -0.632 -0.653 -0.284 -0.002 -1.322 ``` ::: ::figure{vega="n5-m3-horizonte-solape-t"} With the 21-session label, the true alpha has a naive $t$ of 9.2; corrected, 5.7 (Newey-West) or 4.1 (non-overlapping). The weekly label, with no overlap, gives $t = 8.4$ with the same alpha: the long label does not contribute twice the evidence, it contributes less and counts it several times. The dangerous case is `ruido_4`, a feature with no effect of its own that inherits momentum's: with the naive $t$ (4.4) it looks like a discovery; corrected, it stays at 2.0-2.7, the same level as with the weekly label (1.9). In a sweep of hundreds of features (lesson 2.1.8), the naive $t$ of overlapping labels multiplies false positives. ```mermaid flowchart TB E[Overlap] --> I[Inflated t] E --> V[Leak in CV] E --> W[Repeated rows] I --> I2[Newey-West] V --> V2[Purging 5.4.2] W --> W2[Uniqueness 5.3.4] ``` Overlap is not an error in itself: a 21-session label may be the right one for a strategy that holds its positions for a month. The error is treating the rows as independent in the three parts of the process in the diagram. ## A labeling function that stores the end All of the above comes down to a function that returns, for each bar, the end of its label, the return and the class, with an optional threshold that must be computed with information up to $t$: :::cell{title="A labeling function that stores the end"} ```python def horizonte_fijo(precio, h, umbral=None): """Label of each bar with its log return over h bars. precio: Series with a time index (a single session or a continuous series). umbral: Series of log thresholds (or None: sign). """ lp = np.log(precio) r = lp.shift(-h) - lp t1 = pd.Series(precio.index, index=precio.index).shift(-h) if umbral is None: lab = np.sign(r) else: lab = np.sign(r).where(r.abs() > umbral, 0.0) out = pd.DataFrame({"t1": t1, "ret": r, "etiqueta": lab}) return out.dropna() una = b.set_index("ts")["cierre"].iloc[20 * 390:21 * 390] # session 21 umbral = pd.Series(sd[20], index=una.index[:390 - h]) et = horizonte_fijo(una, h, umbral.reindex(una.index)) print(et["etiqueta"].value_counts().sort_index().to_dict()) et.iloc[[0, 1, 359]].round({"ret": 5}) ``` ```output {-1.0: 45, 0.0: 188, 1.0: 127} t1 ret etiqueta ts 2024-07-31 13:30:00+00:00 2024-07-31 14:00:00+00:00 0.01345 1.0 2024-07-31 13:31:00+00:00 2024-07-31 14:01:00+00:00 0.01202 1.0 2024-07-31 19:29:00+00:00 2024-07-31 19:59:00+00:00 0.00605 1.0 ``` ::: The output's index is $t$ and the `t1` column, the end: it is the pair that the validation purging (5.4.2) and sample uniqueness (5.3.4) need. In the example session, with the threshold equal to the forecast volatility, 188 of the 360 labelable minutes fall in class 0, 127 in +1 and 45 in −1: a bullish session. The last 30 bars of the session have no label, because their horizon would cross the close; in intraday data it is advisable not to extend the label into the next session, where the opening gap has nothing to do with the previous day's features. ## Typical mistakes - **Labeling with $h$ in rows instead of in time.** `shift(-h)` on a DataFrame with gaps (holidays, bars without trades) gives different horizons for different rows. You have to know what $h$ is and store `t1`. - **Crossing the session close** without deciding to: the overnight return is a different phenomenon. - **A threshold computed with the whole sample** (for example, the median of $|r|$ over the whole history): the same statistics leak as in lesson 5.2.4. - **Comparing hit rates between different labels.** 60% with class 0 included is not comparable with 51% on a binary label: you have to compare against the proportion of the majority class and, better, with economic metrics (5.1.3). - **Trusting the $t$ of overlapping labels** without correcting it. ## Exercises 1. In FNX's **reversion** minutes, the drift pushes the price toward its 30-minute exponential mean. Measure, for several horizons, how often the sign of the label is the opposite of that of the deviation $\ln P_t - \text{EMA}_{30}$. Which horizon best reflects that regime? ::::solucion :::cell{title="The label in the reversion regime"} ```python ema = pd.DataFrame(C.T).ewm(span=30).mean().T.to_numpy() desv = C - ema rev = (ver["regimen"].to_numpy().reshape(-1, 390) == "reversion") def acierto_reversion(h, minimo=0.001): fut = C[:, h:] - C[:, :-h] dv = desv[:, :-h] m = rev[:, :-h] & (np.abs(dv) > minimo) return (np.sign(fut[m]) == -np.sign(dv[m])).mean(), m.sum() pd.DataFrame({h: acierto_reversion(h) for h in [1, 5, 15, 30, 60]}, index=["hit rate", "minutes"]).T.round(3) ``` ```output hit rate minutes 1 0.517 17406.0 5 0.550 17176.0 15 0.567 16655.0 30 0.555 15830.0 60 0.528 14330.0 ``` ::: In reversion the label reflects the regime much worse than in trend: the hit rate rises from 51.7% with 1 minute to a maximum of 56.7% with 15 minutes and falls to 52.8% with 60. Mean reversion is slow (2% of the deviation per minute) and soon mixes with the regime change, so the useful horizon is shorter than in trend. The same $h$ does not work for both phenomena: another argument for labels whose end depends on the event (lesson 5.3.2) or for separating the two problems with a primary model and a meta-model (lesson 5.3.3). (Deviations smaller than 10 bp are discarded, where the sign of the deviation is almost random.) :::: 2. With the threshold equal to $k$ times the 30-minute forecast volatility, compute the proportion of each class for $k$ = 0.5, 1 and 1.5. Which $k$ leaves the three classes most balanced? ::::solucion :::cell{title="Class distribution by k"} ```python reparto = {} for k in [0.5, 1.0, 1.5]: lab = np.where(fut > k * sd, 1, np.where(fut < -k * sd, -1, 0)) reparto[k] = pd.Series(lab[ok]).value_counts(normalize=True) pd.DataFrame(reparto).T.round(3) ``` ```output 0 -1 1 0.5 0.412 0.298 0.289 1.0 0.707 0.149 0.144 1.5 0.867 0.068 0.065 ``` ::: With $k = 1$, 70.7% of the labels fall in class 0; with $k = 1.5$, 86.7%. The most balanced distribution is that of $k = 0.5$: 41% zeros and around 29-30% of each sign. There is no correct $k$: a high threshold leaves a very imbalanced problem (a classifier that always says 0 is right 87% of the time), and a low one goes back to the binary label with its noise. Choosing it by looking at the validation result is another form of overfitting (lesson 5.5.4); it is better to fix it from the problem (the cost of trading, the minimum move that pays for the bid-ask spread). :::: 3. A long-short strategy on the panel rebalances every Friday and holds the portfolio for one week. A colleague proposes training with `ret_sig_21d` “because it has more signal” (its IC with the alpha is 0.030 versus 0.029). What do you answer? :::solucion That a similar mean IC does not mean more information. The 21-session label, sampled every week, overlaps with the next four: its corrected $t$ (4.1-5.7) is lower than that of the weekly label (8.4), so it contributes less independent evidence, not more. Besides, it measures something other than what the strategy earns: the portfolio is rebuilt every week, and what it gains or loses is that week's return. If you want to use the 21-session one (for example, to reduce turnover with a slower signal), you have to purge the validation with its `t1` (5.4.2), weight by uniqueness (5.3.4) and correct any statistic with Newey-West or with non-overlapping subsamples. :::

Each answer is folded under its question

Self-assessment

  1. In the panel, the sign of the weekly label (return minus the weekly median) matches the sign of the true alpha in 50.9% of the rows. What does it mean?

    • AThat the label is miscalculated
    • BThat the true alpha is useless for prediction
    • CThat a shorter horizon should be used
    • DThat almost all of the label is noise: the planted alpha is small compared with the dispersion of returns, and a perfect model would only get that 50.9% right
    Show answer

    Answer D. That almost all of the label is noise: the planted alpha is small compared with the dispersion of returns, and a perfect model would only get that 50.9% right

    The oracle is the best possible forecast. If even it does not match the sign of the label more than 50.9% of the time, no model can go beyond that; a higher out-of-sample hit rate indicates a leak.

  2. With a fixed 20 bp threshold to label FNX's 30-minute return, 52% of the 9:30 labels are ±1 and only 29% of the 12:00 ones. What is the label mainly measuring?

    • AThat there is more signal in the morning
    • BThat the primary model works better early in the day
    • CThe volatility of each time slot: with a U-shaped curve, the same threshold is small at the open and large at midday
    • DThe volume of each time slot
    Show answer

    Answer C. The volatility of each time slot: with a U-shaped curve, the same threshold is small at the open and large at midday

    With the threshold scaled by the forecast volatility of each time slot (computed with the previous 20 sessions), the fraction of ±1 labels stays between 26% and 32% throughout the day.

  3. In FNX's trend minutes, the sign of the return over h minutes matches the true direction 54.0% of the time with h = 1, 73.5% with h = 45 and 67.6% with h = 180. Why does it fall when the horizon is lengthened?

    • ABecause the regime lasts about 60 minutes on average: beyond that, the label mixes the trend episode with whatever comes next, which does not depend on the current direction
    • BBecause of price rounding
    • CBecause there are fewer observations
    • DBecause volatility falls in the afternoon
    Show answer

    Answer A. Because the regime lasts about 60 minutes on average: beyond that, the label mixes the trend episode with whatever comes next, which does not depend on the current direction

    With short horizons each minute's noise dominates; with long horizons, the regime change. The horizon that best reflects the state at t is close to the typical duration of the phenomenon you want to capture.

  4. The IC of the true alpha with the 21-session label, sampled every week, has a naive t of 9.2; with Newey-West it is 5.7 and with non-overlapping subsamples, 4.1. What is the correct reading?

    • AThat the 21-session label has twice the information of the weekly one
    • BThat each return enters about five consecutive labels: the naive t counts the same evidence several times and has to be corrected
    • CThat Newey-West always halves the t
    • DThat the true alpha is not significant
    Show answer

    Answer B. That each return enters about five consecutive labels: the naive t counts the same evidence several times and has to be corrected

    The 21-session labels of consecutive weeks share four of their five weeks, so the IC series is autocorrelated. The weekly label, with no overlap, gives t = 8.4 with the same alpha.

  5. Why should the labeling function return, in addition to the label, the moment it ends (t1)?

    • ATo plot it
    • BBecause scikit-learn requires it
    • CTo compute the threshold
    • DBecause the validation purging (5.4.2) and uniqueness weights (5.3.4) need to know which price interval each label uses
    Show answer

    Answer D. Because the validation purging (5.4.2) and uniqueness weights (5.3.4) need to know which price interval each label uses

    Two labels overlap if their intervals from t to t1 intersect. Without t1 you cannot purge the training set or measure concurrency; with horizons in sessions, holidays and session closes, t1 is not simply t + h.

Key points

  • The fixed-horizon label is the sign (or the return) from t to t + h, sometimes with a threshold that creates a class 0. Its end, t1, has to be stored with it.
  • Almost all of the label is noise: in the panel, the sign of the weekly label matches that of the true alpha in 50.9% of the rows. Removing small returns only raises it to 51.6% and removing too many lowers it again.
  • A fixed threshold distributes the classes according to volatility: from 35% to 53% of ±1 labels depending on the year in the panel and from 29% to 52% depending on the time of day in FNX. A threshold proportional to the forecast volatility stabilizes it.
  • The horizon decides what the label measures. In FNX, the sign over h minutes reflects the latent trend 54% of the time with 1 minute, 73.5% with 45 and 67.6% with 180.
  • Overlapping labels inflate the evidence: with 21 sessions sampled every week, the naive t of the true alpha is 9.2 and the corrected one, between 4.1 and 5.7.
  • The triple-barrier method (5.3.2) adds the path and an event-dependent horizon; uniqueness weights (5.3.4) and purging (5.4.2) deal with the overlap.

Resources

Documentation

Books

Data

Back to level 5