---
title: "5.1.2 Supervised learning and scikit-learn pipelines"
description: "Supervised, unsupervised and reinforcement learning; Pipeline, ColumnTransformer, custom transformers and hyperparameter search with time-based validation."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/supervised-learning-scikit-learn-pipelines.png"
---

# 5.1.2 · Supervised, unsupervised and reinforcement learning; scikit-learn pipelines

> A trading model is a chain of steps (clean, transform, encode, fit) and every step that learns something from the data can leak information from the future. This lesson places the three types of learning in trading and builds a complete scikit-learn chain on the level's panel: custom transformers, date-based splitting, hyperparameter search with a trading metric and the leak that appears when features are selected outside the validation.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.1 · ML fundamentals applied to finance
- Duration: 70 min
- Prerequisites: [0.5.3 · Selection, filtering, groupby, merge, pivot, resample and moving windows](https://pythonparatrading.com/en/python-trading-course/level-0/pandas-groupby-merge-resample-rolling/), [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/), [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/), [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/)

## By the end of this lesson

- Distinguish supervised, unsupervised and reinforcement learning and place each one in a concrete trading problem.
- Use the scikit-learn API (`fit`, `transform`, `predict`) knowing what each step learns and from which data.
- Build a `Pipeline` with `ColumnTransformer` and custom stateful and stateless transformers compatible with `set_output`.
- Write a date-based splitter with a gap and an IC *scorer* to search hyperparameters with `GridSearchCV` without mixing the future with the past.
- Measure the leak from selecting features with all the data before validating and avoid it by putting the selection inside the pipeline.

## Three ways of learning

The level uses three families of methods. They differ in what they receive to learn from, not in the algorithm:

```mermaid
flowchart LR
  ML[Learning] --> S[Supervised]
  ML --> U[Unsupervised]
  ML --> R[Reinforcement]
  S --> SR[Regression]
  S --> SC[Classification]
  U --> UD[Dimensionality reduction]
  U --> UG[Clustering]
  R --> RP[Policy]
```

- **Supervised.** Each example comes with its features $x$ and an observed label $y$, and the model learns
  $f(x) \approx y$. If $y$ is a number (next week's return, tomorrow's volatility) it is **regression**; if it is a
  class (up or down, which barrier is hit first), **classification**.
- **Unsupervised.** There is only $x$. The model looks for structure: a few directions that summarize many columns
  (PCA), groups of assets that move together, market states that repeat.
- **Reinforcement.** An agent makes decisions in sequence, the environment responds and returns a reward; the agent
  learns a **policy** (what to do in each state) that maximizes the cumulative reward. There is no label for “the
  correct action”, only consequences.

| Family | Trading question | Label or reward | Lessons |
|---|---|---|---|
| Supervised, regression | Which stocks will return more next week? | Future return (minus the mean) | 5.5.1, 5.5.3, 5.5.6 |
| Supervised, regression | What volatility will this asset have tomorrow? | Realized variance | 5.7.3, 5.8.5 |
| Supervised, classification | Is this trade from the primary system worth taking? | Barrier hit first | 5.3.2, 5.3.3 |
| Unsupervised | Which regime is the market in? Which assets are similar? | None | 5.5.5 |
| Reinforcement | How do I sell 200,000 shares in one hour? How do I hedge an option with costs? | Cost or P&L of each step | 5.9.1, 5.9.2 |

Most of the level is supervised learning, and so are most of the mistakes: the label is information from the future
by definition, and every step in the process that touches it in the wrong place leaks it into the past. The rest of
the lesson builds the scikit-learn chain that avoids those mistakes by design.

## The scikit-learn API: what each step learns

scikit-learn has a uniform API. An **estimator** learns from the data with `fit` and stores what it learns in
attributes ending in an underscore (`mean_`, `coef_`). A **transformer** also has `transform`, which applies what it
learned to new data; a **predictor**, `predict`. The rule that orders everything: `fit` is called only with training
data, `transform` and `predict` with any data.

The data is the `panel5` panel from lesson 5.1.1: 21 weekly features (without the `revision_analistas` leak), the
sector and, as the label, the following week's return minus its cross-sectional mean (what matters for ranking
stocks). Training up to 2018 and test from 2019, with a one-week gap:

```python
import numpy as np
import pandas as pd

f = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
d = (f.merge(et[["fecha", "ticker", "ret_sig_semana"]],
             on=["fecha", "ticker"])
      .dropna(subset=["ret_sig_semana"]).reset_index(drop=True))
d["y"] = d["ret_sig_semana"] - d.groupby("fecha")[
    "ret_sig_semana"].transform("mean")
num = [c for c in f.columns
       if c not in ("fecha", "ticker", "sector", "revision_analistas")]
cols = ["fecha", "sector"] + num
ent = d[d["fecha"] < "2019-01-01"].reset_index(drop=True)
pru = d[d["fecha"] >= "2019-01-08"].reset_index(drop=True)
print(len(ent), "training rows;", len(pru), "test rows")
print(ent["fecha"].max().date(), "|", pru["fecha"].min().date())
```
```output
145600 training rows; 145600 test rows
2018-12-28 | 2019-01-11
```

A `StandardScaler` learns the mean and standard deviation of each column in `fit` and uses them in `transform`.
Applied to the test set, it uses the training ones: that is how a live system would see the data, because on Monday
it does not know the means of the next seven years.

```python
from sklearn.preprocessing import StandardScaler

esc = StandardScaler().fit(ent[["log_capitalizacion", "vol_63d"]])
print("learned means:", esc.mean_.round(3))
z = esc.transform(pru[["log_capitalizacion", "vol_63d"]])
print("test mean after transforming:", z.mean(axis=0).round(3))
```
```output
learned means: [8.932 0.29 ]
test mean after transforming: [0.131 0.197]
```

The transformed test features do not have zero mean: the stocks' market cap grew between the two periods (0.13
standard deviations above the training mean) and volatility in 2019-2025 was higher (0.20 standard deviations). That
is correct. If the mean came out exactly zero, the scaling would have been learned with the test data.

## Pipeline: the chain as a single estimator

A `Pipeline` chains transformers and ends in a predictor, and it behaves like a single estimator: `fit` fits each
step with the output of the previous one and `predict` applies the whole chain. The advantage is not convenience: it
is that any scikit-learn validation tool refits **all** the steps with the training data of each split. For a
stateful step not to see the test, it is enough for it to be inside.

Before the pipeline you need the metric and the first custom transformer. The metric is the mean weekly IC (lesson
5.1.3); the `puntuacion_ic` function has the signature of a scikit-learn *scorer*, `(estimator, X, y)`, and takes
the dates from `X` itself:

```python
def ic_semanal(fecha, pred, y):
    """Rank correlation week by week."""
    s = pd.DataFrame({"fecha": np.asarray(fecha),
                      "p": np.asarray(pred), "y": np.asarray(y)})
    r = s.groupby("fecha")[["p", "y"]].rank()
    r = r - r.groupby(s["fecha"]).transform("mean")
    num_ = (r["p"] * r["y"]).groupby(s["fecha"]).sum()
    den = ((r["p"] ** 2).groupby(s["fecha"]).sum()
           * (r["y"] ** 2).groupby(s["fecha"]).sum()) ** 0.5
    return num_ / den


def puntuacion_ic(estimador, X, y):
    return ic_semanal(X["fecha"], estimador.predict(X), y).mean()
```

The custom transformer is a **winsorizer**: it clips each column to its 1% and 99% quantiles. It is stateful: the
quantiles are learned in `fit` and applied in `transform`. A scikit-learn transformer inherits from `BaseEstimator`
(which gives it `get_params` and `set_params`, needed for hyperparameter search) and from `TransformerMixin` (which
gives it `fit_transform`); the parameters are stored in `__init__` untouched, what is learned in attributes ending in
`_`, and `get_feature_names_out` says what the output columns are called:

```python
from sklearn.base import BaseEstimator, TransformerMixin


class Winsorizador(BaseEstimator, TransformerMixin):
    """Clips each column to the q and 1 - q quantiles of the fit."""

    def __init__(self, q=0.01):
        self.q = q

    def fit(self, X, y=None):
        X = pd.DataFrame(X)
        self.bajo_ = X.quantile(self.q)
        self.alto_ = X.quantile(1 - self.q)
        self.feature_names_in_ = np.asarray(X.columns, dtype=object)
        return self

    def transform(self, X):
        return pd.DataFrame(X).clip(self.bajo_, self.alto_, axis=1)

    def get_feature_names_out(self, input_features=None):
        return self.feature_names_in_


w = Winsorizador(0.01).fit(ent[num])
print(pd.DataFrame({"low": w.bajo_, "high": w.alto_})
      .loc[["ret_1s", "sue", "sentimiento"]].round(3))
```
```output
               low   high
ret_1s      -0.112  0.116
sue         -2.990  3.662
sentimiento -2.344  2.307
```

With it, a complete linear pipeline: winsorize, impute the gaps in `sentimiento` (32% of the rows have no coverage)
with the median and a missing-value indicator column, standardize and fit a ridge:

```python
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline

lineal = make_pipeline(Winsorizador(0.01),
                       SimpleImputer(strategy="median",
                                     add_indicator=True),
                       StandardScaler(), Ridge(alpha=1000.0))
lineal.set_output(transform="pandas")
lineal.fit(ent[num], ent["y"])
print(list(lineal.named_steps))
print(lineal[:-1].get_feature_names_out()[-3:])
ic = ic_semanal(pru["fecha"], lineal.predict(pru[num]), pru["y"])
print(f"Mean IC 2019-2025: {ic.mean():.4f}")
```
```output
['winsorizador', 'simpleimputer', 'standardscaler', 'ridge']
['ruido_4' 'ruido_5' 'missingindicator_sentimiento']
Mean IC 2019-2025: 0.0181
```

`lineal[:-1]` is the pipeline without the last step: it shows which columns the ridge receives, including the
`missingindicator_sentimiento` indicator added by the imputer. The 2019-2025 IC is 0.018, close to the ridge in
lesson 5.1.1 (0.019) and below the oracle's ceiling in that period (0.025).

## ColumnTransformer and a stateless transformer

Not all features are treated the same way. The sector is a category and is encoded into binary columns; the numeric
ones are turned into a **cross-sectional rank**: each stock's percentile among the 400 of its week. The rank removes
extreme values and each era's level in one go (a 30% volatility is high in 2017 and normal in 2020). Lesson 5.2.4
compares this and other scalings; here what matters is its form: it is a **stateless** transformer, because each
week is transformed only with that week's data. Its `fit` learns nothing from the features, so it cannot leak
anything from the future.

`ColumnTransformer` applies a different transformer to each group of columns and concatenates the result:

```mermaid
flowchart TB
  X[fecha, sector, features] --> RF[RangoPorFecha]
  RF --> CT[ColumnTransformer]
  CT --> IM[Numeric: impute]
  CT --> OH[Sector: one-hot]
  IM --> M[Ridge]
  OH --> M
```

```python
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder


class RangoPorFecha(BaseEstimator, TransformerMixin):
    """Percentile of each column within its date, minus 0.5."""

    def __init__(self, columnas):
        self.columnas = columnas

    def fit(self, X, y=None):
        self.resto_ = [c for c in X.columns
                       if c not in self.columnas + ["fecha"]]
        return self

    def transform(self, X):
        g = X.groupby("fecha")[self.columnas]
        return (g.rank(pct=True) - 0.5).join(X[self.resto_])

    def get_feature_names_out(self, input_features=None):
        return np.asarray(self.columnas + self.resto_, dtype=object)


prep = ColumnTransformer(
    [("num", SimpleImputer(strategy="constant", fill_value=0.0), num),
     ("sec", OneHotEncoder(handle_unknown="ignore",
                           sparse_output=False), ["sector"])],
    verbose_feature_names_out=False)
modelo = Pipeline([("rango", RangoPorFecha(num)), ("prep", prep),
                   ("ridge", Ridge())])
modelo.set_output(transform="pandas")
Xt = modelo[:-1].fit_transform(ent[cols])
print(Xt.shape)
Xt.iloc[:2, [0, 3, 20, 21, 22]].round(3)
```
```output
(145600, 31)
   ret_1s  mom_12_1  ruido_5  sector_consumo_basico  sector_consumo_ciclico
0  -0.305    -0.350   -0.440                    0.0                     0.0
1  -0.285     0.047   -0.107                    0.0                     0.0
```

The rank leaves each feature between −0.5 and 0.5; the imputer sets to 0 (the median of their week) the stocks with
no sentiment data, which the rank leaves empty. The one-hot encoding adds ten columns, one per sector.
`handle_unknown="ignore"` is a production decision: if a new sector appears, it is encoded as zeros instead of
breaking the system.

## Splitting by date: validating a panel

To choose hyperparameters you need to validate within the training period. Standard cross-validation (`KFold`)
shuffles the rows and, in financial series, promises results that do not exist; lesson 5.4.1 measures it in detail.
scikit-learn's alternative for series, `TimeSeriesSplit`, trains on the past and tests on the future, but it splits
by **rows**. In a panel, each date has 400 rows:

```python
from sklearn.model_selection import TimeSeriesSplit

for ent_i, pru_i in TimeSeriesSplit(n_splits=5).split(ent):
    comunes = set(ent.loc[ent_i, "fecha"]) & set(
        ent.loc[pru_i, "fecha"])
    print(f"{len(ent_i):6d} training rows; "
          f"shared weeks: {len(comunes)}")
```
```output
 24270 training rows; shared weeks: 1
 48536 training rows; shared weeks: 1
 72802 training rows; shared weeks: 1
 97068 training rows; shared weeks: 1
121334 training rows; shared weeks: 1
```

In all five splits the boundary falls in the middle of a week: some of that week's stocks are used for training and
others for testing, with the same market and the same common return. In addition, the label of the last training
week ends one Friday later, inside the test period. Both problems are fixed with a date-based splitter with a
**gap** of at least the label horizon (one week for `ret_sig_semana`; for `ret_sig_21d`, five). A scikit-learn
splitter is any object with `split` and `get_n_splits`:

```python
class SplitPorFechas:
    """Expanding window by date: train on the past, leave
    `hueco` dates unused and test on the next block."""

    def __init__(self, n_splits=5, hueco=1):
        self.n_splits = n_splits
        self.hueco = hueco

    def get_n_splits(self, X=None, y=None, groups=None):
        return self.n_splits

    def split(self, X, y=None, groups=None):
        fechas = np.asarray(X["fecha"])
        unicas = np.unique(fechas)
        bloques = np.array_split(unicas, self.n_splits + 1)
        for b in bloques[1:]:
            i = np.searchsorted(unicas, b[0]) - 1 - self.hueco
            yield (np.flatnonzero(fechas <= unicas[i]),
                   np.flatnonzero(np.isin(fechas, b)))


cv = SplitPorFechas(n_splits=5, hueco=1)
for i_ent, i_pru in cv.split(ent):
    fe, fp = ent.loc[i_ent, "fecha"], ent.loc[i_pru, "fecha"]
    print(fe.min().date(), "to", fe.max().date(), "| test",
          fp.min().date(), "to", fp.max().date())
```
```output
2012-01-13 to 2013-03-01 | test 2013-03-15 to 2014-05-09
2012-01-13 to 2014-05-02 | test 2014-05-16 to 2015-07-10
2012-01-13 to 2015-07-02 | test 2015-07-17 to 2016-09-09
2012-01-13 to 2016-09-02 | test 2016-09-16 to 2017-11-03
2012-01-13 to 2017-10-27 | test 2017-11-10 to 2018-12-28
```

*Fig. — Folds of the date splitter over 2012-2018: expanding training window, a one-week gap and a test block of about 60 weeks; separately, the final 2019-2025 test.* Six rows over a 2012-2026 axis. In the five folds, a grey training bar starting in January 2012 grows to March 2013, May 2014, July 2015, September 2016 and October 2017, followed after a one-week gap by a blue test block of about 60 weeks. The last row has the full 2012-2018 training and the highlighted final 2019-2025 test.

Each split trains on everything before it (expanding window), leaves one week unused and tests on the next block of
about 60 weeks. The figure draws the five splits and, separately, the final 2019-2025 test period, which the search
does not touch. It is a simple walk-forward; lessons 5.4.2 and 5.4.3 add purging, embargo and rolling windows.

## Hyperparameter search with a trading metric

`GridSearchCV` tries each combination in a grid with the splitter it is given and scores it with the *scorer*. The
hyperparameters of a pipeline step are named `step__parameter`. Here the search covers the ridge's regularization,
α, across nine orders of magnitude:

```python
from sklearn.model_selection import GridSearchCV

rejilla = {"ridge__alpha": [1, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7,
                            1e8]}
busqueda = GridSearchCV(modelo, rejilla, scoring=puntuacion_ic,
                        cv=cv, n_jobs=1)
busqueda.fit(ent[cols], ent["y"])
res = pd.DataFrame(busqueda.cv_results_)
res = res.set_index("param_ridge__alpha")[
    ["mean_test_score", "std_test_score"]]
print(res.round(4))
print("chosen:", busqueda.best_params_)
```
```output
                    mean_test_score  std_test_score
param_ridge__alpha
1.0                          0.0043          0.0135
10.0                         0.0043          0.0135
100.0                        0.0045          0.0134
1000.0                       0.0052          0.0128
10000.0                      0.0063          0.0115
100000.0                     0.0068          0.0104
1000000.0                    0.0070          0.0102
10000000.0                   0.0070          0.0102
100000000.0                  0.0070          0.0102
chosen: {'ridge__alpha': 10000000.0}
```

*Fig. — Ridge validation IC by regularisation α, mean of the five folds with its standard error: it rises and levels off once α dominates.* Line with points on a logarithmic α axis between 1 and 100 million: mean IC goes from 0.0043 at α = 1 to 0.0070 from α = 1 million, with error bars of about ±0.005 at each point. The chosen point, α = 10 million, is highlighted.

Two things that are not obvious. First: the validation IC **rises** with α and levels off at huge values. When α
dominates $X^\top X$, the ridge gives $\hat\beta \approx X^\top y/\alpha$: the coefficients tend to zero, but their
direction stops changing, and the IC only looks at the order of the predictions. With so little signal, heavy
shrinkage is what generalizes best: the ridge almost becomes an average of the features weighted by their covariance
with the label. Second: the standard deviation across splits (0.010 for the best one) is larger than the differences
between values of α. Choosing between $\alpha = 10^6$ and $\alpha = 10^8$ with this data is choosing between noises;
lesson 5.5.4 measures how much the validation itself can be overfitted when the grid is large.

`GridSearchCV` retrains the best model on the whole training set (`refit=True`, the default). It is evaluated once on
2019-2025 and compared with an untuned **baseline**: the sum of the ranks of four features with the sign given by the
literature (momentum, value and SUE positive; one-week reversal negative).

```python
mejor = busqueda.best_estimator_
ic_mod = ic_semanal(pru["fecha"], mejor.predict(pru[cols]),
                    pru["y"])
rk = pru.groupby("fecha")[["mom_12_1", "ret_1s", "valor_bp",
                           "sue"]].rank(pct=True)
base = rk["mom_12_1"] - rk["ret_1s"] + rk["valor_bp"] + rk["sue"]
ic_base = ic_semanal(pru["fecha"], base, pru["y"])
for nombre, ic in [("chosen ridge", ic_mod), ("baseline", ic_base)]:
    t = ic.mean() / ic.std() * np.sqrt(len(ic))
    print(f"{nombre:14s} IC {ic.mean():.4f}  t {t:.1f}")
coef = pd.Series(mejor["ridge"].coef_,
                 index=mejor[:-1].get_feature_names_out())
(coef / coef.abs().max()).sort_values().iloc[[0, 1, 2, -3, -2, -1]
                                            ].round(2)
```
```output
chosen ridge   IC 0.0189  t 3.1
baseline       IC 0.0178  t 4.1
ret_1s                      -1.00
rsi_14                      -0.71
sector_telecomunicaciones   -0.62
sector_energia               0.53
mom_12_1                     0.55
sector_industria             0.65
dtype: float64
```

The chosen ridge gives an IC of 0.019 in 2019-2025 and the four-feature baseline 0.018, with a higher t-statistic
(4.1 versus 3.1: it is more stable from one week to the next). The difference in means is smaller than the error of
either one. This is the comparison you must always make: if a pipeline with 21 features, imputation, encoding and a
search does not clearly beat a sum of four ranks, its complexity adds nothing.

The largest coefficients (relative to the largest) tell what it learned. The one-week reversal with a negative sign
and momentum with a positive sign are effects of the generator; the negative RSI inherits the reversal. But three of
the six are **sectors**: telecommunications negative, energy and industrials positive. The generator puts no sector
effect in the expected return; those coefficients are the sectors that, by chance, returned more or less than the
mean in 2012-2018. With such strong regularization, the ridge bets in proportion to each column's covariance with
the label, and that of a binary sector column is simply its past return. A model can improve in validation by
learning a period's noise; removing the sector or neutralizing the portfolio by sector (lesson 5.2.4) is a decision
made on economic reasoning, not on what the search says.

## The leak inside the validation: selecting with all the data

The most expensive pipeline mistake is not scaling with the test data, which here would change little. It is
**selecting** with it. The experiment: 500 columns of pure noise, with no relationship at all with the label, over
two years of data (2012-2013, 41,200 rows). The wrong way picks the 10 most correlated with the label using all the
rows and then validates a ridge with them. The right way puts the selection inside the pipeline, so in each split
the choice is made only with the training rows. As a control, the 10 columns chosen the wrong way are also evaluated
on data the selection did not see (2014-2018):

```python
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.model_selection import cross_val_score


class SinFecha(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return X.drop(columns="fecha")


rng = np.random.default_rng(0)
ruido = [f"r{i:03d}" for i in range(500)]
datos = d[d["fecha"] < "2019-01-01"].reset_index(drop=True)
Z = pd.DataFrame(rng.standard_normal((len(datos), 500)),
                 columns=ruido).assign(fecha=datos["fecha"])
dos = (datos["fecha"] < "2014-01-01").to_numpy()
Z2, y2 = Z[dos].reset_index(drop=True), datos.loc[dos, "y"]

elegidas = SelectKBest(f_regression, k=10).fit(
    Z2[ruido], y2).get_feature_names_out()
mal = cross_val_score(make_pipeline(SinFecha(), Ridge(1.0)),
                      Z2[list(elegidas) + ["fecha"]], y2, cv=cv,
                      scoring=puntuacion_ic)
bien = cross_val_score(
    make_pipeline(SinFecha(), SelectKBest(f_regression, k=10),
                  Ridge(1.0)),
    Z2, y2, cv=cv, scoring=puntuacion_ic)
despues = Ridge(1.0).fit(Z2[elegidas], y2).predict(Z[~dos][elegidas])
ic_desp = ic_semanal(Z.loc[~dos, "fecha"], despues,
                     datos.loc[~dos, "y"]).mean()
for nombre, v in [("outside the pipeline", mal),
                  ("inside the pipeline", bien)]:
    print(f"{nombre:20s} IC {v.mean():.3f} ± "
          f"{v.std(ddof=1) / np.sqrt(len(v)):.3f}", v.round(3))
print(f"selected, 2014-2018  IC {ic_desp:.3f}")
```
```output
outside the pipeline IC 0.028 ± 0.009 [0.035 0.02  0.022 0.006 0.058]
inside the pipeline  IC 0.008 ± 0.004 [ 0.004  0.018 -0.002  0.     0.019]
selected, 2014-2018  IC -0.001
```

The figure at the top summarizes the cell. The wrong way gives a mean validation IC of 0.028 with 500 noise
columns: the same level as the oracle's ceiling in the panel. There is no signal at all; the selection had already
seen the labels of each split's test weeks and chose the columns that, by chance, resembled them. With the selection
inside the pipeline, the IC is 0.008 with an error of 0.004, consistent with zero. And the “chosen” columns predict
nothing in the following five years. Hastie, Tibshirani and Friedman call it “the wrong way to do cross-validation”
(section 7.10.2 of their book); in finance it is more dangerous than in other fields because noise is almost
everything and any selection using the label finds something.

<aside class="callout" data-kind="aviso">

The rule is mechanical: **everything** that looks at the label or learns parameters from the data goes inside the
pipeline and is refitted in each split. That includes feature selection, imputers, scalers, PCA, class balancing and
any threshold chosen with the data. What cannot go in a pipeline (a feature chosen by looking at a chart of the whole
history, an idea taken from a paper that uses the same period) also counts as an attempt: it goes into the experiment
journal of lesson 2.2.4 and into the Deflated Sharpe of lesson 5.4.4.

</aside>

There are leaks the pipeline cannot prevent, because they are not in the process but in the data: a feature the
vendor dates before it is known (`revision_analistas`, lesson 5.2.2) or labels that overlap with the test (lessons
5.3.4 and 5.4.2). The pipeline protects against procedural mistakes; data mistakes are found by auditing dates.

## Exercises

1. Replace the pipeline's ridge with a scikit-learn `HistGradientBoostingRegressor` (`max_iter=100`,
   `learning_rate=0.05`, `max_leaf_nodes=8`, `min_samples_leaf=2000`, `random_state=0`) and evaluate it with
   `cross_val_score` and the same splitter. Does it beat the ridge in validation?

**Solution.**

```python
from sklearn.ensemble import HistGradientBoostingRegressor

hgb = Pipeline([
    ("rango", RangoPorFecha(num)), ("prep", prep),
    ("hgb", HistGradientBoostingRegressor(
        max_iter=100, learning_rate=0.05, max_leaf_nodes=8,
        min_samples_leaf=2000, random_state=0))])
hgb.set_output(transform="pandas")
v = cross_val_score(hgb, ent[cols], ent["y"], cv=cv,
                    scoring=puntuacion_ic)
print("boosting:", v.round(4), round(v.mean(), 4))
print("chosen ridge:", round(busqueda.best_score_, 4))
```
```output
boosting: [ 0.0053 -0.0085  0.0005  0.0099 -0.0118] -0.0009
chosen ridge: 0.007
```

Changing the last step is enough: the splitter, the *scorer* and the transformers are the same, and the boosting
model is refitted in each split like the ridge. It does not improve: its mean validation IC is −0.001 versus the
ridge's 0.007, with two clearly negative splits. The standard deviation across splits is about 0.009, so the
standard error of the mean of five is about 0.004 and the difference is about two errors: a hint, not proof. With so
little signal, the heavily regularized linear model is a hard baseline to beat; lesson 5.5.3 tunes boosting more
carefully.

2. Check how much the ridge's 2019-2025 IC changes if the linear pipeline's `StandardScaler` is fitted on the whole
   2012-2025 period (the “small” leak) instead of only on the training set.

**Solution.**

```python
todo = pd.concat([ent, pru])
w = Winsorizador(0.01).fit(ent[num])
imp = SimpleImputer(strategy="median", add_indicator=True)
imp.set_output(transform="pandas")
imp.fit(w.transform(ent[num]))
fuga = StandardScaler().fit(imp.transform(w.transform(todo[num])))
r = Ridge(1000.0).fit(fuga.transform(
    imp.transform(w.transform(ent[num]))), ent["y"])
p = r.predict(fuga.transform(imp.transform(w.transform(pru[num]))))
ic_fuga = ic_semanal(pru["fecha"], p, pru["y"]).mean()
ic_ok = ic_semanal(pru["fecha"], lineal.predict(pru[num]),
                   pru["y"]).mean()
print(f"no leak {ic_ok:.4f} | with leak {ic_fuga:.4f}")
```
```output
no leak 0.0181 | with leak 0.0182
```

The difference is small: standardizing with means from the whole period barely changes the order of the predictions
within each week. That is why this leak goes unnoticed, and why it is dangerous as a habit: the same move with a
feature selector or with a threshold chosen by looking at the label turns noise into signal, as in the last cell of
the lesson.

3. You are going to predict `ret_sig_21d` (the 21-session return, which overlaps between consecutive weeks) with the
   same pipeline. What would you change in the splitter and why?

**Solution.**

The gap. The label of a row from Friday $t$ ends 21 sessions later, about four weeks on. With a one-week gap, the
labels of the last three or four training weeks cover returns that fall inside the test block, and the model would
learn from them. The gap must be at least five dates (the four weeks of overlap plus one of margin). If, in addition,
the splits used interleaved test blocks (not only forward), you would also have to purge the training rows after the
test whose information overlaps and add an embargo: that is the purged k-fold of lesson 5.4.2. And in the
evaluation, with overlapping labels, the ICs of consecutive weeks are not independent: the t-statistic has to be
corrected (Newey-West, lesson 2.1.1).

## Self-assessment

1. What type of learning is training a model to predict whether a trade from a primary system will hit the take profit before the stop?

   - A. Supervised (classification): the label is the triple-barrier outcome of each trade
   - B. Unsupervised: there are no labels
   - C. Reinforcement: the agent decides
   - D. None: it is a backtest

   *Answer A: Supervised (classification): the label is the triple-barrier outcome of each trade.* There is an observed label for each example (which barrier was hit first), so it is supervised classification. It is the meta-labeling of lessons 5.3.2 and 5.3.3.

2. A `StandardScaler` is fitted on the whole 2012-2025 history and then a model is validated with 2019-2025 data. What has happened?

   - A. Nothing: scaling does not use the label
   - B. The training scaling uses means and standard deviations computed with test-period data: a leak, small in this case but of the same kind as the serious ones
   - C. The model will be more accurate
   - D. `fit_transform` has been misused

   *Answer B: The training scaling uses means and standard deviations computed with test-period data: a leak, small in this case but of the same kind as the serious ones.* Every stateful step (`fit`) must learn only from the training data of each split. Inside a `Pipeline`, cross-validation refits it in every fold and the leak disappears.

3. Why is scikit-learn's `TimeSeriesSplit` not enough for the 400-stock panel?

   - A. Because it does not accept pandas
   - B. Because it only makes three splits
   - C. Because it shuffles the rows
   - D. Because it splits by rows: with rows sorted by date, the boundary falls inside a week and that week ends up in both training and test; also, without a gap, the label of the last training week overlaps with the test

   *Answer D: Because it splits by rows: with rows sorted by date, the boundary falls inside a week and that week ends up in both training and test; also, without a gap, the label of the last training week overlaps with the test.* In the cell, the five `TimeSeriesSplit` splits share one week between training and test. A date-based splitter with a gap avoids it; full purging and embargo are lesson 5.4.2.

4. The 10 features most correlated with the label are chosen among 500 noise columns using the whole period, and then a ridge is validated with them. What comes out?

   - A. An IC close to zero, because they are noise
   - B. A scikit-learn error
   - C. A mean validation IC of 0.028, similar to the oracle's, which disappears with new data: the selection had already seen the test labels
   - D. A negative IC

   *Answer C: A mean validation IC of 0.028, similar to the oracle's, which disappears with new data: the selection had already seen the test labels.* It is “the wrong way to do cross-validation” of Hastie, Tibshirani and Friedman. With the selection inside the pipeline, the validation IC is 0.008 ± 0.004, and the “chosen” features give −0.001 in 2014-2018.

5. In the search, the ridge's validation IC rises with α until it levels off at huge values. Why?

   - A. Because a large α is always better
   - B. Because there is a bug in the splitter
   - C. Because the features are in ranks
   - D. Because the IC only looks at the order of the predictions: with a very large α, the coefficients are proportional to each feature's covariance with the label and the order no longer changes. With so little signal, heavy shrinkage is what generalizes best

   *Answer D: Because the IC only looks at the order of the predictions: with a very large α, the coefficients are proportional to each feature's covariance with the label and the order no longer changes. With so little signal, heavy shrinkage is what generalizes best.* When α dominates X′X, the ridge gives β ≈ X′y/α: the scale tends to zero, but the direction is fixed and the IC, which is scale-invariant, stops changing.

## Key points

- Supervised: there is a label (return, sign, barrier). Unsupervised: structure without a label (PCA, regimes, groups of assets). Reinforcement: sequential decisions with a reward (execution, hedging, market making).
- Every step with `fit` learns something from the data. Inside a `Pipeline`, validation refits it in each split and it does not see the test.
- A custom transformer inherits from `BaseEstimator` and `TransformerMixin`, stores what it learns in attributes ending in `_` and defines `get_feature_names_out` to work with `set_output`.
- In a panel you split by dates, not by rows, and with a gap at least equal to the label horizon.
- Selecting features with all the data and validating afterwards turns 500 noise columns into a validation IC of 0.028, equal to the oracle's ceiling.
- Hyperparameter search is scored with a trading metric (weekly IC) and the chosen model is compared with a simple baseline: here, 0.019 versus 0.018.

## Resources

### Research

- [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min

### Documentation

- [scikit-learn: Pipelines and composite estimators](https://scikit-learn.org/stable/modules/compose.html) · scikit-learn.org
- [scikit-learn: Common pitfalls and recommended practices (data leakage)](https://scikit-learn.org/stable/common_pitfalls.html) · scikit-learn.org
- [scikit-learn: Developing scikit-learn estimators](https://scikit-learn.org/stable/developers/develop.html) · scikit-learn.org
- [scikit-learn: Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html) · scikit-learn.org

### Books

- [Hastie, T., Tibshirani, R. and Friedman, J. (2009): The Elements of Statistical Learning, 2nd ed., section 7.10.2](https://hastie.su.domains/ElemStatLearn/) · hastie.su.domains

### Data

- [Level 5 data: weekly panel labels (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_etiquetas.parquet) · Download · panel5_etiquetas.parquet

Previous lesson: [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/)

Next lesson: [5.1.3 · Economic metrics for trading models: beyond accuracy](https://pythonparatrading.com/en/python-trading-course/level-5/economic-metrics-for-trading-models/)