---
title: "5.10.2 Data drift and concept drift"
description: "PSI, KS and adversarial validation against the model's IC; CUSUM, Page-Hinkley and a regime filter on data with changes on known dates."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/data-drift-concept-drift/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/data-drift-concept-drift.png"
---

# 5.10.2 · Data drift and concept drift: detecting them and what to do

> A trading model is estimated on the past and used in a future that changes. Sometimes the features change and the model still works; sometimes the relationship with the target changes and the features give no warning. With the level's drift data, which have each change on a known date, we measure what each tool detects, how long it takes, how many false alarms it gives and what to do afterwards.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.10 · Interpretability, robustness and model risk
- Duration: 70 min
- Prerequisites: [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/), [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/), [2.1.7 · Regime detection: hidden Markov models and structural breaks](https://pythonparatrading.com/en/python-trading-course/level-2/regime-detection-hmm-structural-breaks/)

## By the end of this lesson

- Distinguish data drift (the distribution of the features changes) from concept drift (the relationship with the target changes), and know which one degrades a model.
- Measure data drift with PSI, KS and adversarial validation, and calibrate their thresholds on periods without changes (autocorrelation, large samples, folds by month).
- Monitor performance with a moving average of the IC, a Page CUSUM, Page-Hinkley and a two-state filter, and choose the threshold by its false alarm rate and its delay.
- Compare retraining (expanding window, rolling window, weighted), switching off and retraining after an alarm, in a walk-forward with the truth known.

## Two drifts, and only one hurts

A supervised model learns the relationship between some features $X$ and a target $y$ from a sample. What it learns
is an approximation to $P(y \mid X)$, and what it uses to estimate it are examples drawn from $P(X)$. Either of the two
can change once the model is already in production, and the literature separates them (Gama et al., 2014;
Quiñonero-Candela et al., 2009):

| Type | What changes | Example in markets | Does it degrade the model? |
|---|---|---|---|
| Data drift (*covariate shift*) | $P(X)$ | volatility rises and all the risk features shift | not necessarily |
| Concept drift | $P(y \mid X)$ | a factor stops paying or reverses (momentum crash, lesson 5.1.1) | yes |
| Label drift (*prior shift*) | $P(y)$ | a long bull market: more positive labels | depends on the model and the metric |

Concept drift, in turn, can be abrupt (a regulatory change, a competitor arbitraging the signal), gradual (an anomaly
that is published and slowly fades) or recurrent (calm and stress regimes that come back).

```mermaid
flowchart TB
  D[Drift] --> X["Data: P(X)"]
  D --> C["Concept: P(y|X)"]
  C --> B[Abrupt]
  C --> G[Gradual]
  C --> R[Recurrent]
```

Data drift is the one you can see: it is enough to compare today's feature distribution with the training one, and
that can be done every day, without waiting for any label. Concept drift is the one that hurts, and it cannot be seen
by looking at the features: it only shows up in the results, once the labels the model predicted wrongly have arrived.
In real data you never know which of the two has happened or when. The level's `deriva5_*` data do tell you.

## The lab: six segments with known dates

`deriva5_muestras.parquet` has 30 fictitious assets over 3,018 NYSE sessions (2014-2025), six standardized features and
the next day's return already aligned. The target is linear in the features with the signal-to-noise ratio of daily
returns: a true IC of 0.10, an $R^2$ of 1%. The level's README publishes the segments:

| Seg. | From | Type | What changes |
|---|---|---|---|
| 0 | 2014-01-02 | base | f1, f2 and f3 with an effect (+8, +5 and −4 bp per unit); f4, f5 and f6 with no effect |
| 1 | 2017-01-03 | data | the mean of f1 rises by one standard deviation and the dispersion of f3 doubles; the relationship does not change |
| 2 | 2019-07-01 | concept | the effect of f1 changes sign (+8 to −8 bp); the features go back to the base |
| 3 | 2021-01-04 | irrelevant data | f4, with no effect, rises by 1.5 standard deviations and its dispersion ×1.5 |
| 4 | 2022-07-01 | gradual concept | over 12 months, the effect of f2 falls from 5 to 0 and that of f5 rises from 0 to 6 |
| 5 | 2023-07-03 | stable | the new relationship, with no further changes |

The protagonist is a **fixed model**: an OLS estimated on 2014-2016 and never retrained. Its quality is measured as
throughout the level, with the daily IC: the cross-sectional correlation between its prediction and the next day's
return of the 30 assets (lesson 5.1.3).

```python
import numpy as np
import pandas as pd

d = pd.read_parquet("deriva5_muestras.parquet")
seg = pd.read_csv("deriva5_verdad.csv", parse_dates=["desde"])
F = [f"f{k}" for k in range(1, 7)]
fechas = pd.DatetimeIndex(np.sort(d["fecha"].unique()))
n, NA = len(fechas), d["activo"].nunique()
X = d[F].to_numpy(float).reshape(n, NA, 6)  # session × asset × f
Y = d["objetivo"].to_numpy(float).reshape(n, NA)
s = np.searchsorted(seg["desde"], fechas, side="right") - 1

base = s == 0                                # 2014-2016
X1 = np.column_stack([np.ones(base.sum() * NA),
                      X[base].reshape(-1, 6)])
beta = np.linalg.lstsq(X1, Y[base].ravel(), rcond=None)[0]
print("fixed model coefficients (bp):",
      np.round(beta[1:] * 1e4, 1))

def ic_diario(P, Y):
    """Cross-sectional correlation between P and Y in each session."""
    P = P - P.mean(1, keepdims=True)
    Yc = Y - Y.mean(1, keepdims=True)
    return ((P * Yc).sum(1)
            / np.sqrt((P**2).sum(1) * (Yc**2).sum(1)))

ic = pd.Series(ic_diario(X @ beta[1:], Y), index=fechas)
tabla = seg[["segmento", "desde", "tipo"]].copy()
tabla["desde"] = tabla["desde"].dt.date
tabla["ic_fijo"] = ic.groupby(s).mean().round(3).to_numpy()
tabla
```
```output
fixed model coefficients (bp): [ 8.9  6.1 -4.5 -0.  -0.   1. ]
   segmento       desde               tipo  ic_fijo
0         0  2014-01-02               base    0.120
1         1  2017-01-03              datos    0.117
2         2  2019-07-01           concepto   -0.019
3         3  2021-01-04  datos_irrelevante   -0.026
4         4  2022-07-01   concepto_gradual   -0.035
5         5  2023-07-03            estable   -0.051
```

The OLS recovers the base relationship with its sampling error (8.9, 6.1 and −4.5 bp against 8, 5 and −4) and zeros
for the features with no effect. Its mean IC does not move with the 2017 data drift (0.117 against 0.120) and sinks
with the 2019 concept drift (−0.019). From then on it does not recover: in 2021 the drift of f4 is irrelevant to it,
and the gradual drift of 2022 also takes away the f2 part (−0.051 at the end).

## Data drift: PSI and KS

The Population Stability Index (PSI) comes from credit risk. The reference data are cut into $B$ bins (here,
deciles), the fraction $p_i$ of the reference and $q_i$ of the current period falling in each one is measured, and the
sum is taken:

$$
\text{PSI} = \sum_{i=1}^{B} (q_i - p_i)\,\ln\frac{q_i}{p_i}
$$

It is a symmetrized Kullback-Leibler divergence over the histogram: zero if the two histograms coincide, and it grows
with any difference, in mean, dispersion or shape. The usual rule of thumb reads a PSI below 0.1 as stable and above
0.25 as strong drift.

```python
def psi(ref, cur, nb=10):
    """Population Stability Index with deciles of the reference."""
    cortes = np.quantile(ref, np.linspace(0, 1, nb + 1)[1:-1])
    p = np.bincount(np.searchsorted(cortes, ref), minlength=nb)
    q = np.bincount(np.searchsorted(cortes, cur), minlength=nb)
    p = np.maximum(p / len(ref), 1e-6)
    q = np.maximum(q / len(cur), 1e-6)
    return float(np.sum((q - p) * np.log(q / p)))

ref = X[base]
filas = {}
for k in range(1, 6):
    cur = X[s == k]
    filas[f"{k} {seg['tipo'][k]}"] = [
        psi(ref[..., j].ravel(), cur[..., j].ravel())
        for j in range(6)]
pd.DataFrame(filas, index=F).T.round(2)
```
```output
                       f1   f2   f3    f4   f5   f6
1 datos              0.97  0.0  0.5  0.00  0.0  0.0
2 concepto           0.00  0.0  0.0  0.00  0.0  0.0
3 datos_irrelevante  0.00  0.0  0.0  1.23  0.0  0.0
4 concepto_gradual   0.00  0.0  0.0  1.27  0.0  0.0
5 estable            0.00  0.0  0.0  1.28  0.0  0.0
```

The PSI sees exactly the planted data drifts: 0.97 in f1 and 0.50 in f3 in segment 1 (the doubled dispersion of f3
counts as well), 1.23 in f4 from 2021, and zeros everywhere else. Above all, zeros in segment 2, the concept drift
one, and in the gradual one. The figure at the top puts the two things together: the PSI jumps in 2017 and in 2021
without the IC noticing, and the IC sinks in July 2019 without any PSI moving.

The classic alternative is the two-sample Kolmogorov-Smirnov test (`scipy.stats.ks_2samp`), which compares the
empirical distribution functions and gives a p-value. Before using either of them as an alarm you need to know what
they give when nothing is happening: each month of 2016 is compared with 2014-2015, with no planted change.

```python
from scipy import stats

ref = X[fechas.year <= 2015].reshape(-1, 6)   # 2014-2015
meses = pd.period_range("2016-01", "2016-12", freq="M")
psis, pvals = [], []
for m in meses:
    cur = X[fechas.to_period("M") == m].reshape(-1, 6)
    for j in range(6):
        psis.append(psi(ref[:, j], cur[:, j]))
        pvals.append(stats.ks_2samp(ref[:, j], cur[:, j]).pvalue)
psis, pvals = np.array(psis), np.array(pvals)
print(f"72 tests (12 months × 6 features) with no drift")
print(f"PSI: median {np.median(psis):.3f}, maximum {psis.max():.3f}")
print(f"KS with p < 0.05: {np.mean(pvals < 0.05):.0%} "
      f"(expected: 5%)")
ks_ano = [stats.ks_2samp(ref[:, j],
                         X[fechas.year == 2016][..., j].ravel())
          for j in range(6)]
print("all of 2016, KS p-value:",
      [f"{r.pvalue:.3f}" for r in ks_ano])
print("D statistic:", [round(float(r.statistic), 3)
                       for r in ks_ano])
```
```output
72 tests (12 months × 6 features) with no drift
PSI: median 0.018, maximum 0.055
KS with p < 0.05: 33% (expected: 5%)
all of 2016, KS p-value: ['0.000', '0.146', '0.009', '0.000', '0.315', '0.018']
D statistic: [0.029, 0.016, 0.023, 0.04, 0.013, 0.022]
```

Without drift, the monthly PSI stays at noise levels (median 0.018, maximum 0.055), well below 0.25. KS does not: 33%
of the 72 tests come out with $p < 0.05$, six times what is expected, and with all of 2016 against 2014-2015 four of the
six features “change” with $p < 0.05$ (three with $p \le 0.009$). There are two reasons. The features are AR(1) with
$\rho = 0.5$ within each asset: the rows are not independent, the effective sample size is several times smaller than
the number of rows and the p-value comes out too small. And with thousands of rows, tiny differences are significant:
the D statistics are between 0.013 and 0.040, differences of a few hundredths in the distribution function.

<aside class="callout" data-kind="aviso">

A KS p-value with financial data (autocorrelated, with thousands of rows) is not a probability of a false alarm. What
is useful is an effect size (the D statistic, the PSI) with a threshold calibrated on a period with no known changes.
And with six features monitored every month there are six tests per month: the false alarm rate is counted for the
whole set, not for each one (lesson 2.1.8).

</aside>

## Multivariate drift: adversarial validation

PSI and KS look at each feature separately. A drift in the relationship between features (two that stop being
correlated) cannot be seen that way. **Adversarial validation** looks at all of them at once: a classifier is trained to
distinguish the reference data from the current data. If it cannot (AUC ≈ 0.5), the two samples are
indistinguishable; if it can, the classifier's importance says which features have changed. Rabanser et al. (2019)
compare several drift detectors on several datasets: the best in their study is a two-sample test on a reduced
representation, and classifiers that separate domains turn out to be useful for describing the drift and seeing
whether it is harmful.

Two details decide whether the AUC means anything. The folds must be **whole months** (rows from neighboring days are
almost identical, as in lesson 5.4.1) and the AUC is computed **in each fold** and averaged. If the predictions of all
the folds are pooled and a single AUC is computed, it comes out below 0.5 with no drift: each fold has a different class
proportion and the classifier, with nothing to learn, predicts that proportion, which is the wrong way round in the
test fold.

```python
import lightgbm as lgb
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedGroupKFold

mref = fechas.year == 2016                 # the last year seen
mes = np.repeat((fechas.year * 100 + fechas.month).to_numpy(), NA)
mes = mes.reshape(n, NA)

def adversaria(m):
    """AUC of a classifier that separates 2016 from sessions m."""
    A = np.vstack([X[mref].reshape(-1, 6), X[m].reshape(-1, 6)])
    c = np.r_[np.zeros(mref.sum() * NA), np.ones(m.sum() * NA)]
    g = np.r_[mes[mref].ravel(), mes[m].ravel()]
    aucs, imp = [], np.zeros(6)
    cv = StratifiedGroupKFold(4, shuffle=True, random_state=0)
    for tr, te in cv.split(A, c, g):          # whole months
        clf = lgb.LGBMClassifier(
            n_estimators=60, num_leaves=8, min_child_samples=100,
            random_state=0, n_jobs=4, deterministic=True,
            force_row_wise=True, verbose=-1).fit(A[tr], c[tr])
        p = clf.predict_proba(A[te])[:, 1]
        aucs.append(roc_auc_score(c[te], p))  # AUC of each fold
        imp += clf.booster_.feature_importance("gain")
    return np.mean(aucs), F[int(imp.argmax())], imp.max() / imp.sum()

semestre = fechas.year.astype(str) + np.where(
    fechas.month <= 6, "-S1", "-S2")
adv = {}
for sm in semestre.unique():
    if sm.startswith("2016"):
        continue
    auc, top, peso = adversaria(semestre == sm)
    adv[sm] = {"auc": auc, "feature": top, "weight": peso,
               "ic_fijo": ic[semestre == sm].mean()}
adv = pd.DataFrame(adv).T.astype({"auc": float, "weight": float,
                                  "ic_fijo": float})
adv.loc[["2014-S1", "2015-S2", "2017-S1", "2019-S1", "2019-S2",
         "2020-S2", "2021-S1", "2023-S1", "2025-S2"]].round(3)
```
```output
           auc feature  weight  ic_fijo
2014-S1  0.493      f3   0.203    0.138
2015-S2  0.518      f4   0.198    0.103
2017-S1  0.826      f1   0.535    0.142
2019-S1  0.824      f1   0.533    0.123
2019-S2  0.519      f2   0.198   -0.031
2020-S2  0.496      f1   0.185   -0.025
2021-S1  0.782      f4   0.905   -0.029
2023-S1  0.780      f4   0.916   -0.026
2025-S2  0.790      f4   0.919   -0.027
```

In the half-years without drift the AUC stays between 0.49 and 0.52, what two identical samples give. In 2017-2019 it
rises to 0.83, with `f1` as the most important feature (53% of the gain). From 2021 it stays at 0.78-0.79, with `f4`
(92%). And in 2019-S2 and 2020-S2, after the concept change, it goes back to 0.52 and 0.50 while the IC of the fixed
model is negative (−0.031 and −0.025). Adversarial validation answers its own question very well, whether today's data
look like the training data, and that question is not the one that matters for the P&L.

So what is monitoring the features good for? **Diagnosing** (when performance drops, knowing whether the features
changed helps to understand why), warning about data errors (a vendor that changes a unit, a column that arrives
empty) and monitoring while the labels have not arrived. With a 21-day target, today's performance is known
within a month; the features, today.

## Monitoring performance: moving average, CUSUM and Page-Hinkley

Concept drift can only be seen in performance. The daily IC is a noisy series: with 30 assets and a true IC of 0.12,
its standard deviation is 0.18 (almost $1/\sqrt{29}$), so a day with a negative IC says nothing. The simplest detector,
the one in the level's README, is the 63-session moving average of the IC with an alarm when it falls below zero.

```python
cambio = pd.Timestamp("2019-07-01")
mm = ic.rolling(63).mean()
fecha_alarma = mm[(mm.index >= cambio) & (mm < 0)].index[0]
retraso = ((fechas >= cambio) & (fechas <= fecha_alarma)).sum()
print("first moving average < 0:", fecha_alarma.date(),
      f"({retraso} sessions after the change)")
antes = mm["2017-01-01":"2019-06-28"]
print(f"minimum before the change: {antes.min():.3f}")
```
```output
first moving average < 0: 2019-08-29 (43 sessions after the change)
minimum before the change: 0.071
```

It alarms on 2019-08-29, 43 sessions after the change, with no false alarms before (the moving average did not fall
below 0.071 in 2017-2019). It is easy to explain and slow: a 63-day window needs half of its content to be after the
change for the mean to cross zero.

Page's (1954) **CUSUM** is the classic sequential detector. For a drop in the mean from $\mu_0$:

$$
S_t = \max\left(0,\; S_{t-1} + (\mu_0 - k) - x_t\right), \qquad \text{alarm if } S_t > h
$$

$S_t$ accumulates how far the IC stays below $\mu_0 - k$ and resets to zero when the model is doing well. The slack
$k$ is usually set at half of the jump to be detected (here, from 0.12 to 0: $k = 0.06$) and the threshold $h$ decides
the balance between false alarms and delay. It is the same recursion as the CUSUM filter of lesson 5.3.5, which there
sampled prices and here monitors a model. **Page-Hinkley** is its variant without $\mu_0$: it compares each data point
with the mean since the last reset.

$$
m_t = \sum_{i \le t} \left(\overline{x}_i - x_i - \delta\right), \qquad \text{alarm if } m_t - \min_{j \le t} m_j > \lambda
$$

```python
x = ic.to_numpy()
t0 = np.searchsorted(fechas, pd.Timestamp("2017-01-03"))
mu0 = ic[base].mean()

def cusum(x, mu0, k, h):
    """One-sided Page CUSUM for a drop in the mean."""
    S, alarmas = 0.0, []
    for t, v in enumerate(x):
        S = max(0.0, S + (mu0 - k) - v)
        if S > h:
            alarmas.append(t)
            S = 0.0                       # reset after the alarm
    return alarmas

def page_hinkley(x, delta, lam):
    """Page-Hinkley: drop against the mean since the reset."""
    m = M = suma = 0.0
    cnt, alarmas = 0, []
    for t, v in enumerate(x):
        cnt += 1
        suma += v
        m += suma / cnt - v - delta
        M = min(M, m)
        if m - M > lam:
            alarmas.append(t)
            m = M = suma = 0.0
            cnt = 0
    return alarmas

def resumen(alarmas):
    f = fechas[t0:][alarmas]
    falsas = int((f < cambio).sum())
    tras = f[f >= cambio]
    r = int(((fechas >= cambio) & (fechas <= tras[0])).sum())
    return falsas, tras[0].date(), r

print(f"mean reference IC (2014-2016): {mu0:.3f}")
for h in [0.5, 1.0, 2.0, 3.0]:
    fa, f, r = resumen(cusum(x[t0:], mu0, 0.06, h))
    print(f"CUSUM h={h}: {fa:2d} false alarms 2017-2019, "
          f"first after the change {f} ({r} sessions)")
for lam in [1.0, 2.0, 3.0]:
    fa, f, r = resumen(page_hinkley(x[t0:], 0.03, lam))
    print(f"Page-Hinkley lambda={lam}: {fa:2d} false alarms, "
          f"{f} ({r} sessions)")
```
```output
mean reference IC (2014-2016): 0.120
CUSUM h=0.5: 19 false alarms 2017-2019, first after the change 2019-07-08 (5 sessions)
CUSUM h=1.0:  3 false alarms 2017-2019, first after the change 2019-07-16 (11 sessions)
CUSUM h=2.0:  0 false alarms 2017-2019, first after the change 2019-07-19 (14 sessions)
CUSUM h=3.0:  0 false alarms 2017-2019, first after the change 2019-08-01 (23 sessions)
Page-Hinkley lambda=1.0:  8 false alarms, 2019-07-11 (8 sessions)
Page-Hinkley lambda=2.0:  1 false alarms, 2019-07-18 (13 sessions)
Page-Hinkley lambda=3.0:  0 false alarms, 2019-07-29 (20 sessions)
```

The CUSUM with $h = 2$ alarms on 2019-07-19, after 14 sessions, with no false alarm in the previous two and a half
years, which also contain the 2017 data drift. With $h = 0.5$ it alarms after 5 sessions, at the cost of 19 false
alarms over that period, about seven per year. Page-Hinkley behaves similarly: with $\lambda = 3$, 20 sessions with no
false alarms. The top half of the figure follows the CUSUM statistic (without reset) in 2019: flat until July and,
from the change on, an almost straight rise.

*Fig. — March to October 2019: CUSUM statistic on the fixed model's daily IC, without reset (top), and filtered probability that the model has stopped working according to a two-state filter (bottom). Both cross their threshold on the same day.* Top, the CUSUM stays near zero until the 1 July change, rises almost in a straight line and crosses the h = 2 threshold on 19 July, amber point, 14 sessions later; by October it exceeds 8. Bottom, the filtered probability is near 0 until July, passes 0.9 on the same 19 July and stays near 1.

## False alarms against delay

A detector is not judged by one episode. The 2019 change is a single realization, and 14 sessions can be luck. To
measure the trade-off, many histories are simulated: 2,000 ten-year paths with ICs drawn at random from the days
without change (2017-2019-S1) to count false alarms, and one-year paths with ICs from after the change (2019-S2 to
2020) to measure the delay.

```python
tc = np.searchsorted(fechas, cambio)
t21 = np.searchsorted(fechas, pd.Timestamp("2021-01-01"))
sano, roto = x[t0:tc], x[tc:t21]     # IC before and after
rng = np.random.default_rng(0)
NS, T = 2000, 2520                   # 2,000 paths of 10 years
a = rng.choice(sano, (NS, T))
b = rng.choice(roto, (NS, 252))

def sim_cusum(h, k=0.06):
    S, fa = np.zeros(NS), np.zeros(NS)
    for t in range(T):
        S = np.maximum(0, S + (mu0 - k) - a[:, t])
        fa += S > h
        S[S > h] = 0
    S, r = np.zeros(NS), np.full(NS, 252)
    for t in range(252):
        S = np.maximum(0, S + (mu0 - k) - b[:, t])
        r[(S > h) & (r == 252)] = t + 1
    return fa.mean() / 10, np.median(r)

def sim_media(w):
    c = np.cumsum(a, 1)
    bajo = (c[:, w:] - c[:, :-w]) / w < 0
    fa = (bajo[:, 1:] & ~bajo[:, :-1]).sum(1).mean() / 10
    c = np.cumsum(np.hstack([np.zeros((NS, 1)), a[:, -w:], b]), 1)
    tras = ((c[:, w:] - c[:, :-w]) / w)[:, 1:] < 0
    r = np.where(tras.any(1), tras.argmax(1) + 1, 252)
    return fa, np.median(r)

filas = [("CUSUM", h, *sim_cusum(h)) for h in [1, 1.5, 2, 2.5, 3]]
filas += [("moving average", w, *sim_media(w)) for w in [21, 42, 63]]
pd.DataFrame(filas, columns=["detector", "parameter",
                             "false/year", "median delay"]
             ).round(3)
```
```output
         detector  parameter  false/year  median delay
0           CUSUM        1.0       1.030          11.0
1           CUSUM        1.5       0.193          17.0
2           CUSUM        2.0       0.038          23.0
3           CUSUM        2.5       0.007          30.0
4           CUSUM        3.0       0.001          36.0
5  moving average       21.0       0.204          18.0
6  moving average       42.0       0.002          36.0
7  moving average       63.0       0.000          54.0
```

*Fig. — False alarms per year versus median delay in detecting the IC drop, over 2,000 simulated paths built from the daily ICs before and after the change: CUSUM with several thresholds h and 21-, 42- and 63-session moving averages with a zero threshold.* Two curves falling from left to right. CUSUM: h = 1 gives 1.03 false alarms per year and an 11-session delay; h = 1.5, 0.19 and 17; h = 2, highlighted, 0.04 and 23; h = 3, 0.001 and 36. Moving average: 21 sessions gives 0.20 and 18; 42, 0.002 and 36; 63, none and 54. At the same false-alarm rate, the CUSUM arrives a few sessions earlier.

The typical delay of the CUSUM with $h = 2$ is 23 sessions, not 14, with 0.04 false alarms per year (one every 25
years). The 63-session moving average takes a median of 54: the 43 in the data were also somewhat lucky. The figure
puts the two families on the same plane. At the same false alarm rate, the CUSUM arrives a few sessions earlier: with
0.2 per year, 17 sessions against 18; with about 0.002-0.007, 30 against 36. The advantage exists (the CUSUM is optimal
for this problem in Lorden's sense), but it is modest. The important decision is where to sit on the curve: how much a
false alarm costs (switching off a model that works) against how much each day of delay costs (trading with a broken
model).

<aside class="callout" data-kind="nota">

The simulation resamples individual days and assumes the daily ICs are independent, which is almost true here (the
target is independent noise every day). With autocorrelated ICs (overlapping labels, like the panel's `ret_sig_21d`)
you have to resample by blocks, and false alarms will be more frequent than in a naive simulation.

</aside>

## Regimes: the two-state filter

Lesson 2.1.7 detected regimes with a hidden Markov model and distinguished between **filtered** probabilities (with the
information up to each day) and **smoothed** ones (with the whole sample, including the future). The same idea
monitors a model: two states, “works” (mean IC $\mu_0 = 0.12$) and “does not work” (mean IC 0), with the same
deviation, a probability of switching state of 0.002 per day and the filter recursion:

$$
\pi_t \propto (\pi_{t-1} A) \odot \phi(x_t)
$$

where $A$ is the transition matrix and $\phi(x_t)$ the density of today's IC in each state. The parameters are fixed in
advance, with 2014-2016, instead of estimating them on the whole sample: estimating them with data from 2019-2025 would
be looking at the future even though the filter is used.

```python
from scipy.stats import norm

medias = np.array([mu0, 0.0])     # the model works / does not work
sd = ic[base].std()
p = 0.998                         # expected duration: 500 sessions
A = np.array([[p, 1 - p], [1 - p, p]])
pi, prob = np.array([1.0, 0.0]), []
for v in x[t0:]:
    pi = (pi @ A) * norm.pdf(v, medias, sd)  # predict and update
    pi = pi / pi.sum()
    prob.append(pi[1])
prob = pd.Series(prob, index=fechas[t0:])
print(f"std of the daily IC: {sd:.3f}")
print("maximum P(broken) before the change:",
      round(prob[:"2019-06-28"].max(), 2))
print("first P(broken) > 0.9:",
      prob[(prob.index >= cambio) & (prob > 0.9)].index[0].date())
print(prob.resample("QE").mean()["2019":"2020"].round(2)
      .to_string())
```
```output
std of the daily IC: 0.184
maximum P(broken) before the change: 0.63
first P(broken) > 0.9: 2019-07-19
2019-03-31    0.02
2019-06-30    0.01
2019-09-30    0.83
2019-12-31    0.98
2020-03-31    0.98
2020-06-30    0.98
2020-09-30    0.97
2020-12-31    0.99
Freq: QE-DEC
```

The probability that the model is broken exceeds 0.9 on 2019-07-19, the same day as the CUSUM. It is no coincidence:
with two Gaussian states with known means, the log-likelihood ratio of each day is linear in $x_t$ with a slope
proportional to the difference of means, and the CUSUM with $k$ at the midpoint accumulates exactly that ratio. The
filter adds the prior probability of change and gives a figure that can be read directly. Its closest false alarm
before the change reached 0.63, the bottom half of the figure.

**Recurrent** regimes call for a different response. In the level's panel momentum reverses in every rebound (lesson
5.1.1): a detector that alarms once the rebound has already started arrives late to a regime that lasts a few months,
and retraining with rebound data teaches the model a relationship that is going to reverse again. If the regimes come
back, it is better for the model to see them as an observable feature and learn the interaction (lesson 5.10.1); drift
detection is for what does not come back.

## What to do: retrain, reweight, switch off

An alarm does not say what to do. First you diagnose: if the features also changed (PSI, adversarial validation), it
may be data drift with a real effect on the model, a data error or both; if not, it is concept drift.

```mermaid
flowchart TB
  M[IC monitor] --> A{Alarm?}
  A -- no --> M
  A -- yes --> D{High PSI or AUC?}
  D -- no --> C[Concept]
  D -- yes --> E[Data or errors]
  C --> R[Switch off, retrain]
  E --> R
```

The possible responses, compared in a walk-forward that retrains on the first day of each month with the targets
already known:

- **Retrain with an expanding window**: the whole past.
- **Retrain with a rolling window**: only the last year (252 sessions) or half year (126).
- **Reweight**: the whole past with weights that halve every 126 sessions.
- **The oracle**: the true coefficients of each day, the ceiling.

```python
def ols(i0, i1, semivida=None):
    """OLS with sessions i0..i1-1 (exponential weights)."""
    A = np.column_stack([np.ones((i1 - i0) * NA),
                         X[i0:i1].reshape(-1, 6)])
    y = Y[i0:i1].ravel()
    if semivida:
        edad = np.repeat(np.arange(i1 - i0)[::-1], NA)
        w = np.sqrt(0.5 ** (edad / semivida))
        A, y = A * w[:, None], y * w
    return np.linalg.lstsq(A, y, rcond=None)[0][1:]

mes_n = (fechas.year * 12 + fechas.month).to_numpy()
reentreno = {t for t in range(t0, n) if mes_n[t] != mes_n[t - 1]}

def walk_forward(entrenar):
    """Retrains on the first day of each month with what is known."""
    P, b = np.zeros((n, NA)), None
    for t in range(t0, n):
        if t in reentreno:
            b = entrenar(t - 1)      # targets known up to t
        P[t] = X[t] @ b
    return P

b_verdad = pd.read_csv("deriva5_coef_verdad.csv").iloc[:, 1:]
esquemas = {
    "fixed (2014-2016)": lambda t: beta[1:],
    "expanding": lambda t: ols(0, t),
    "rolling 252": lambda t: ols(t - 252, t),
    "rolling 126": lambda t: ols(t - 126, t),
    "weights, half-life 126": lambda t: ols(0, t, 126),
}
pred = {k: walk_forward(f) for k, f in esquemas.items()}
pred["oracle (true b)"] = np.einsum(
    "tik,tk->ti", X, b_verdad.to_numpy())
res = {}
for k, P in pred.items():
    v = pd.Series(ic_diario(P[t0:], Y[t0:]), index=fechas[t0:])
    res[k] = {"2017-2025": v.mean(),
              "2019-S2 to 2020": v["2019-07":"2020-12"].mean(),
              "2022-S2 to 2023-S1": v["2022-07":"2023-06"].mean()}
pd.DataFrame(res).T.round(3)
```
```output
                        2017-2025  2019-S2 to 2020  2022-S2 to 2023-S1
fixed (2014-2016)           0.007           -0.019              -0.035
expanding                   0.052            0.001               0.027
rolling 252                 0.101            0.055               0.105
rolling 126                 0.101            0.066               0.098
weights, half-life 126      0.101            0.054               0.104
oracle (true b)             0.111            0.091               0.108
```

Over the whole 2017-2025 period the fixed model barely adds anything (IC 0.007), the expanding window reaches 0.052 and
the three variants that forget the past, 0.101, close to the oracle (0.111). The difference lies in the 18 months that
follow the concept change: the expanding window gives 0.001, because up to 2019 it accumulates five years of a positive
f1 effect; the 126-session rolling window gives 0.066 and the 252-session one, 0.055. In the gradual period of
2022-2023 all the adaptive schemes follow the oracle (0.098-0.105 against 0.108): a change spread over twelve months is
easy to follow with a one-year window.

This is the opposite of what lesson 5.4.3 found on the weekly panel, where the expanding window won, and there is no
contradiction. There the effects are small and oscillate without a trend; here the change is large relative to the
noise and permanent. The optimal window depends on that (Pesaran and Timmermann, 2007, cited in lesson 5.4.3).

*Fig. — Coefficient of f1 estimated every month with an expanding window, a 252-session rolling window and a 126-session one, against the true value, which goes from +8 to −8 bp per unit in July 2019.* Lines from 2017 to 2025. The truth is a dashed step from +8 to −8 in July 2019. The 126-session rolling window crosses zero in late 2019 and oscillates between −11 and −4 afterwards; the 252-session one takes until January 2020 and is somewhat steadier. The expanding window, highlighted, already falls from 9 to 6.5 between 2017 and 2019 and then keeps falling slowly: it does not turn negative until November 2024.

The figure follows the coefficient of f1 estimated each month. The 126-session rolling window crosses zero at the end
of 2019 and the 252-session one in January 2020; afterwards they oscillate around −8 with their estimation error. The
expanding window was already falling before the change (from 9 to 6.5 in 2017-2019: the shift of f1 in 2017, with no
change in the target's mean, attenuates the slope of an OLS with a single constant for both periods) and does not turn
negative until November 2024.

The combination with the detector: the fixed model until the CUSUM alarm; after that, **switched off** for a few
sessions (no position) and retrained only with the data after the alarm.

```python
alarma = t0 + cusum(x[t0:], mu0, 0.06, 2.0)[0]
print("CUSUM alarm (h = 2):", fechas[alarma].date())

def tras_alarma(espera):
    """Fixed model until the alarm; switched off for `espera`
    sessions; then OLS only with data since the alarm (monthly)."""
    P, b = np.zeros((n, NA)), beta[1:]
    for t in range(t0, n):
        if alarma <= t < alarma + espera:
            b = np.zeros(6)
        elif t >= alarma + espera and (t in reentreno
                                       or t == alarma + espera):
            b = ols(alarma, t - 1)
        P[t] = X[t] @ b
    with np.errstate(invalid="ignore"):   # switched off: P = 0, IC 0/0
        v = ic_diario(P[t0:], Y[t0:])
    return pd.Series(np.nan_to_num(v), index=fechas[t0:])

for espera in [21, 63, 126]:
    v = tras_alarma(espera)
    print(f"switched off {espera:3d} sessions: IC 2017-2025 "
          f"{v.mean():.3f} · 2019-S2 to 2020 "
          f"{v['2019-07':'2020-12'].mean():.3f}")
```
```output
CUSUM alarm (h = 2): 2019-07-19
switched off  21 sessions: IC 2017-2025 0.103 · 2019-S2 to 2020 0.075
switched off  63 sessions: IC 2017-2025 0.101 · 2019-S2 to 2020 0.063
switched off 126 sessions: IC 2017-2025 0.100 · 2019-S2 to 2020 0.054
```

*Fig. — Mean IC of each scheme over the 18 months after the July 2019 concept drift (walk-forward with monthly retraining; the oracle uses the true coefficients).* Horizontal bars. Fixed 2014-2016: −0.019. Expanding: 0.001. Weights with a 126-session half-life: 0.054. Rolling 252: 0.055. Rolling 126: 0.066. CUSUM alarm, switch off for 21 sessions and retrain only on data since the alarm, highlighted: 0.075. Oracle: 0.091.

With 21 sessions switched off, the IC over the 18 months after the change is 0.075, better than any fixed window and
0.016 from the oracle. Waiting longer is costly: with 126 sessions switched off it falls to 0.054, because switching off
also switches off f2 and f3, which were still working. Switching off is an emergency measure (not trading with a model
under suspicion), not a solution: the value lies in coming back soon with a model estimated on data from the new
regime.

```python
movil = {fechas[t].date(): ols(t - 252, t) * 1e4
         for t in sorted(reentreno)}
movil = pd.DataFrame(movil, index=F).T
verdad = b_verdad.set_axis(F, axis=1).set_index(fechas) * 100
fechas_ver = ["2019-07-01", "2019-10-01", "2020-01-02",
              "2022-07-01", "2023-01-03", "2023-07-03", "2024-07-01"]
comp = pd.DataFrame({
    f: {"b1 rolling": movil.loc[pd.Timestamp(f).date(), "f1"],
        "b1 true": verdad.loc[f, "f1"],
        "b2 rolling": movil.loc[pd.Timestamp(f).date(), "f2"],
        "b2 true": verdad.loc[f, "f2"],
        "b5 rolling": movil.loc[pd.Timestamp(f).date(), "f5"],
        "b5 true": verdad.loc[f, "f5"]} for f in fechas_ver}).T
comp.round(1)
```
```output
            b1 rolling  b1 true  b2 rolling  b2 true  b5 rolling  b5 true
2019-07-01         7.9     -8.0         4.0      5.0         0.9      0.0
2019-10-01         2.4     -8.0         4.9      5.0         1.5      0.0
2020-01-02        -0.5     -8.0         4.7      5.0         0.9      0.0
2022-07-01        -7.2     -8.0         2.6      5.0         1.3      0.0
2023-01-03        -7.9     -8.0         2.2      2.4         3.0      3.1
2023-07-03        -8.2     -8.0         1.8      0.0         4.3      6.0
2024-07-01       -10.1     -8.0        -1.0      0.0         6.6      6.0
```

The table shows the limits of the 252-session rolling window against the true coefficients. In July 2019 the model
still believes in the old effect (7.9 against −8.0), in October it is at 2.4 and in January 2020, at −0.5. In the
gradual drift, the coefficient of f2 falls (2.6 in July 2022, 1.8 in July 2023, when the truth is already 0) and that of
f5 rises (from 1.3 to 4.3, against 0 and 6): the one-year window runs about six months behind, which is what averaging
a year of noisy data costs.

| Situation | Reasonable response |
|---|---|
| High PSI or AUC, normal performance | investigate (data error?), do not retrain by reflex |
| Performance drops, stable features | concept drift: switch off or reduce, retrain with recent data |
| Gradual change | rolling window or weights with a half-life of the order of the change |
| Recurrent regime | model the regime as a feature, do not chase it by retraining |
| Any | the full procedure (detector, threshold, response) fixed in advance and validated in walk-forward |

## Exercises

1. The 2017 data drift is a textbook *covariate shift*. The classic correction is to **reweight** the training data by
   the density ratio $P_{\text{current}}(x)/P_{\text{base}}(x)$, which is estimated with the adversarial classifier as
   $p/(1-p)$. Estimate the 2014-2016 OLS with those weights (current = 2017-2019-S1) and compare coefficients and IC
   with the unweighted fixed model.

**Solution.**

```python
actual = (fechas >= "2017-01-01") & (fechas < "2019-07-01")
A = np.vstack([X[base].reshape(-1, 6), X[actual].reshape(-1, 6)])
c = np.r_[np.zeros(base.sum() * NA), np.ones(actual.sum() * NA)]
clf = lgb.LGBMClassifier(
    n_estimators=60, num_leaves=8, min_child_samples=100,
    random_state=0, n_jobs=4, deterministic=True,
    force_row_wise=True, verbose=-1).fit(A, c)
pb = clf.predict_proba(X[base].reshape(-1, 6))[:, 1]
w = pb / (1 - pb)
w = w / w.mean()
sw = np.sqrt(w)
beta_w = np.linalg.lstsq(X1 * sw[:, None], Y[base].ravel() * sw,
                         rcond=None)[0]
print("unweighted (bp):", np.round(beta[1:] * 1e4, 1))
print("weighted (bp):", np.round(beta_w[1:] * 1e4, 1))
print(f"effective size: {w.sum()**2 / (w**2).sum() / w.size:.0%} "
      "of the rows")
for nombre, b in [("unweighted", beta), ("weighted", beta_w)]:
    v = ic_diario(X[actual] @ b[1:], Y[actual]).mean()
    print(f"IC 2017-2019-S1 {nombre}: {v:.3f}")
```
```output
unweighted (bp): [ 8.9  6.1 -4.5 -0.  -0.   1. ]
weighted (bp): [ 5.5  4.4 -4.3  0.9 -2.   1.2]
effective size: 13% of the rows
IC 2017-2019-S1 unweighted: 0.117
IC 2017-2019-S1 weighted: 0.112
```

The weights do not help, they hurt. To make 2014-2016 look like 2017-2019, the classifier gives a lot of weight to the
few base rows with high f1 and extreme f3, and the effective sample size falls to 13% of the rows. The coefficients
become noisier (that of f1 falls from 8.9 to 5.5, that of f5 comes out at −2.0 with no true effect) and the IC in
2017-2019-S1 falls from 0.117 to 0.112. Reweighting corrects the bias of a **misspecified** model that extrapolates to
a sparsely populated area of the training data; with a correct linear model, as here, unweighted OLS is already
unbiased in any area and the weights only add variance. And it is useless against concept drift: if $P(y \mid X)$
changes, no weighting of the past teaches the new relationship.

2. The CUSUM monitored the fixed model. Now monitor the 252-session rolling window model with the same detector
   ($k = 0.06$, $h = 2$, $\mu_0$ = its mean IC in 2017-2018). When does it alarm, and what happens in the gradual
   drift?

**Solution.**

```python
ic_movil = pd.Series(ic_diario(pred["rolling 252"][t0:], Y[t0:]),
                     index=fechas[t0:])
mu_m = ic_movil[:"2018-12-31"].mean()
al = cusum(ic_movil.to_numpy(), mu_m, 0.06, 2.0)
print(f"mean IC 2017-2018: {mu_m:.3f}")
print("alarms:", [str(f.date()) for f in ic_movil.index[al]])
print(ic_movil.groupby(ic_movil.index.year).mean().round(3)
      .to_string())
```
```output
mean IC 2017-2018: 0.112
alarms: ['2019-07-19', '2019-08-23', '2019-11-26']
2017    0.112
2018    0.112
2019    0.058
2020    0.084
2021    0.097
2022    0.105
2023    0.097
2024    0.133
2025    0.115
```

The CUSUM of the rolling model also alarms on 2019-07-19 and again on 2019-08-23 and 2019-11-26: the one-year window
takes about six months to change sign, and in the meantime its IC drops (0.058 on average in 2019). In the gradual
drift of 2022-2023 it does not alarm: the window follows the change with its delay and the annual IC stays between
0.097 and 0.105. Two lessons: a model that is retrained also needs a monitor, because its adaptation has a delay that
can cost months; and after each retraining or model change you have to recalibrate $\mu_0$ and reset the detector, or
the alarms will be measuring the old model.

3. Design the monitoring of a weekly stock selection model whose label is the 21-session return (overlapping, like the
   panel's `ret_sig_21d`). What do you monitor every day, what every week, and how do you calibrate the thresholds?

**Solution.**

Labels at 21 sessions take a month to become known, so performance can only be monitored with a month of delay added
to that of the detector. In the meantime, the features and the predictions are monitored every day: the PSI of each
feature and of the distribution of the predictions against training, and a weekly adversarial validation with folds of
whole weeks; they serve to catch data errors and abrupt changes in the environment, not to decide that the model has
stopped working. Performance is monitored with the weekly IC of the labels already completed; since they overlap, it is
advisable to use only non-overlapping weeks or a CUSUM on ICs of four-week blocks. The thresholds ($h$, the window) are
calibrated with a block simulation over a period with no known changes, deciding beforehand how many false alarms per
year are acceptable, and the full procedure (detector, threshold, switch-off, retraining) is evaluated in a
walk-forward before it is used. And with several signals monitored, the false alarm rate is counted for the whole set
(lesson 2.1.8).

## Self-assessment

1. In 2017 the PSI of `f1` rises to 0.97 and that of `f3` to 0.50, and the IC of the fixed model goes from 0.120 to 0.117. What does this indicate?

   - A. That the model is broken and has to be retrained as soon as possible
   - B. That the PSI is computed wrongly, because such a high PSI always degrades the model
   - C. A data drift that does not touch the relationship with the target: the distribution of the features changed, but the model still ranks just as well
   - D. A concept drift, because the PSI is above 0.25

   *Answer C: A data drift that does not touch the relationship with the target: the distribution of the features changed, but the model still ranks just as well.* The PSI measures whether the distribution of a feature changes, not whether its relationship with the target changes. Here the mean of `f1` rises by one standard deviation and the dispersion of `f3` doubles, but the effect of each feature is the same. A PSI alarm is a signal to investigate, not to retrain.

2. In July 2019 the effect of `f1` changes sign. What detects the change?

   - A. Only what looks at the model's performance (its IC, its errors), because the features do not change distribution: the PSI and adversarial validation stay at 0.00 and 0.52
   - B. The PSI of `f1`, which rises above 0.25
   - C. The KS of `f1`, with p < 0.05
   - D. The SHAP importance of `f1`, which falls to zero

   *Answer A: Only what looks at the model's performance (its IC, its errors), because the features do not change distribution: the PSI and adversarial validation stay at 0.00 and 0.52.* It is pure concept drift: P(X) does not change and P(y|X) does. No measure on the features can see it. It only shows up in the results, and therefore late: new labels are needed to notice that the model is failing.

3. With 72 KS tests (12 months of 2016 × 6 features) and no planted drift at all, 33% come out with p < 0.05. Why?

   - A. Because 2016 has a hidden drift
   - B. Because KS does not work for continuous variables
   - C. Because the data cover 30 assets
   - D. Because KS assumes independent observations and the features are autocorrelated (AR(1) of 0.5): with thousands of dependent rows, its p-values come out too small

   *Answer D: Because KS assumes independent observations and the features are autocorrelated (AR(1) of 0.5): with thousands of dependent rows, its p-values come out too small.* With dependence, the effective sample size is much smaller than the number of rows and the test is too liberal. Moreover, with large samples any tiny difference comes out significant. An effect size (the D statistic, the PSI) with a threshold calibrated on periods without changes is better.

4. A CUSUM with h = 2 alarms 14 sessions after the 2019 change and the 63-session moving average of the IC, 43. In the simulation, the median delay of the CUSUM with h = 2 is 23 sessions. How should this be read?

   - A. The CUSUM always takes 14 sessions
   - B. The 14 in the data are a lucky realization; the typical delay, with 0.04 false alarms per year, is about 23 sessions. A detector is judged by its pair of false alarms and delay over many realizations, not by one episode
   - C. The simulation is wrong, because it does not match the data
   - D. The moving average is better, because it gives no false alarms

   *Answer B: The 14 in the data are a lucky realization; the typical delay, with 0.04 false alarms per year, is about 23 sessions. A detector is judged by its pair of false alarms and delay over many realizations, not by one episode.* A single change is a single sample of the delay. The simulation with the ICs from before and after gives the distribution; with the same number of false alarms, the CUSUM arrives a few sessions earlier than a moving average, not twenty.

5. After the 2019 change, the expanding window has an IC of 0.001 and the 252-session rolling window, 0.055. Lesson 5.4.3 recommended the expanding window on the weekly panel. Is there a contradiction?

   - A. Yes: lesson 5.4.3 was wrong
   - B. Yes: there is a leak in the rolling window here
   - C. No: with a break that is large and permanent relative to the noise, the earlier history teaches the opposite of what is happening and is better forgotten; with small effects that oscillate, as in the panel, using all the data removes more error than it adds
   - D. No: the expanding window always wins in the long run

   *Answer C: No: with a break that is large and permanent relative to the noise, the earlier history teaches the opposite of what is happening and is better forgotten; with small effects that oscillate, as in the panel, using all the data removes more error than it adds.* The optimal window depends on the size and persistence of the changes relative to the noise. Here the coefficient of `f1` goes from +8 to −8 bp and stays there: the expanding window does not reflect it with the correct sign until November 2024.

## Key points

- Data drift (P(X) changes) and concept drift (P(y|X) changes) are different problems. In the level's data, the two data drifts trigger the PSI (0.97 and 1.23) without touching the model's IC, and the concept drift sinks it from 0.120 to −0.019 without moving any PSI.
- PSI, KS and adversarial validation monitor the features. They need thresholds calibrated without changes: KS gives 33% false alarms with autocorrelated features, and the adversarial AUC is computed per fold and with folds of whole months.
- The drift that matters is monitored in performance. A CUSUM on the daily IC with k = half the expected drop and h = 2 alarms after 14 days; the 63-session moving average, after 43. A two-state filter alarms on the same day as the CUSUM.
- A detector is chosen by its pair of false alarms and delay measured by simulation. At the same false alarm rate, the CUSUM beats the moving average by a few sessions.
- In the face of a large and permanent break, the expanding window takes years to change sign. The rolling window adapts in a few months. Retraining only with data after the alarm gives the best IC after the change (0.075 against the oracle's 0.091).
- Switching off is the emergency response, not the solution: it also switches off what is still working. Detecting, diagnosing with the PSI and adversarial validation, and retraining is a procedure that is fixed and validated before it is used.

## Resources

### Documentation

- [Page (1954): Continuous Inspection Schemes, Biometrika 41(1-2), 100-115](https://doi.org/10.1093/biomet/41.1-2.100) · doi.org
- [Gama, Žliobaitė, Bifet, Pechenizkiy and Bouchachia (2014): A survey on concept drift adaptation, ACM Computing Surveys 46(4)](https://doi.org/10.1145/2523813) · doi.org
- [Rabanser, Günnemann and Lipton (2019): Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, arXiv:1810.11953](https://arxiv.org/abs/1810.11953) · arxiv.org
- [SciPy: scipy.stats.ks_2samp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html) · docs.scipy.org

### Books

- [Quiñonero-Candela, Sugiyama, Schwaighofer and Lawrence (eds.) (2009): Dataset Shift in Machine Learning, MIT Press](https://doi.org/10.7551/mitpress/9780262170055.001.0001) · doi.org

### Data

- [Level 5 data: samples with drift (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/deriva5_muestras.parquet) · Download · deriva5_muestras.parquet
- [Level 5 data: segments and true coefficients (CSV)](https://pythonparatrading.com/curso/datos/nivel-5/deriva5_coef_verdad.csv) · Download · deriva5_coef_verdad.csv

Previous lesson: [5.10.1 · SHAP, partial dependence and the stability of feature importances](https://pythonparatrading.com/en/python-trading-course/level-5/shap-partial-dependence-stability/)

Next lesson: [5.10.3 · Ensembles, signal combination and synthetic data](https://pythonparatrading.com/en/python-trading-course/level-5/ensembles-signal-combination-synthetic-data/)