--- title: "7.3.2 Retraining, validation and champion/challenger" description: "A monthly retraining cycle with automatic gates that stop three planted faults, and champion/challenger with MLflow aliases." url: "https://pythonparatrading.com/en/python-trading-course/level-7/model-retraining-validation-champion-challenger/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/model-retraining-validation-champion-challenger.png" --- # 7.3.2 · Scheduled retraining, automatic validation and champion/challenger > Retraining every month is easy; the hard part is not deploying in the month the data pipeline breaks. This lesson builds the full cycle on data with a regime change and three faults on known dates: it trains, passes four automatic gates, compares with the production model, tests it in shadow mode and promotes or discards it with aliases in the MLflow registry. - Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/) - Module: 7.3 · MLOps for trading models - Duration: 70 min - Prerequisites: [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/), [5.10.2 · Data drift and concept drift: detecting them and what to do](https://pythonparatrading.com/en/python-trading-course/level-5/data-drift-concept-drift/), [4.5.3 · Paper trading, shadow trading and slippage monitoring](https://pythonparatrading.com/en/python-trading-course/level-4/paper-trading-shadow-trading-and-slippage/), [7.3.1 · Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end](https://pythonparatrading.com/en/python-trading-course/level-7/experiment-model-data-tracking-mlflow-dvc/) ## By the end of this lesson - Schedule a retraining cycle with training and validation windows that only use targets already closed on the day of the cycle. - Write automatic validation gates (data contract, leakage, behavior and performance) and know which fault each one stops. - Compare a challenger with the champion over the same sessions with a paired test and choose between requiring superiority or only passing the gates. - Test a challenger in shadow mode with operational checks and promote it or roll back by moving aliases in the MLflow model registry. ## A cycle, not a training run Lesson 5.4.3 decided **when** to retrain and with which window, and lesson 5.10.2, how to detect that the world has changed. Both work in a lab: a loop that retrains a thousand times with clean data. In production, every retraining is an event: a process that starts on the first business day of the month, asks for data from a pipeline that may have failed that week, trains, and has to decide with nobody watching whether the model that just came out replaces the one that moves money. This lesson builds that process. MLflow's experiment and model registry is introduced in lesson 7.3.1; here it is used for what is needed at the end: versions and aliases. ```mermaid flowchart TB C[Calendar] --> D[Data snapshot] D --> E[Train] E --> P[Gates] P -->|fails| X[Rejection and alert] P -->|passes| R[Registry] R --> S[Shadow] S --> V[Champion] ``` In MLOps vocabulary, the **champion** is the model in production and the **challenger** the one that aims to replace it. For the handover to be automatic, three things are needed: knowing which data each candidate was trained on (the snapshot), a list of checks that a broken model does not pass (the gates) and a mechanism to switch models and undo the switch without deploying code (the registry aliases). ## The lab: a changing regime and three faults The data is synthetic, with a known answer. Fifty assets, five persistent features (an AR(1) with coefficient 0.9, like a momentum or a value signal that change slowly) and a next-day return that depends on them with coefficients that change on **April 1, 2024**: the first feature goes from adding to subtracting, the third starts to count, and the second and fourth stay the same. ```python import numpy as np import pandas as pd from scipy.stats import rankdata from sklearn.linear_model import Ridge rng = np.random.default_rng(732) fechas = pd.bdate_range("2020-06-01", "2025-12-31") T, N, K = len(fechas), 50, 5 CAMBIO = pd.Timestamp("2024-04-01") F = np.zeros((T, N, K)) # features at the close of t F[0] = rng.standard_normal((N, K)) for t in range(1, T): # AR(1), persistence 0.9 F[t] = 0.9 * F[t - 1] + np.sqrt(0.19) * rng.standard_normal((N, K)) B_ANTES = np.array([0.4, 0.3, 0.0, 0.2, 0.0]) B_DESPUES = np.array([-0.3, 0.3, 0.4, 0.2, 0.0]) B = np.where((fechas < CAMBIO)[:, None], B_ANTES, B_DESPUES) VOL = 0.012 # R[t]: return from t to t+1, known at the close of t+1 R = VOL * (0.15 * np.einsum("tnk,tk->tn", F, B) + rng.standard_normal((T, N))) def ic_dia(P, Y): """Spearman IC of each day (rows) between prediction and return.""" a = rankdata(P, axis=1) - (N + 1) / 2 b = rankdata(Y, axis=1) - (N + 1) / 2 return (a * b).sum(1) / np.sqrt((a * a).sum(1) * (b * b).sum(1)) oraculo = pd.Series(ic_dia(np.einsum("tnk,tk->tn", F, B), R), index=fechas) print(f"{T} sessions x {N} assets x {K} features") print("oracle IC before and after the change:", round(oraculo[:CAMBIO].iloc[:-1].mean(), 3), round(oraculo[CAMBIO:].mean(), 3)) ``` ```output 1458 sessions x 50 assets x 5 features oracle IC before and after the change: 0.077 0.078 ``` The oracle's IC (the predictions with each day's true coefficients) is 0.077 before the change and 0.078 after: the problem does not get harder, it becomes **a different one**. A model that does not notice will keep betting on the first feature just as it starts to subtract. On top of that, three faults in the data pipeline, each one visible only in the snapshot that a specific cycle receives, as happens in reality: the vendor fixes the data, or someone reverts the code change, and the following month everything is fine. ```python AVERIAS = { pd.Timestamp("2023-09-01"): "f1 frozen in August", pd.Timestamp("2023-12-01"): "target with the flipped sign", pd.Timestamp("2025-02-03"): "f5 built from the target", } def foto(ciclo): """Features and targets as the data pipeline delivers them on the day of the cycle (with the fault, if there is one that day).""" Fx, Rx = F.copy(), R.copy() a = AVERIAS.get(ciclo, "") if a.startswith("f1"): # the vendor repeats the value i = np.where((fechas >= "2023-08-01") & (fechas < "2023-09-01"))[0] Fx[i, :, 0] = F[i[0] - 1, :, 0] elif a.startswith("target"): # precio_t / precio_t+1 - 1 Rx = -R elif a.startswith("f5"): # a join on the wrong date Fx[:, :, 4] = F[:, :, 4] + R / VOL return Fx, Rx for c, a in AVERIAS.items(): print(c.date(), a) ``` ```output 2023-09-01 f1 frozen in August 2023-12-01 target with the flipped sign 2025-02-03 f5 built from the target ``` - **September 2023**: the vendor repeated July's last value of `f1` throughout August. It is the most common failure of data that arrives through an external service: there is no error, just a number that does not change. - **December 2023**: a change in the code that computes the target inverts it (today's price divided by tomorrow's, instead of the other way around). Every column looks fine. - **February 2025**: a `merge` on the wrong date puts information from the target itself into `f5`. It is leakage: the model learns from something it will not have live. The cycle runs before the open on the first business day of each month, from February 2023 to December 2025. On that day, the last closed target is the return from the day before yesterday to yesterday: the row of the day before yesterday is the last one that can go into training. Behind it, 42 sessions (two months) are set aside for validation, and behind those, 252 for training: ```python VENT, VAL = 252, 42 # training and validation sessions inicio = fechas.get_loc(pd.Timestamp("2023-01-02")) ciclos = [fechas[fechas >= d][0] for d in pd.date_range("2023-02-01", "2025-12-01", freq="MS")] def ajustar(Fx, Rx, a, b): X, y = Fx[a:b].reshape(-1, K), Rx[a:b].ravel() return Ridge(alpha=1.0).fit(X, y) def predecir(m, Fx, a, b): return m.predict(Fx[a:b].reshape(-1, K)).reshape(b - a, N) def ventanas(ciclo): """Training and validation indices of the cycle. The last row with a closed target is the one two sessions earlier.""" i = fechas.get_loc(ciclo) fin = i - 1 # rows < fin: target known return (fin - VAL - VENT, fin - VAL), (fin - VAL, fin) (e0, e1), (v0, v1) = ventanas(pd.Timestamp("2024-05-01")) print(len(ciclos), "cycles, from", ciclos[0].date(), "to", ciclos[-1].date()) print("cycle 2024-05-01 · trains", fechas[e0].date(), "to", fechas[e1 - 1].date(), "· validates", fechas[v0].date(), "to", fechas[v1 - 1].date()) champion_0 = ajustar(F, R, inicio - 1 - VENT, inicio - 1) print("initial champion coefficients (x1000):", (champion_0.coef_ * 1e3).round(2)) ``` ```output 35 cycles, from 2023-02-01 to 2025-12-01 cycle 2024-05-01 · trains 2023-03-15 to 2024-02-29 · validates 2024-03-01 to 2024-04-29 initial champion coefficients (x1000): [ 0.59 0.44 -0.1 0.39 0.08] ``` The initial champion was trained on the 252 sessions before January 2023 and recovers the structure of the first regime well: large positive coefficients on `f1`, `f2` and `f4`, almost zero on `f3` and `f5`. Two details of the cycle that matter in production: - The **snapshot** (`foto(ciclo)`) is what the pipeline delivered that day, not what is in the database today. If the vendor corrects August in October, retraining “the September cycle” with today's data gives a different model. For a cycle to be reproducible, the snapshot is stored or identified by a fingerprint, as done further below. - Each cycle's model is trained with a fixed recipe (Ridge, `alpha=1`, 252 sessions). Changing the recipe (other features, another algorithm) is not a retraining: it is a new model, and it is validated as in lessons 5.4.3 and 5.4.4 before entering the cycle. ## Four gates A gate is an automatic check with a threshold fixed in advance. If it fails, the candidate is not registered as a challenger, the champion stays and an alert fires (lesson 4.5.4). The order goes from cheap and general to expensive and specific: | Gate | What it checks | Which fault it stops | |---|---|---| | Data | NaN, values repeated from one day to the next in the last 63 sessions | frozen or empty data | | Leakage | correlation of each feature with the same day's target below 0.3 | a feature built from the future | | Behavior | correlation between the predictions of the candidate and of the champion on the same inputs ≥ 0.5 | inverted or empty models | | Performance | daily IC of the candidate against the champion's on the validation set | worse (or not better) models | The leakage gate rests on a domain fact: in daily data no legitimate feature explains the next day's return with a correlation of 0.3; in the lab, the largest in the cycles without faults is 0.085. The behavior gate rests on two models of the same recipe one month apart being very similar: a candidate that predicts the opposite of the champion is not an improvement, it is a mistake. And the performance gate uses the **day-by-day** IC difference over the same sessions, a paired test: both models see the same market, so most of the noise cancels out in the difference. Its standard error is computed with Newey-West, because with persistent features the ICs of consecutive days are correlated: $$ t = \frac{\overline{d}}{\sqrt{\hat{\sigma}^2_{NW} / n}}, \qquad d_t = IC_t^{\text{challenger}} - IC_t^{\text{champion}}, \qquad \hat{\sigma}^2_{NW} = \hat{\gamma}_0 + 2\sum_{k=1}^{L}\left(1 - \frac{k}{L+1}\right)\hat{\gamma}_k $$ ```python def t_hac(d, L=5): """t of the mean of d with Newey-West variance (L lags).""" m, e = d.mean(), d - d.mean() v = e @ e / len(d) for k in range(1, L + 1): v += 2 * (1 - k / (L + 1)) * (e[k:] @ e[:-k]) / len(d) return m / np.sqrt(v / len(d)) if v > 0 else 0.0 def puertas(ciclo, Fx, Rx, cand, champ, exigir_mejora): """Returns (the gate that fails or 'ok', detail).""" (e0, e1), (v0, v1) = ventanas(ciclo) # 1. data contract: repeated values in the last 63 ult = Fx[v1 - 63:v1] rep = (ult[1:] == ult[:-1]).mean(axis=(0, 1)) if np.isnan(Fx[e0:v1]).any(): return "data", "NaN present" if rep.max() > 0.05: return "data", f"f{rep.argmax() + 1} repeated {rep.max():.0%}" # 2. leakage: no feature may explain the day's target X, y = Fx[e0:v1].reshape(-1, K), Rx[e0:v1].ravel() rho = np.array([np.corrcoef(X[:, k], y)[0, 1] for k in range(K)]) if np.abs(rho).max() > 0.3: k = np.abs(rho).argmax() return "leakage", f"corr(f{k + 1}, target) = {rho[k]:.2f}" # 3. behavior: similar to the champion on the same inputs pc = predecir(cand, Fx, v0, v1) pp = predecir(champ, Fx, v0, v1) r = np.corrcoef(pc.ravel(), pp.ravel())[0, 1] if r < 0.5: return "behavior", f"corr. with champion {r:+.2f}" if not exigir_mejora: return "ok", f"corr. with champion {r:+.2f}" # 4. performance: daily IC against the champion, same sessions d = ic_dia(pc, Rx[v0:v1]) - ic_dia(pp, Rx[v0:v1]) t = t_hac(d) det = f"dIC {d.mean():+.3f}, t {t:+.1f}" return ("ok" if d.mean() > 0.01 and t > 2 else "no improvement"), det for c in [pd.Timestamp("2023-06-01"), *AVERIAS]: Fx, Rx = foto(c) (e0, e1), _ = ventanas(c) cand = ajustar(Fx, Rx, e0, e1) print(c.date(), *puertas(c, Fx, Rx, cand, champion_0, True)) ``` ```output 2023-06-01 no improvement dIC +0.003, t +0.9 2023-09-01 data f1 repeated 35% 2023-12-01 behavior corr. with champion -0.95 2025-02-03 leakage corr(f5, target) = 0.70 ``` All three faults are caught, each one by a different gate, and a normal cycle reaches the last one: - **September 2023**: 35% of the `f1` values in the last 63 sessions are equal to the previous day's. August is 22 of those 63 sessions; with real data that moves, the proportion is zero. - **December 2023**: the features are fine and nothing explains the day's target, so it passes the first two. But its predictions have a correlation of −0.95 with the champion's: the model has learned the market backwards. - **February 2025**: `f5` has a correlation of 0.70 with the same day's target. No honest feature does that. - **June 2023**: a cycle without faults in the initial regime. The candidate beats the champion by 0.003 of IC with t = 0.9: there is no reason to switch models. ## Champion/challenger: require an improvement? The underlying question is what a candidate has to prove to replace the champion. There are two reasonable answers, and the lab lets you measure both, alongside two baselines: never retraining, and retraining every month while deploying whatever comes out. - **Fixed**: the January 2023 champion forever. - **Always**: every cycle retrains on the last 252 sessions and deploys, with no gates. - **Superiority**: classic champion/challenger. The candidate passes the four gates and, at the performance gate, has to beat the champion by more than 0.01 of IC with t > 2. The validated model is deployed. - **Gates**: the candidate passes the first three gates; if it does, it is refitted on the last 252 sessions (including the validation ones) and deployed. It is not required to improve. ```python def politica(nombre): """Simulates the cycles and returns the live daily IC, the decision log and the deployed models.""" champ, vigentes, reg = champion_0, [(inicio, champion_0)], [] for c in ciclos: if nombre == "fixed": break Fx, Rx = foto(c) (e0, e1), (v0, v1) = ventanas(c) i = fechas.get_loc(c) if nombre == "always": # no gates champ = ajustar(Fx, Rx, v1 - VENT, v1) vigentes.append((i, champ)) reg.append((c, "deployed", "")) continue cand = ajustar(Fx, Rx, e0, e1) mejora = nombre == "superiority" g, det = puertas(c, Fx, Rx, cand, champ, mejora) reg.append((c, g, det)) if g == "ok": # gates: refit on everything champ = cand if mejora else ajustar(Fx, Rx, v1 - VENT, v1) vigentes.append((i, champ)) P = np.zeros((T, N)) for j, (a, m) in enumerate(vigentes): b = vigentes[j + 1][0] if j + 1 < len(vigentes) else T P[a:b] = predecir(m, F, a, b) ic = pd.Series(ic_dia(P[inicio:], R[inicio:]), index=fechas[inicio:]) return ic, reg, len(vigentes) - 1 res = {n: politica(n) for n in ("fixed", "always", "superiority", "gates")} filas = {} for n, (ic, reg, k) in res.items(): mes = ic.groupby(ic.index.to_period("M")).mean() filas[n] = {"IC": ic.mean(), "before": ic[:"2024-03"].mean(), "after": ic["2024-04":].mean(), "2023-12": mes["2023-12"], "2025-02": mes["2025-02"], "deployments": k} pd.DataFrame(filas).T.round(3).astype({"deployments": int}) ``` ```output IC before after 2023-12 2025-02 deployments fixed 0.032 0.094 -0.012 0.079 -0.006 0 always 0.067 0.087 0.053 -0.091 0.021 35 superiority 0.064 0.094 0.042 0.079 0.095 4 gates 0.074 0.098 0.057 0.087 0.094 32 ``` The figure at the top accumulates the daily IC of the four. Before the change they move together (from 0.087 to 0.098 of mean IC). Afterwards, the fixed model drops to −0.012: it keeps betting on `f1`. The other three recover, at different speeds: - **Always** reacts before anyone else, because every month it deploys the freshest model, but it also deploys the broken ones: −0.091 of IC in December 2023 (the inverted model) and 0.021 in February 2025 (the leaky one, which live does not have the information it learned from and bets on noise). - **Superiority** deploys no broken model and only switches models four times in three years, but it is slow to adapt: after the change, 0.042. - **Gates** has the best IC before (0.098) and after (0.057) the change: the freshness of “always” without its two bad months. The difference between “always” and “gates” comes almost entirely from those two months: the same retraining, with and without checking the result. The September 2023 fault, on the other hand, barely costs anything: one month of frozen `f1` in a 252-session window only dampens one coefficient. That a fault does not cost much in this lab does not mean it will not cost anything in another; the gate is there so you do not have to find out. Why is the classic policy so slow? The table of the cycles around the change tells the story: ```python ic, reg, _ = res["superiority"] tabla = pd.DataFrame(reg, columns=["cycle", "gate", "detail"]) tabla["cycle"] = tabla["cycle"].dt.date.astype(str) tabla[tabla["cycle"].between("2024-03-01", "2025-01-31")] ``` ```output cycle gate detail 13 2024-03-01 no improvement dIC +0.009, t +1.5 14 2024-04-01 no improvement dIC +0.001, t +0.1 15 2024-05-01 no improvement dIC +0.001, t +0.1 16 2024-06-03 no improvement dIC +0.001, t +0.1 17 2024-07-01 no improvement dIC +0.005, t +0.5 18 2024-08-01 no improvement dIC +0.014, t +0.9 19 2024-09-02 no improvement dIC +0.018, t +1.2 20 2024-10-01 no improvement dIC +0.031, t +1.8 21 2024-11-01 ok dIC +0.052, t +3.3 22 2024-12-02 ok dIC +0.013, t +3.2 23 2025-01-01 ok dIC +0.021, t +2.5 ``` In April, May and June 2024 the candidate's training window is still almost entirely from the old regime (the validation takes up the last two months), so candidate and champion predict almost the same (0.001 of IC difference). From August on, the candidate starts to see the new regime: it improves by 0.014 (t = 0.9), by 0.018 (t = 1.2), by 0.031 (t = 1.8). It is already better, but with 42 validation sessions it cannot prove it until November: 0.052 with t = 3.3. Seven months after the change. *Fig. — t statistic of the challenger's daily IC against the champion in each monthly cycle's validation, under the superiority rule (the lesson's synthetic data) (schematic).* Monthly bars from 2023 to 2025. Until October 2024 all stay below the threshold of 2, also after the April 2024 regime change: 0.1 in April, 0.9 in August, 1.8 in October. In November 2024 the highlighted bar reaches 3.3 and the challenger is promoted; December 2024, January and August 2025 also clear the threshold. Three crosses mark the cycles rejected for data, behaviour and leakage. The delay has three components, and it pays to know which one dominates before trying to reduce it: | Component | In the lab | How to shorten it | What it costs | |---|---|---|---| | Data from the new regime in the window | several months | shorter or weighted window (lesson 5.4.3) | more variance in the coefficients | | Reserved validation | 42 sessions | shorter validation | less power: more t values below 2 | | Evidence required (t > 2) | one or two cycles | lower threshold | promoting noise | That is why superiority makes sense when the challenger is **a different recipe**: another algorithm, other features, another target. There the question “is it better?” is the right one and getting it wrong is costly, because a different model can fail in ways the champion does not. When the challenger is the same recipe with one more month of data, what was validated was the recipe (in the walk-forward of lesson 5.4.3, with its Deflated Sharpe from 5.4.4), and what needs checking every month is that this month's model is not broken. That is what the gates policy does. ## The shadow test: the challenger with the real inputs A model that passes the gates has still never run with the production pipeline: the cycle's snapshot is a copy, and live the features arrive from another process, at another time and sometimes in another format. The **shadow** test (lesson 4.5.3, applied to the model instead of to orders) runs the challenger in parallel with the champion for a few days, with the same inputs, and stores its predictions without trading on them. What the shadow test checks is not whether it wins. With 50 assets, the standard deviation of a single day's IC is about 0.14 ($1/\sqrt{50}$); over five sessions, the mean IC has an error of about 0.06, larger than any plausible improvement. What it checks is that the model **behaves**: ```python c = pd.Timestamp("2024-11-01") i = fechas.get_loc(c) Fx, Rx = foto(c) (e0, e1), _ = ventanas(c) challenger = ajustar(Fx, Rx, e0, e1) champ_vigente = champion_0 # superiority: still the initial one def comprobar_sombra(champ, chal, a, b, max_rotacion=0.6): """Operational checks, not statistical ones: the IC of five sessions is noise.""" pc, pp = predecir(chal, F, a, b), predecir(champ, F, a, b) top = lambda p: set(np.argsort(p)[-10:]) # top 10 rot = np.mean([1 - len(top(pc[j]) & top(pc[j - 1])) / 10 for j in range(1, b - a)]) return {"finite": bool(np.isfinite(pc).all()), "corr_champion": round(float(np.corrcoef( pc.ravel(), pp.ravel())[0, 1]), 2), "rel_dispersion": round(float(pc.std() / pp.std()), 2), "top10_turnover": round(float(rot), 2), "ok": bool(np.isfinite(pc).all() and rot < max_rotacion)} comprobar_sombra(champ_vigente, challenger, i, i + 5) ``` ```output {'finite': True, 'corr_champion': 0.77, 'rel_dispersion': 0.87, 'top10_turnover': 0.28, 'ok': True} ``` The November 2024 challenger gives finite predictions, with a correlation of 0.77 with the champion (they are similar, but not the same model: it has already learned something about the new regime), 13% lower dispersion and a turnover of the top ten assets of 28% per session. These are numbers to compare with the champion's at the time and with a fixed maximum: a model that changed half the portfolio every day would generate transaction costs that the validation, without costs, does not see. ## The registry: one run per cycle and aliases What remains of each cycle, in the MLflow SQLite store of the `ops` environment (lesson 7.3.1): a **run** with the parameters, the gate that stopped the candidate or `ok`, and the SHA-256 fingerprint of the data snapshot it was trained on. With the fingerprint you can check years later that a cycle reproduces: same snapshot, same model. ```python import hashlib import mlflow from mlflow import MlflowClient def huella(Fx, Rx, a, b): """SHA-256 of the data snapshot used for training.""" h = hashlib.sha256(Fx[a:b].tobytes()) h.update(Rx[a:b].tobytes()) return h.hexdigest()[:12] mlflow.set_experiment("senal-n7") for c, g, det in reg: Fx, Rx = foto(c) (e0, e1), _ = ventanas(c) with mlflow.start_run(run_name=f"ciclo-{c.date()}"): mlflow.log_params({"ventana": VENT, "validacion": VAL, "alpha": 1.0, "ciclo": str(c.date())}) mlflow.set_tags({"puerta": g, "detalle": det, "datos_sha256": huella(Fx, Rx, e0, e1)}) runs = mlflow.search_runs(filter_string="tags.puerta != 'no improvement'", order_by=["params.ciclo"]) print(len(mlflow.search_runs()), "runs") runs[["params.ciclo", "tags.puerta", "tags.datos_sha256"]] ``` ```output 35 runs params.ciclo tags.puerta tags.datos_sha256 0 2023-09-01 data 86b09d034105 1 2023-12-01 behavior 7805683890e5 2 2024-11-01 ok e3a71b5c0551 3 2024-12-02 ok 9e01f0587dad 4 2025-01-01 ok eff27095c2e9 5 2025-02-03 leakage 91fbc679f221 6 2025-08-01 ok 132ff59aad40 ``` Thirty-five runs, one per cycle; the query filters out the “no improvement” ones and leaves in view what matters in an audit: the three rejections with their reason and the four cycles that passed. Models are only registered when they are going to be deployed: registering 35 versions that nobody will use fills the registry with noise. The tag and parameter keys (`puerta`, gate; `ciclo`, cycle) are the registry's identifiers and stay as they are. Going to production is an **alias**. In the MLflow registry, an alias is a name that points to a specific version of a registered model (`champion` to 1, `challenger` to 2), and the live process loads `models:/senal-n7@champion` without knowing which version it is. The fixed stages MLflow used before (`Staging`, `Production`) have been deprecated since version 2.9.0, and the documentation proposes exactly this use of aliases[^1]: ```python NOMBRE = "senal-n7" ejemplo = F[i].astype("float64")[:2] versiones = {} for etiqueta, modelo in (("inicial", champion_0), ("2024-11-01", challenger)): with mlflow.start_run(run_name=f"modelo-{etiqueta}"): info = mlflow.sklearn.log_model(modelo, name="modelo", input_example=ejemplo) versiones[etiqueta] = mlflow.register_model(info.model_uri, NOMBRE).version cli = MlflowClient() cli.set_registered_model_alias(NOMBRE, "champion", versiones["inicial"]) cli.set_registered_model_alias(NOMBRE, "challenger", versiones["2024-11-01"]) def promover(cli, nombre): """The challenger becomes champion; the champion, 'anterior'.""" antes = cli.get_model_version_by_alias(nombre, "champion") nuevo = cli.get_model_version_by_alias(nombre, "challenger") cli.set_registered_model_alias(nombre, "anterior", antes.version) cli.set_registered_model_alias(nombre, "champion", nuevo.version) cli.delete_registered_model_alias(nombre, "challenger") def marcha_atras(cli, nombre): antes = cli.get_model_version_by_alias(nombre, "anterior") cli.set_registered_model_alias(nombre, "champion", antes.version) def alias(cli, nombre): al = cli.get_registered_model(nombre).aliases return dict(sorted(al.items())) print("before:", alias(cli, NOMBRE)) promover(cli, NOMBRE) print("promoted:", alias(cli, NOMBRE)) vivo = mlflow.sklearn.load_model(f"models:/{NOMBRE}@champion") print("the live process loads the challenger:", np.allclose(vivo.predict(F[i]), challenger.predict(F[i]))) marcha_atras(cli, NOMBRE) print("rollback:", alias(cli, NOMBRE)) ``` ```output Successfully registered model 'senal-n7'. Created version '1' of model 'senal-n7'. Registered model 'senal-n7' already exists. Creating a new version of this model... Created version '2' of model 'senal-n7'. before: {'challenger': 2, 'champion': 1} promoted: {'anterior': 1, 'champion': 2} the live process loads the challenger: True rollback: {'anterior': 1, 'champion': 1} ``` Promoting means moving two aliases: `champion` moves to version 2 and version 1 stays as `anterior` (previous). The live process, which loaded by alias, receives the challenger without changing a line of code. Rolling back means moving `champion` to `anterior`: one second, without retraining or deploying anything. That is the property that justifies the whole registry. If the new model misbehaves on its first day, the decision to undo cannot wait for someone to find the previous model's file. The full lifecycle of a version: ```mermaid stateDiagram-v2 [*] --> CANDIDATE CANDIDATE --> REJECTED: gate fails CANDIDATE --> CHALLENGER: gates ok CHALLENGER --> CHAMPION: shadow ok CHALLENGER --> REJECTED: shadow fails CHAMPION --> PREVIOUS: promotion PREVIOUS --> CHAMPION: rollback PREVIOUS --> ARCHIVED ``` ## What gets automated and what does not The lesson's cycle decides on its own, and that is what you want for repetitive work. Some decisions should not be automated: - **The first deployment of a new recipe** goes through full validation and through a person. The gates check that this month's model is not broken; they do not decide whether the recipe is good. - **Rejection alerts** reach someone. A rejection leaves the champion trading on older data; three rejections in a row are a system that has stopped learning, and that is also a failure. - **Gate thresholds** only change through a review and stay in the repository history. - **Shutting down the model**, if its live performance collapses between two cycles, cannot wait for the next cycle: that is lesson 7.3.3. ## Exercises 1. Repeat the superiority policy with a 126-session training window instead of 252. When is the first challenger promoted after the change? How many times are models switched in total? **Solution.** ```python VENT = 126 ic126, reg126, k126 = politica("superiority") VENT = 252 print("promotions:", [str(c.date()) for c, g, _ in reg126 if g == "ok"]) print(f"mean IC {ic126.mean():.3f} · after the change " f"{ic126['2024-04':].mean():.3f}") desde_oct = [g for c, g, _ in reg126 if c >= pd.Timestamp("2024-10")] print("gates since October 2024:", pd.Series(desde_oct).value_counts().to_dict()) ``` ```output promotions: ['2024-03-01'] mean IC 0.034 · after the change -0.009 gates since October 2024: {'behavior': 14, 'leakage': 1} ``` With 126 sessions, the only challenger that gets promoted is the March 2024 one, **before** the change, and afterwards the mean IC is −0.009: worse than with 252. The short window learns the new regime sooner (in August 2024 the candidate already improves the IC by 0.033, with t = 1.8), but cannot prove it, and from October on the candidate resembles the March champion so little (correlation of 0.11, then negative) that the behavior gate rejects it in every cycle: 14 behavior rejections up to December 2025, and one for the February leakage. It is a deadlock: the champion cannot be replaced because it is so outdated that any current model looks broken next to it. Gates interact. The behavior gate asks “does it look like what we had?”, and the answer stops being useful when what we had is old. Two fixes: compare behavior with the previous cycle's candidate (same recipe, one month apart), not with the champion; and treat several consecutive rejections by the same gate as an alert for a person, not as a decision that repeats itself every month. 2. A fourth fault: the target is computed with a one-day lag (the return from tomorrow to the day after tomorrow instead of from today to tomorrow). Write its snapshot and run it through the gates in the May 2025 cycle. Does any gate stop it? **Solution.** ```python def foto_retrasada(ciclo): Fx, _ = foto(ciclo) return Fx, np.roll(R, -1, axis=0) # return from t+1 to t+2 c = pd.Timestamp("2025-05-01") _, (_, v1_previo) = ventanas(pd.Timestamp("2025-04-01")) champ = ajustar(F, R, v1_previo - VENT, v1_previo) # April's model Fx, Rx = foto_retrasada(c) (e0, e1), _ = ventanas(c) cand = ajustar(Fx, Rx, e0, e1) for mejora in (False, True): print(*puertas(c, Fx, Rx, cand, champ, mejora)) print("coefficients x1000:", (cand.coef_ * 1e3).round(2), (champ.coef_ * 1e3).round(2)) ``` ```output ok corr. with champion +1.00 no improvement dIC -0.003, t -1.1 coefficients x1000: [-0.52 0.27 0.54 0.31 -0.07] [-0.66 0.36 0.6 0.32 -0.08] ``` No gate stops it. The features are fine and none explains the same day's target; the candidate has a correlation of 1.00 with the champion, and with the superiority rule it comes out as “no improvement” (−0.003 of IC). Since the features persist from one day to the next (coefficient 0.9), they predict tomorrow's return almost as well as the return of the day after: the model learns the same coefficients, somewhat smaller, and live it barely shows. The gates policy would have deployed it. It is a fault of the **target**, not of the model, and it is tested in the pipeline that computes it: a unit test that recomputes the return of a handful of assets and dates from the prices and compares it with the column (lesson 7.1.1), and a check that the date of each target is later than that of its features. The cycle's gates are a safety net, not the only one: an error that does not change the model cannot be seen by looking at the model. 3. A colleague proposes speeding up the superiority policy by comparing the challenger with the champion on the validation sessions and, if it wins, deploying it refitted on all the data. What gets validated and what gets deployed in that case? **Solution.** One model is validated and another is deployed. The refitted model has not been compared with anything; what was compared was the candidate without the last 42 sessions. In practice they tend to be similar, which is why the proposal is reasonable if it is stated clearly: what is being validated is the **recipe** (“training on the last 252 sessions gives a model better than the champion”), not the file that gets deployed. For that to hold, the refitted model has to be checked with the first three gates (which do not depend on the validation) and run through the shadow test. The strict alternative, deploying exactly the validated model, is the lesson's: it gives up 42 sessions of freshness in exchange for the file in production being the one that passed the test. [^1]: MLflow, “Model Registry Workflows” (https://mlflow.org/docs/latest/ml/model-registry/workflow/): “As of MLflow 2.9.0, Model Stages have been deprecated and will be removed in a future major release”; the documentation's example creates a `champion` alias with `set_registered_model_alias` and loads the model with `models:/@champion`. Accessed on October 23, 2026. ## Self-assessment 1. The December 1, 2023 cycle trains with a target computed as today's price divided by tomorrow's price minus 1. Which gate stops it and why? - A. The data gate, because there are repeated values - B. The leakage gate, because a feature explains the same day's target - C. The behavior gate: the candidate predicts almost the opposite of the champion with the same inputs (correlation −0.95) - D. None: the model is deployed and it shows up live *Answer C: The behavior gate: the candidate predicts almost the opposite of the champion with the same inputs (correlation −0.95).* With the sign of the target flipped, Ridge learns the coefficients with the opposite sign. The features are fine and do not explain the day's target, so the first two gates pass; the third compares the predictions of the candidate and of the champion on the same inputs and finds a correlation of −0.95. 2. With the superiority rule, the regime changes on April 1, 2024 and the first challenger is promoted on November 1. Where does most of the delay come from? - A. From the 252-session window taking months to contain enough data from the new regime and from the 42-session validation needing a large difference to reach t > 2 - B. From MLflow taking a long time to register the model - C. From the five-session shadow test - D. From the champion being retrained every month *Answer A: From the 252-session window taking months to contain enough data from the new regime and from the 42-session validation needing a large difference to reach t > 2.* In August the candidate already improves the IC by 0.014, but with t = 0.9; in October, 0.031 with t = 1.8. The evidence grows as the window fills with the new regime, and it does not cross the threshold until November (0.052, t = 3.3). The shadow test adds days, not months. 3. Why does the “gates” policy not compare the refitted model with the champion on the validation set? - A. Because it is slower - B. Because MLflow does not allow it - C. Because the new model is always better - D. Because the champion, refitted the previous month with the same recipe, was trained on almost all of those sessions: the comparison would favor the one that has seen them *Answer D: Because the champion, refitted the previous month with the same recipe, was trained on almost all of those sessions: the comparison would favor the one that has seen them.* A fair comparison requires sessions that are out of sample for both models. When the challenger is the same recipe with one more month of data, what gets validated is the recipe (in the walk-forward of lesson 5.4.3) and what gets checked every month is that the new model is not broken. 4. The policy that retrains and deploys every month without gates has a mean IC of 0.067; the gates policy, 0.074. What explains almost all of the difference? - A. That the gates policy uses more data - B. Two months with broken models: −0.091 in December 2023 and 0.021 in February 2025, against 0.087 and 0.094 with gates - C. That the gates policy requires superiority - D. The generator's randomness *Answer B: Two months with broken models: −0.091 in December 2023 and 0.021 in February 2025, against 0.087 and 0.094 with gates.* Both policies retrain with the same recipe and the same window; they differ only in that one deploys whatever comes out and the other checks it first. The August 2023 fault barely costs anything, but the target with the flipped sign and the leaky feature cost a whole month each. 5. What does the lesson's five-session shadow test check? - A. That the challenger's IC is higher than the champion's - B. That the challenger has a positive Sharpe - C. That the challenger works with the system's real inputs: finite predictions, similarity to the champion, reasonable dispersion and portfolio turnover - D. That the model is properly registered in MLflow *Answer C: That the challenger works with the system's real inputs: finite predictions, similarity to the champion, reasonable dispersion and portfolio turnover.* Five sessions of IC are noise (the standard deviation of a single day's IC with 50 assets is about 0.14). The shadow test is not a second statistical validation: it checks that the new model, fed by the production pipeline, does nothing strange before it moves money. ## Key points - A retraining cycle is a scheduled pipeline: snapshot of the day's data, training with closed targets, automatic gates, registration, shadow and promotion. Every step leaves a trace. - The data, leakage and behavior gates stop the lesson's three faults: a frozen feature (35% repeated values), a target with the flipped sign (correlation −0.95 with the champion) and a feature built from the target (correlation 0.70). - Requiring the challenger to beat the champion with t > 2 avoids deploying noise, but in the lab it takes seven months to react to a regime change. - When the challenger is the same recipe with more data, passing the gates is enough: mean IC of 0.074 against 0.067 without gates, 0.064 with superiority and 0.032 without retraining. Superiority is kept for changing the recipe. - The shadow test checks the model with the real pipeline (finite predictions, similarity, turnover), not its return: five sessions of IC are noise. - In the registry, champion and challenger are aliases that point to versions: promoting and rolling back means moving an alias, without touching the code of the live process. ## Resources ### Research - [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min ### Documentation - [MLflow: Model Registry Workflows (aliases, tags and the end of stages)](https://mlflow.org/docs/latest/ml/model-registry/workflow/) · mlflow.org - [Google: Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml) · developers.google.com ### Books - [Sculley, D. et al. (2015): Hidden Technical Debt in Machine Learning Systems. NeurIPS](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems) · papers.nips.cc - [Breck, E. et al. (2017): The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction](https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/) · research.google - [Newey, W. K. and West, K. D. (1987): A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica 55(3)](https://doi.org/10.2307/1913610) · doi.org Previous lesson: [7.3.1 · Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end](https://pythonparatrading.com/en/python-trading-course/level-7/experiment-model-data-tracking-mlflow-dvc/) Next lesson: [7.3.3 · Live model monitoring and automatic shutdown](https://pythonparatrading.com/en/python-trading-course/level-7/live-model-monitoring-automatic-shutdown/)