---
title: "7.3.3 Live model monitoring and automatic shutdown"
description: "A monitor that compares live IC with the expected IC, separates data failures from loss of edge and disables the model with a permit in Redis."
url: "https://pythonparatrading.com/en/python-trading-course/level-7/live-model-monitoring-automatic-shutdown/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/live-model-monitoring-automatic-shutdown.png"
---
# 7.3.3 · Live model monitoring and automatic shutdown
> A model does not warn you when it stops working: it keeps producing predictions with the same confidence. This lesson builds the monitor of a model in production over a year of data in which the edge disappears on June 16: it checks the inputs every morning, compares the IC with what the validation promised, reduces and shuts down with a graduated policy, and passes the decision to the executor with a permit that expires if the monitor goes silent.
- Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/)
- Module: 7.3 · MLOps for trading models
- Duration: 70 min
- Prerequisites: [4.2.5 · Drawdown control and kill switch](https://pythonparatrading.com/en/python-trading-course/level-4/drawdown-control-and-kill-switch/), [5.10.2 · Data drift and concept drift: detecting them and what to do](https://pythonparatrading.com/en/python-trading-course/level-5/data-drift-concept-drift/), [7.2.3 · Orchestration and monitoring: task graphs, structured logs, metrics, alerts and heartbeats](https://pythonparatrading.com/en/python-trading-course/level-7/orchestration-and-monitoring-trading-systems/)
## By the end of this lesson
- Order what is monitored in a production model by when it becomes known: inputs before trading, outcomes with the label delay.
- Separate pipeline failures (pause for the day, automatic) from loss of edge (reduce and shut down, with manual re-arming).
- Compare live IC with the expected IC using a z in two windows, and calibrate the thresholds of a graduated policy by their false alarm rate and their delay.
- Pass the decision to the executor with a permit in Redis that expires and that, when in any doubt, is worth zero.
- Publish the model's metrics for Prometheus and record every state change in an audit log.
## A model does not warn you when it dies
The pieces are already in earlier lessons. Lesson 4.2.5 taught a strategy's kill switch: a CUSUM on its returns and
an operational switch with manual re-arming. Lesson 5.10.2 measured data drift (PSI, KS, adversarial validation) and
concept drift (rolling mean of the IC, CUSUM, Page-Hinkley) in a lab. Lesson 7.2.3 set up a system's metrics, alerts
and heartbeats. This lesson brings them together around a **model in production**: what is checked every morning
before letting it trade, what its performance is compared with, who decides to reduce it or shut it down and how that
decision reaches the process that sends the orders.
```mermaid
flowchart TB
M[Live model] --> O[Monitor]
E[Delayed IC] --> O
O --> K[Permit in Redis]
O --> G[Metrics]
O --> A[Audit]
K --> X[Executor]
```
The monitor is a separate process, like the risk engine in lesson 6.4.5: it does not train or predict, it only
watches and decides. Its only output towards trading is a **permit**: the size multiplier with which the executor can
trade the model today.
What can be monitored in a model is ordered by **when it becomes known**:
| Layer | Examples | When it is known | What it decides |
|---|---|---|---|
| Process | heartbeat, latency, errors | immediately | system alerts (7.2.3) |
| Inputs | coverage, frozen values, ranges | before trading | pause for the day |
| Drift | PSI and KS of features and predictions (5.10.2) | before trading | warning and diagnosis |
| Outcome | IC, hit rate, calibration | when the label closes | reduce or shut down |
| Business | P&L, slippage (4.5.3) | at the close | review |
The outcome row is the one that matters and the one that arrives late. A model that predicts tomorrow's return can be
evaluated the day after tomorrow; one that predicts the return twenty sessions ahead, in a month. The layers above do
not tell you whether the model makes money, but they arrive in time to stop it from trading with broken data.
## The lab: a year and a half with the answer known
Fifty assets and a model whose predictions are correlated with what the market pays. Until **June 16, 2025** the
next day's return depends on that component; from that day on, it does not. It is the death of the edge from lesson
4.2.5, applied to the model instead of the strategy. On top of that, three pipeline incidents: two days with frozen
features in March and one day in April with 35% of the predictions missing. The period from July to December 2024
serves as a reference: the model was already in production and its live IC is known.
```python
import numpy as np
import pandas as pd
rng = np.random.default_rng(702)
fechas = pd.bdate_range("2024-07-01", "2025-12-31")
T, N = len(fechas), 50
MUERTE = pd.Timestamp("2025-06-16") # the edge disappears
S = rng.standard_normal((T, N)) # what the market pays
P = S + rng.standard_normal((T, N)) # model prediction
alfa = np.where(fechas < MUERTE, 0.12, 0.0)
Z = alfa[:, None] * S + rng.standard_normal((T, N))
R = 0.01 * Z # return from t to t+1
INCIDENTES = { # pipeline failures
"2025-03-11": "frozen features",
"2025-03-12": "frozen features",
"2025-04-22": "35% of predictions missing",
}
i = fechas.get_loc(pd.Timestamp("2025-03-10"))
P[i + 1] = P[i + 2] = P[i] # the vendor does not update
j = fechas.get_loc(pd.Timestamp("2025-04-22"))
P[j, rng.choice(N, 17, replace=False)] = np.nan
def ic_dia(p, r):
"""Spearman IC of one day, only with valid predictions."""
m = np.isfinite(p)
a = p[m].argsort().argsort() - (m.sum() - 1) / 2
b = r[m].argsort().argsort() - (m.sum() - 1) / 2
return (a * b).sum() / np.sqrt((a * a).sum() * (b * b).sum())
ic = pd.Series([ic_dia(p, r) for p, r in zip(P, R)], index=fechas)
antes = ic["2025":MUERTE].iloc[:-1]
print("mean IC: 2nd half of 2024",
round(ic[:"2024-12"].mean(), 3), "| 2025 until the death",
round(antes.mean(), 3), "| after", round(ic[MUERTE:].mean(), 3))
print("standard deviation of the daily IC:",
round(ic[:"2024-12"].std(), 3))
```
```output
mean IC: 2nd half of 2024 0.084 | 2025 until the death 0.071 | after -0.028
standard deviation of the daily IC: 0.128
```
The mean IC is 0.084 in the second half of 2024, 0.071 in 2025 until the death and −0.028 afterwards. The standard
deviation of the daily IC, 0.128, is that of any model with 50 assets ($\approx 1/\sqrt{50}$): a single day says
almost nothing.
The monitor's first rule is not to use what is not known yet. Before the open on March 12, today's predictions are
already there, but the last known IC is the one for the 10th: the return from the 11th to the 12th has not closed yet.
```python
def lo_que_se_sabe(dia):
"""Before the open of `dia`: today's predictions, and the ICs
up to the day before yesterday (its return closed yesterday)."""
k = fechas.get_loc(dia)
return P[k], ic.iloc[:k - 1]
hoy = pd.Timestamp("2025-03-12")
pred, ics = lo_que_se_sabe(hoy)
print("today:", hoy.date(), "| last known IC:", ics.index[-1].date())
print("predictions equal to yesterday's:",
f"{np.mean(pred == P[fechas.get_loc(hoy) - 1]):.0%}")
```
```output
today: 2025-03-12 | last known IC: 2025-03-10
predictions equal to yesterday's: 100%
```
## Pause: when the pipeline is what fails
The input checks are run every morning on the day's predictions, before they reach the executor. They are cheap, they
do not need to wait for any outcome, and they isolate a type of failure that must not be confused with a loss of
edge:
```python
def operativas(k, min_cobertura=0.9, max_repetidas=0.5):
"""List of problems with the predictions of day k (empty if
there are none)."""
p, ayer = P[k], P[k - 1]
problemas = []
cobertura = np.isfinite(p).mean()
if cobertura < min_cobertura:
problemas.append(f"coverage {cobertura:.0%}")
repetidas = np.mean(p == ayer)
if repetidas > max_repetidas:
problemas.append(f"repeated {repetidas:.0%}")
return problemas
inicio = fechas.get_loc(pd.Timestamp("2025-01-02"))
for k in range(inicio, T):
if p := operativas(k):
print(fechas[k].date(), p)
```
```output
2025-03-11 ['repeated 100%']
2025-03-12 ['repeated 100%']
2025-04-22 ['coverage 66%']
```
Three hits, exactly the three planted incidents: on March 11 and 12 all the predictions are equal to those of the day
before (the vendor did not update the features and the model, being deterministic, repeats its output), and on April
22 only 66% of the assets have a prediction. No other day in 2025 triggers anything.
The response is a **pause**: that day the model does not trade, and the next day, if the checks pass, it comes back on
its own. It is not the shutdown of the model. A healthy model with frozen data has not lost its edge; shutting it down
and waiting for someone to re-arm it punishes the model for a pipeline failure, and the pipeline is what has to be
fixed (with its own alert). The reverse is also a mistake: treating a loss of edge as a passing failure that recovers
on its own.
There are more checks of this kind: ranges of each feature, that the distribution of the predictions does not change
shape (the PSI from lesson 5.10.2 applied to the model's output), that the loaded model version is the one behind the
`champion` alias (lesson 7.3.2). They all share the property that makes them useful: they are known before trading.
## Expected versus observed
To judge the live IC you need to know what was expected. That number comes from the model's validation (the IC of
the walk-forward or of the challenger in lesson 7.3.2), **discounted**: the IC of a validation is that of the best of
several attempts, and live it is usually lower (lesson 5.4.4). The lesson uses 0.06. The standard deviation of the
daily IC is measured live, over the reference period.
With both, the comparison is a z of the mean IC over the last $n$ sessions:
$$
z_n = \frac{\overline{IC}_n - \mu_0}{\sigma / \sqrt{n}}
$$
With $\mu_0$ the expected IC and $\sigma$ the standard deviation of the daily IC. A model that performs as expected has
a $z$ around zero; a dead one, with a mean IC of zero, drifts to $-\mu_0\sqrt{n}/\sigma$: −2.1 with 21 sessions and
−3.7 with 63. That is the difference with lesson 5.10.2, which watched whether the IC **changed**: here the question
is whether the model delivers **what was promised** when it was deployed.
```python
MU0 = 0.06 # expected IC: the validation one, discounted
SIGMA = ic[:"2024-12"].std() # of the daily IC, measured live
def z_ic(ics, n):
"""z of the mean of the last n ICs against the expected IC."""
x = ics.iloc[-n:]
return (x.mean() - MU0) / (SIGMA / np.sqrt(len(x)))
def zs(dia):
_, ics = lo_que_se_sabe(dia)
return tuple(round(float(z_ic(ics, n)), 2) for n in (21, 63))
for d in ("2025-03-03", "2025-05-02", "2025-07-01", "2025-09-01"):
print(d, "z21, z63 =", zs(pd.Timestamp(d)))
```
```output
2025-03-03 z21, z63 = (0.53, 1.19)
2025-05-02 z21, z63 = (2.23, 0.73)
2025-07-01 z21, z63 = (-1.07, 1.07)
2025-09-01 z21, z63 = (-1.83, -3.49)
```
In March and May the model is doing well (on May 2, even better than expected in the short window: +2.23). On July 1,
two weeks after the death, the long window still reads +1.07: it is full of good days. On September 1 both are below
−1.8, and the long one is at −3.49.
## A graduated policy
With two windows and two possible actions (reducing the size and shutting down) you build a small state machine:
```mermaid
stateDiagram-v2
[*] --> ACTIVE
ACTIVE --> WATCH: z21 < -2
WATCH --> ACTIVE: z21 >= -2
ACTIVE --> REDUCED: z63 < -2
WATCH --> REDUCED: z63 < -2
REDUCED --> ACTIVE: z63 > -1
REDUCED --> DISABLED: z63 < -3
DISABLED --> ACTIVE: manual re-arm
```
Four design decisions:
- **The short window only warns.** With 21 sessions the error of the mean IC is 0.028, almost half the expected IC:
a healthy model falls below −2 frequently. It is there to look earlier, not to act.
- **The cheap action needs less evidence than the expensive one.** Halving the size undoes itself and costs little if
it was a false alarm; shutting down requires re-arming and leaves the model out until someone reviews it.
- **Hysteresis.** From REDUCED the model does not return to ACTIVE when it crosses −2 upwards, but when it goes above
−1. Without it, a $z$ hovering around the threshold would make the size change every day, with its costs.
- **Manual re-arming**, as in lesson 4.2.5 and in Article 12 of Delegated Regulation (EU) 2017/589 cited in that
lesson: whatever shuts down a model is something nobody understands yet.
```python
from dataclasses import dataclass, field
MULT = {"ACTIVE": 1.0, "WATCH": 1.0, "REDUCED": 0.5,
"DISABLED": 0.0}
@dataclass
class Politica:
z_vigilar: float = -2.0 # 21-session window: warns only
z_reducir: float = -2.0 # 63-session window: half size
z_apagar: float = -3.0 # 63-session window: to zero
z_volver: float = -1.0 # hysteresis to return to ACTIVE
estado: str = "ACTIVE"
cambios: list = field(default_factory=list)
def decidir(self, dia, z21, z63):
e = self.estado
if e == "DISABLED":
return e # only a person re-arms it
if z63 < self.z_apagar:
e = "DISABLED"
elif z63 < self.z_reducir:
e = "REDUCED"
elif e == "REDUCED" and z63 < self.z_volver:
e = "REDUCED" # no return until z63 > -1
elif z21 < self.z_vigilar:
e = "WATCH"
else:
e = "ACTIVE"
if e != self.estado:
self.cambios.append((dia.date(), self.estado, e,
round(z21, 2) + 0.0,
round(z63, 2) + 0.0))
self.estado = e
return e
def ejecutar_anio(pol):
filas = []
for k in range(inicio, T):
dia = fechas[k]
_, ics = lo_que_se_sabe(dia)
estado = pol.decidir(dia, z_ic(ics, 21), z_ic(ics, 63))
pausa = operativas(k)
filas.append({"fecha": dia, "estado": estado,
"pausa": bool(pausa),
"mult": 0.0 if pausa else MULT[estado]})
return pd.DataFrame(filas).set_index("fecha")
pol = Politica()
diario = ejecutar_anio(pol)
for c in pol.cambios:
print(*c)
print("days paused:", int(diario["pausa"].sum()))
```
```output
2025-04-04 ACTIVE WATCH -2.07 0.0
2025-04-07 WATCH ACTIVE -1.86 -0.15
2025-07-15 ACTIVE WATCH -2.03 -0.26
2025-07-30 WATCH REDUCED -3.28 -2.11
2025-08-06 REDUCED DISABLED -3.46 -3.01
days paused: 3
```
The figure at the top follows the two windows over the whole year. Before the death, a single alarm: the short window
touches −2.07 on Friday, April 4, and by Monday it is back. It is a false alarm and the policy treats it as such,
because the long window is at 0. Afterwards, the short one warns on July 15, the long one crosses −2 on July 30 (32
sessions after the death) and −3 on August 6 (37 sessions). The three paused days are the three incidents.
Is it worth it? The model's portfolio buys the ten assets with the best prediction and sells the ten worst every day,
with a cost of 5 bp per leg:
```python
def cartera(k, n=10):
"""Return of buying the n best and selling the n worst."""
ok = np.where(np.isfinite(P[k]))[0]
orden = ok[np.argsort(P[k, ok])]
return R[k, orden[-n:]].mean() - R[k, orden[:n]].mean()
rc = pd.Series([cartera(k) for k in range(inicio, T)],
index=fechas[inicio:])
con = rc * diario["mult"]
COSTE = 0.0005 # 5 bp per full turnover
for nombre, serie, m in (("no policy", rc, 1.0),
("with policy", con, diario["mult"])):
neto = serie - COSTE * 2 * m # two legs, daily turnover
print(f"{nombre:13} before {neto[:MUERTE].iloc[:-1].sum():+.1%}"
f" · after {neto[MUERTE:].sum():+.1%}")
```
```output
no policy before +13.3% · after -19.0%
with policy before +15.0% · after -4.5%
```
*Fig. — Cumulative net return (sum of daily returns, with 5 bp cost per leg) of the model's long-short portfolio in 2025, with and without the switch-off policy (the lesson's synthetic data) (schematic).* Two lines rising almost together until mid-June 2025, to 13.3% without the policy and 15.0% with it. After the edge dies on 16 June, the line without the policy falls steadily to −5.7% at year end; the line with the policy falls less, is halved on 30 July, flattens when switched off on 6 August and ends at +10.4%.
From the death to the end of the year, the portfolio loses 19.0% without the policy and 4.5% with it (sums of daily
returns). Before the death the policy costs nothing: 15.0% versus 13.3%, because the three pauses avoided trading
with broken data and its costs. A dead strategy does not earn zero: it loses its transaction costs every day, and the
monitor is what puts an end to that.
The −2 and −3 thresholds do not come from a table: they are chosen by looking at what they cost. Two numbers per
threshold, the same ones as in lesson 4.2.5: how many times the policy acts in a year in which the model is healthy
and how long it takes to act when the model dies. Thousands of years are simulated with normal daily ICs of mean
$\mu_0$ and standard deviation $\sigma$, and thousands of deaths:
```python
def simular(mu, dias, n, pol_kw, semilla=0):
"""n paths of normal daily ICs; returns the first day in
REDUCED and in DISABLED (or dias if it never gets there)."""
g = np.random.default_rng(semilla)
x = mu[None, :] + SIGMA * g.standard_normal((n, len(mu)))
c = np.cumsum(np.pad(x, ((0, 0), (1, 0))), axis=1)
m63 = (c[:, 63:] - c[:, :-63]) / 63 # rolling means
z63 = (m63 - MU0) / (SIGMA / np.sqrt(63))
red = np.full(n, dias); apa = np.full(n, dias)
for t in range(z63.shape[1]):
nuevo_r = (z63[:, t] < pol_kw["z_reducir"]) & (red == dias)
red[nuevo_r] = t
nuevo_a = (z63[:, t] < pol_kw["z_apagar"]) & (apa == dias)
apa[nuevo_a] = t
return red, apa
filas = {}
for zr, za in ((-1.5, -2.5), (-2.0, -3.0), (-2.5, -3.5)):
kw = {"z_reducir": zr, "z_apagar": za}
sano = np.full(252 + 63, MU0) # a healthy year
r0, a0 = simular(sano, 252, 4000, kw)
muerto = np.r_[np.full(63, MU0), np.zeros(252)] # dies at 63
r1, a1 = simular(muerto, 252, 4000, kw, semilla=1)
filas[f"{zr} / {za}"] = {
"P_reduce_healthy": (r0 < 252).mean(),
"P_shutdown_healthy": (a0 < 252).mean(),
"reduce_median": np.median(r1),
"shutdown_median": np.median(a1),
"shutdown_p90": np.quantile(a1, 0.9)}
pd.DataFrame(filas).T.round(3)
```
```output
P_reduce_healthy P_shutdown_healthy reduce_median shutdown_median shutdown_p90
-1.5 / -2.5 0.665 0.145 23.0 40.0 61.0
-2.0 / -3.0 0.364 0.041 31.0 48.0 75.0
-2.5 / -3.5 0.145 0.008 40.0 57.0 108.0
```
| Thresholds (reduce / shut down) | What it gains | What it costs |
|---|---|---|
| −1.5 / −2.5 | shuts down after a median of 40 sessions | a healthy year reduces with probability 0.67 and shuts down with 0.15 |
| −2 / −3 | shuts down after a median of 48 sessions | reduces with 0.36 and shuts down with 0.04 |
| −2.5 / −3.5 | a healthy year almost never shuts down (0.008) | shuts down after a median of 57 sessions and 108 at the 90th percentile |
The lesson's policy is the middle one. A healthy year reducing at some point with probability 0.36 seems like a lot,
and it is acceptable because reducing is the cheap action. What has to be rare is shutting down. And there is a limit
that no threshold avoids: with an expected IC of 0.06 and 50 assets, a dead model needs weeks to be distinguished
from a healthy one. To detect sooner you need more information per day (more assets, a higher expected IC) or to
accept more false alarms.
## The permit to trade
The monitor's decision has to reach the executor, which is another process. The simplest and most robust way is a
**permit** in a shared store: the monitor writes it every morning with its state, its multiplier and an expiry time;
the executor reads it before each order. Three properties:
- **Fail-safe.** If there is no permit, if it has expired or if the store does not respond, the multiplier is zero.
- **Expiry.** The permit is valid for 26 hours. If the monitor goes down, nobody needs to notice: the next day the
permit has expired and the model stops trading. It is the version of the heartbeat from lesson 7.2.3 that acts
instead of warning.
- **One direction only.** The monitor writes and the executor reads. The executor cannot renew or change the permit.
```python
import json
import fakeredis
import redis
servidor = fakeredis.FakeServer()
kv = fakeredis.FakeRedis(server=servidor, decode_responses=True)
CLAVE = "permiso:senal-n7"
VALIDEZ = 26 * 3600 # the monitor renews it every morning
def publicar(kv, ahora, estado, mult, motivo=""):
"""The monitor writes the day's permit (the only thing it does)."""
kv.hset(CLAVE, mapping={"estado": estado, "mult": mult,
"valido_hasta": ahora + VALIDEZ,
"motivo": motivo})
def multiplicador(kv, ahora):
"""The executor asks before each order. When in doubt, 0."""
try:
p = kv.hgetall(CLAVE)
except redis.exceptions.RedisError:
return 0.0, "Redis not responding"
if not p:
return 0.0, "no permit"
if ahora > float(p["valido_hasta"]):
return 0.0, "permit expired"
return float(p["mult"]), p["estado"]
t0 = 1_750_000_000 # injected clock (seconds)
publicar(kv, t0, "REDUCED", 0.5, "z63 < -2")
print(multiplicador(kv, t0 + 3600))
print(multiplicador(kv, t0 + 30 * 3600)) # the monitor did not renew
servidor.connected = False # Redis goes down
print(multiplicador(kv, t0 + 3600))
servidor.connected = True
```
```output
(0.5, 'REDUCED')
(0.0, 'permit expired')
(0.0, 'Redis not responding')
```
In the cell, Redis is simulated by fakeredis with the same `redis-py` client, and the clock is injected (as in lesson
4.2.5) so that the expiry can be tested without waiting 26 hours. In production, the expiry can be set by Redis itself
(`SET` with `EX`, or `EXPIRE`), so that the key disappears on its own; the executor still treats it the same way,
because “no permit” and “permit expired” lead to the same zero.
Every state change goes into the audit log with the numbers that caused it, in JSON with sorted keys (lesson 7.2.3):
```python
import io
import structlog
buf = io.StringIO() # in production, a JSONL file
structlog.configure(
processors=[structlog.processors.JSONRenderer(sort_keys=True)],
logger_factory=structlog.PrintLoggerFactory(file=buf))
log = structlog.get_logger().bind(modelo="senal-n7", version=3)
for dia, antes, despues, z21, z63 in pol.cambios:
log.warning("cambio_estado", fecha=str(dia), antes=antes,
despues=despues, z21=z21, z63=z63)
print(*buf.getvalue().splitlines()[-2:], sep="\n")
```
```output
{"antes": "WATCH", "despues": "REDUCED", "event": "cambio_estado", "fecha": "2025-07-30", "modelo": "senal-n7", "version": 3, "z21": -3.28, "z63": -2.11}
{"antes": "REDUCED", "despues": "DISABLED", "event": "cambio_estado", "fecha": "2025-08-06", "modelo": "senal-n7", "version": 3, "z21": -3.46, "z63": -3.01}
```
And the model's metrics are published for Prometheus alongside the system's. Lesson 7.2.3 explains how they are
collected and how the rules are written; here it is enough to cover what is specific to the model: the $z$ of each
window, the current multiplier and the **age of the last outcome**, which warns that the monitor has gone blind:
```python
from prometheus_client import (CollectorRegistry, Gauge,
generate_latest)
reg = CollectorRegistry()
g_z = Gauge("modelo_ic_z", "z of the IC against the expected IC",
["modelo", "ventana"], registry=reg)
g_estado = Gauge("modelo_multiplicador", "Permitted size (0 to 1)",
["modelo"], registry=reg)
g_edad = Gauge("modelo_ultimo_ic_dias",
"Sessions since the last known IC",
["modelo"], registry=reg)
dia = pd.Timestamp("2025-07-30")
z21, z63 = zs(dia)
g_z.labels("senal-n7", "21").set(z21)
g_z.labels("senal-n7", "63").set(z63)
g_estado.labels("senal-n7").set(diario.loc[dia, "mult"])
g_edad.labels("senal-n7").set(2)
print("\n".join(l for l in generate_latest(reg).decode().splitlines()
if not l.startswith("#")))
```
```output
modelo_ic_z{modelo="senal-n7",ventana="21"} -3.28
modelo_ic_z{modelo="senal-n7",ventana="63"} -2.11
modelo_multiplicador{modelo="senal-n7"} 0.5
modelo_ultimo_ic_dias{modelo="senal-n7"} 2.0
```
```python
import yaml
REGLAS = """
groups:
- name: modelos
rules:
- alert: ModeloRendimientoBajo
expr: modelo_ic_z{ventana="63"} < -2
labels: {severidad: aviso}
- alert: ModeloDesactivado
expr: modelo_multiplicador == 0
labels: {severidad: critica}
- alert: ModeloSinResultados
expr: modelo_ultimo_ic_dias > 3
labels: {severidad: critica}
"""
for r in yaml.safe_load(REGLAS)["groups"][0]["rules"]:
print(f'{r["alert"]:22} {r["expr"]:34} {r["labels"]["severidad"]}')
```
```output
ModeloRendimientoBajo modelo_ic_z{ventana="63"} < -2 aviso
ModeloDesactivado modelo_multiplicador == 0 critica
ModeloSinResultados modelo_ultimo_ic_dias > 3 critica
```
## After shutting down
A disabled model does not come back because its $z$ goes up. It comes back when someone understands what happened:
1. **Diagnose.** Did the features change (PSI, adversarial validation from lesson 5.10.2) or only their relationship
with the outcome? Was there a change in the pipeline or in the market (a regime change, a new rule, a strategy
that has become crowded)?
2. **Decide what trades in the meantime.** Nothing (multiplier zero), the previous model (the `anterior` alias from
lesson 7.3.2, if its recipe is still valid) or a more conservative version.
3. **Come back through the front door.** A fixed or retrained model enters as a challenger in the cycle of lesson
7.3.2, with its gates and its shadow period. This monitor's manual re-arm is only used when the shutdown was an
error of the monitor itself (for example, a wrongly set expected IC).
## Exercises
1. Repeat the calibration with the −2 / −3 policy assuming that the true IC of the healthy model is 0.05 or 0.04
instead of 0.06 (the monitor still expects 0.06). What happens to the false alarms?
**Solution.**
```python
kw = {"z_reducir": -2.0, "z_apagar": -3.0}
for mu_real in (0.06, 0.05, 0.04):
r, a = simular(np.full(252 + 63, mu_real), 252, 4000, kw)
print(f"true IC {mu_real:.2f}: P(reduce) {(r < 252).mean():.2f}"
f" · P(shut down) {(a < 252).mean():.2f} in a healthy year")
```
```output
true IC 0.06: P(reduce) 0.36 · P(shut down) 0.04 in a healthy year
true IC 0.05: P(reduce) 0.73 · P(shut down) 0.18 in a healthy year
true IC 0.04: P(reduce) 0.95 · P(shut down) 0.51 in a healthy year
```
With a true IC of 0.05, a healthy year reduces at some point with probability 0.73 and shuts down with 0.18; with
0.04, with 0.95 and 0.51: half of the healthy years would end with the model shut down. The policy does not
distinguish a dead model from one that performs below what was promised, because to the policy both are the same
thing: an IC below the expected one.
There are two readings, and both are useful. The first: the expected IC has to be prudent from the start (the
validation one with a clear discount). The second: a model that delivers a third less than what was validated is
also news that someone should review; it may not need to be shut down, but its position sizing and the expected IC
should be recomputed with the first months of live trading, and the decision written down.
2. Add to the policy the rule of the `ModeloSinResultados` alert: if the last known IC is more than five sessions
old, the state is at most REDUCED. Test it with a stoppage of the price pipeline: the last IC that arrives is the
one for February 17.
**Solution.**
```python
def decidir_con_edad(pol, dia, ics, max_edad=5):
"""Age: sessions since the last known IC, today included (2
on a normal day). Above max_edad, the monitor is blind:
REDUCED at most."""
edad = len(fechas[(fechas > ics.index[-1]) & (fechas <= dia)])
estado = pol.decidir(dia, z_ic(ics, 21), z_ic(ics, 63))
if edad > max_edad and estado in ("ACTIVE", "WATCH"):
return "REDUCED", edad
return estado, edad
p2 = Politica()
_, ics = lo_que_se_sabe(pd.Timestamp("2025-02-27"))
ics_parados = ics[:"2025-02-17"] # prices stop arriving
for d in ("2025-02-19", "2025-02-27"):
print(d, decidir_con_edad(p2, pd.Timestamp(d), ics_parados))
```
```output
2025-02-19 ('ACTIVE', 2)
2025-02-27 ('REDUCED', 8)
```
On February 19 the last known IC is still the one for the 17th (age 2, the normal one) and the state is ACTIVE. On
the 27th, with eight sessions without outcomes, the rule takes it to REDUCED even though the $z$, computed with old
ICs, still says everything is fine. If the stoppage continues, a second rule (for example, more than ten sessions)
should take it to zero.
The `ModeloSinResultados` alert fires earlier (with an age greater than 3): first a person is warned, and if the
problem persists, the policy acts on its own. It is the same principle as the permit with an expiry: a control that
runs out of information cannot keep authorizing as if it had it.
3. A colleague proposes that the disabled model re-arm itself when $z_{63}$ goes back above −1. What would you
answer?
**Solution.**
That the $z$ of a shut-down model keeps moving by chance, and a dead model can rise above −1 for a few days: with a
true IC of zero, the 63-session mean has a standard deviation of 0.016 and its $z$ oscillates around −3.7 with a
standard deviation of 1, so crossing −1 is rare but not impossible within a year, and each crossing would switch it
back on only to shut it down weeks later, with losses in between. Besides, the $z$ only measures the IC: it does not
say whether the cause of the shutdown (a broken pipeline, a regime change) has been resolved. And if the model does
not trade, its IC can still be computed in shadow mode, but that is exactly what the 7.3.2 cycle does with a
challenger, with gates and with a person who decides. Automatic return undoes what manual re-arming protects.
## Self-assessment
1. On March 11 and 12 the model's predictions are identical to those of the 10th. What does the lesson's policy do?
- A. It disables the model until a person re-arms it
- B. It pauses those two days (multiplier 0) and goes back to trading on its own as soon as the predictions change
- C. Nothing: the IC has not dropped
- D. It halves the size for a month
*Answer B: It pauses those two days (multiplier 0) and goes back to trading on its own as soon as the predictions change.* Frozen predictions are a pipeline failure, not a model failure: they are visible before trading and are fixed by fixing the pipeline. The pause does not touch the model's statistical state or require re-arming; shutting down the model because of a data failure would leave it idle for no reason.
2. Why does the policy only warn (WATCH) with the 21-session window and leave reducing and shutting down to the 63-session one?
- A. Because the 21-session window cannot be computed until February
- B. Because Prometheus does not support short windows
- C. Because the 63-session window reacts sooner
- D. Because with 21 sessions the error of the mean IC is almost twice as large: a healthy model crosses −2 frequently, as on April 4 in the lesson (−2.07)
*Answer D: Because with 21 sessions the error of the mean IC is almost twice as large: a healthy model crosses −2 frequently, as on April 4 in the lesson (−2.07).* The standard error of the mean is σ/√n: with σ = 0.128, 0.028 over 21 sessions versus 0.016 over 63. The short window is useful to look earlier, not to act: in the lab, the only alarm before the death of the edge comes from it and lasts one weekend.
3. The edge disappears on June 16. When does the lesson's policy act?
- A. It halves the size on July 30 (z63 = −2.11) and shuts down on August 6 (z63 = −3.01)
- B. It shuts down on June 17
- C. It does not act at all in 2025
- D. It shuts down on July 15, when the 21-session window falls below −2
*Answer A: It halves the size on July 30 (z63 = −2.11) and shuts down on August 6 (z63 = −3.01).* The 63-session mean takes time to fill up with days without edge: 37 sessions to shut down, somewhat less than the calibration median of 48. Faster is only possible with thresholds that, in a healthy year, would shut the model down many more times.
4. The executor reads the permit in Redis before each order and Redis does not respond. What does `multiplicador` return?
- A. The last value it read, kept in memory
- B. 1, so as not to miss opportunities
- C. 0: the permit is fail-safe, and without a response there is no trading
- D. It raises an exception and the executor decides
*Answer C: 0: the permit is fail-safe, and without a response there is no trading.* A control that fails open stops controlling precisely when it is needed most. If the monitor does not write (the permit expires) or the store does not respond, the executor cannot know whether the model is still authorized, and the safe choice is not to trade.
5. In the calibration, with thresholds −2 and −3, a healthy year reduces the size at some point with probability 0.36. Is that a problem?
- A. Yes: the threshold must be raised until it never reduces
- B. Not necessarily: reducing is cheap and reversible (it comes back with z63 > −1); what has to be rare is shutting down, which in a healthy year happens with probability 0.04
- C. Yes: it means the model is wrong
- D. No: false alarms cost nothing
*Answer B: Not necessarily: reducing is cheap and reversible (it comes back with z63 > −1); what has to be rare is shutting down, which in a healthy year happens with probability 0.04.* That is what a graduated policy is for: the cheap action is triggered with less evidence and the expensive one with more. A reduce threshold as demanding as the shutdown threshold would lose the advantage of having two steps.
## Key points
- What is monitored in a model is ordered by when it becomes known: the inputs (coverage, frozen values) before trading; the outcomes (IC, hit rate), with the label delay.
- A pipeline failure pauses the day and recovers on its own; a loss of edge reduces and shuts down, and only a person re-arms. Mixing them up shuts down healthy models or lets dead models trade.
- Performance is compared with what the validation promised: z of the mean IC against the expected IC over 21 and 63 sessions. The short window warns; the long one acts.
- In the lab the edge dies on June 16; the monitor reduces on July 30 and shuts down on August 6. After the death, the portfolio loses 19.0% without the policy and 4.5% with it.
- Thresholds are chosen by simulation: with −2 and −3, a healthy year reduces at some point with probability 0.36 and shuts down with 0.04; a dead model is shut down after a median of 48 sessions. If the expected IC is inflated, false alarms soar.
- The decision reaches the executor as a permit with an expiry in Redis: no permit, expired or no response, the multiplier is zero.
## Resources
### Research
- [Finite-State Machines for Quantitative Trading Execution in Python](https://pythonparatrading.com/en/finite-state-machines-trading-python/) · Research-0029 · 47 min
### Documentation
- [Prometheus: Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) · prometheus.io
- [Redis: EXPIRE (keys with a timeout)](https://redis.io/docs/latest/commands/expire/) · redis.io
- [structlog: documentation](https://www.structlog.org/en/stable/) · structlog.org
- [Google SRE Book: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) · sre.google
### Books
- [Breck, E. et al. (2017): The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction](https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/) · research.google
Previous lesson: [7.3.2 · Scheduled retraining, automatic validation and champion/challenger](https://pythonparatrading.com/en/python-trading-course/level-7/model-retraining-validation-champion-challenger/)
Next lesson: [7.3.4 · Model serving: ONNX, CPU or GPU inference and the cost of LLMs in production](https://pythonparatrading.com/en/python-trading-course/level-7/model-serving-onnx-llm-costs/)