---
title: "5.8.3 In practice: zero-shot, covariates and fine-tuning"
description: "Context, quantiles and their calibration, covariates with TimesFM-2.5 and fine-tuning Chronos-Bolt on realized volatility, measured out of sample."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/zero-shot-covariates-fine-tuning-practice/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/zero-shot-covariates-fine-tuning-practice.png"
---
# 5.8.3 · In practice: context, quantiles, covariates and fine-tuning
> Forecasting with a foundation model is a single call, but using it well takes several decisions: how much history to give it, how to read and correct its quantiles, when to add covariates and whether it is worth tuning it on your own data. This lesson takes each decision on the level 5 realized volatility and measures it out of sample, with paired differences. Honest spoiler: almost everything that looks like an obvious improvement here improves nothing.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.8 · Time series foundation models
- Duration: 75 min
- Prerequisites: [5.8.2 · Foundation model families: architecture, data and license](https://pythonparatrading.com/en/python-trading-course/level-5/chronos-timesfm-moirai-kronos-families/), [5.7.3 · Probabilistic models: quantile loss, DeepAR and intervals](https://pythonparatrading.com/en/python-trading-course/level-5/probabilistic-forecasting-quantile-loss-deepar/), [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/)
## By the end of this lesson
- Choose the context length by measuring the out-of-sample error, and know why it depends on the memory of the series.
- Measure the calibration of a foundation model's quantiles, overall and conditional, and recalibrate them with a per-quantile conformal adjustment.
- Turn quantiles of the log into a forecast of the expected variance and evaluate it with QLIKE.
- Know which models accept covariates in the installed versions, and use those of TimesFM-2.5 (XReg) with a placebo control.
- Fine-tune Chronos-Bolt's output head on your own data on CPU, monitor the validation loss and decide with an honest test whether fine-tuning pays off.
## The decisions the call does not make
Lessons 5.8.1 and 5.8.2 used the models almost with their default options. In practice several things have to be
decided, and each one can be measured with an out-of-sample test:
```mermaid
flowchart TB
A[Transform] --> B[Context]
B --> C[Quantiles]
C --> D[Recalibrate]
D --> E[Covariates]
E --> F[Fine-tuning]
F --> G[Honest test]
```
The test bench is still the one from the previous lessons: the log of the realized variance from `vol5`, with a
one-day forecast and a rolling origin over 2018-2025. The transformation is already decided (the log: almost Gaussian
and unbounded). The rest is decided here.
```python
import numpy as np
import pandas as pd
import torch
from chronos import BaseChronosPipeline
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
vol = pd.read_parquet("vol5_diario.parquet")
log_rv, rend = {}, {}
for a in vol["activo"].unique():
d = vol[vol["activo"] == a].set_index("fecha")
log_rv[a] = np.log(d["rv_5min"].to_numpy())
rend[a] = d["rendimiento"].to_numpy()
fechas = pd.DatetimeIndex(d.index)
prueba = np.flatnonzero(fechas.year >= 2018)
bolt = BaseChronosPipeline.from_pretrained(
"amazon/chronos-bolt-small", device_map="cpu",
dtype=torch.float32)
def zero_shot(y, origenes, contexto, lote=256):
"""Tomorrow's 0.1…0.9 quantiles with `contexto` sessions."""
salida = []
for k in range(0, len(origenes), lote):
v = np.stack([y[i - contexto:i] for i in origenes[k:k + lote]])
q, _ = bolt.predict_quantiles(
inputs=torch.tensor(v, dtype=torch.float32),
prediction_length=1)
salida.append(q[:, 0, :].numpy().astype(float))
return np.concatenate(salida)
print(sorted(log_rv), len(prueba), "test sessions")
```
```output
['accion', 'bono', 'cripto', 'divisa', 'indice', 'oro'] 2011 test sessions
```
## How much history to give it
Chronos-Bolt accepts up to 2,048 sessions of context and TimesFM-2.5 up to 16,384. More context gives more information
about the level and dynamics of the series, but it costs time (more patches) and can bring in history that no longer
looks like the present. The cell sweeps from 16 to 1,024 sessions with one origin every 16 (126 days per asset) and
compares each context with the 512 one through the paired difference.
```python
orig = prueba[::16]
contextos = [16, 32, 64, 128, 256, 512, 1024]
ecm, err = {}, {}
for a in ("indice", "cripto"):
real = log_rv[a][orig]
for c in contextos:
e = (real - zero_shot(log_rv[a], orig, c)[:, 4]) ** 2
ecm[(a, c)], err[(a, c)] = e.mean(), e
tabla = pd.Series(ecm).unstack(0).round(3)
tabla.index.name = "context"
print(tabla.to_string())
for a in ("indice", "cripto"):
for c in (16, 64):
d = err[(a, c)] - err[(a, 512)]
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{a}: {c} − 512 sessions {d.mean():+.4f} (t = {t:+.1f})")
```
```output
cripto indice
context
16 0.429 0.240
32 0.412 0.220
64 0.417 0.228
128 0.402 0.223
256 0.393 0.221
512 0.387 0.219
1024 0.390 0.218
indice: 16 − 512 sessions +0.0205 (t = +1.7)
indice: 64 − 512 sessions +0.0092 (t = +1.1)
cripto: 16 − 512 sessions +0.0413 (t = +3.1)
cripto: 64 − 512 sessions +0.0296 (t = +3.1)
```
*Fig. — One-day mean squared error of zero-shot Chronos-Bolt by context length, relative to 512 sessions (126 origins in 2018-2025 per asset).* Two lines on a log scale of context, from 16 to 1,024 sessions. Crypto, highlighted, falls from 1.109 at 16 sessions to 1.0 at 512 and 1.008 at 1,024: more context helps up to about two years. The index goes from 1.096 at 16 sessions to values between 0.995 and 1.041 from 32 onwards, and no difference is significant (16 vs 512: t = 1.7).
The two assets behave differently. On the **index**, the error drops from 0.240 with 16 sessions to 0.220 with 32 and
from there it moves between 0.218 and 0.228 with no clear order; not even 16 versus 512 is significant (t = 1.7). On
**crypto**, the error falls steadily from 0.429 to 0.387 with 512 sessions, and the difference against 16 or 64
sessions has t = 3.1. The reason is noise: crypto RV, with twelve jumps a year on average and a large measurement
error, needs more days for the model to estimate its level; on the index, with no jumps, a month and a half is enough.
With 1,024 sessions neither gains anything. In practice, 256-512 sessions is a reasonable choice for daily volatility,
and the test has to be repeated on each series.
## Quantiles: do they cover what they promise?
Chronos-Bolt returns nine quantiles. A quantile $q_\tau$ is **calibrated** if the realized value falls below it on a
fraction $\tau$ of the days. Lesson 5.7.3 showed how to measure it with trained models; the difference here is that the
model has not seen the series, so its calibration is whatever it brings from pretraining. With one origin every two
sessions (1,006 days per asset) and a context of 256:
```python
C = 256
tau = np.arange(1, 10) / 10
orig = prueba[::2]
zs = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
filas = {}
for a, q in zs.items():
real = log_rv[a][orig]
filas[a] = (real[:, None] <= q).mean(0)
frec = pd.DataFrame(filas, index=tau).round(3)
frec.index.name = "nominal"
print(frec.T.to_string())
```
```output
nominal 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
indice 0.124 0.235 0.332 0.428 0.507 0.600 0.690 0.786 0.901
cripto 0.103 0.206 0.296 0.394 0.480 0.583 0.685 0.796 0.912
```
The header figure plots the difference between the observed and the nominal frequency, with the ±2 standard error
band (with 1,006 days, the standard error of a 50 % frequency is $\sqrt{0.25/1{,}006} \approx 0.016$). On **crypto**
the nine quantiles fall inside or close to the band: calibration is good. On the **index**, the lower tail is too
narrow: 12.4 % of the days fall below the 0.1 quantile and 33.2 % below the 0.3 one, when they should be 10 % and 30 %.
The upper tail is fine (90.1 % at 0.9). Put another way: the model does not expect days as calm as those of the
synthetic index.
Overall coverage can hide problems that only show up when you condition, as lesson 5.7.3 showed. The test is split
into thirds by the forecast median (a value known the day before):
```python
def cobertura(real, q):
return np.mean((real >= q[:, 0]) & (real <= q[:, 8]))
for a, q in zs.items():
real = log_rv[a][orig]
tercio = pd.qcut(q[:, 4], 3, labels=["low", "medium", "high"])
cob = {t: cobertura(real[tercio == t], q[tercio == t])
for t in ("low", "medium", "high")}
print(f"{a:<7} total {cobertura(real, q):.1%} " + " ".join(
f"{t} {v:.1%}" for t, v in cob.items()))
```
```output
indice total 77.6% low 76.2% medium 75.8% high 80.9%
cripto total 80.8% low 80.4% medium 83.3% high 78.8%
```
On the index the 10 to 90 % interval covers 76.2 % and 75.8 % in the low and medium volatility thirds, and 80.9 % in
the high one: it is on normal days that it fails, consistent with the narrow lower tail. On crypto the pattern is the
opposite and milder (83.3 % in the middle third, 78.8 % in the high one). With about 335 days per third, the standard
error of each coverage is about 2.2 points, so only the index deviations in the low and medium thirds are clearly
outside.
## Recalibrating: a per-quantile conformal adjustment
If the quantiles are biased in a stable way, they can be corrected with a calibration period. The simplest version of
**conformalized quantile regression** (Romano, Patterson and Candès, 2019) shifts each quantile by the empirical
quantile of its residuals: with the residuals $e = y - q_\tau$ from the calibration period, the corrected quantile is
$q_\tau + Q_\tau(e)$. It is calibrated on 2018-2020 and measured on 2021-2025, without looking at the test:
```python
def pinball(real, q):
u = real[:, None] - q
return np.maximum(tau * u, (tau - 1) * u).mean()
cal = fechas.year[orig] <= 2020
ajustes = {}
for a, q in zs.items():
real = log_rv[a][orig]
# per-quantile shift: nominal frequency in 2018-2020
res = real[cal, None] - q[cal]
delta = np.array([np.quantile(res[:, j], tau[j])
for j in range(9)])
qa = q + delta
ajustes[a] = delta
c0 = cobertura(real[~cal], q[~cal])
c1 = cobertura(real[~cal], qa[~cal])
print(f"{a:<7} 2021-2025: coverage {c0:.1%} to {c1:.1%}; pinball "
f"{pinball(real[~cal], q[~cal]):.4f} to "
f"{pinball(real[~cal], qa[~cal]):.4f}")
print({a: d.round(3).tolist() for a, d in ajustes.items()})
```
```output
indice 2021-2025: coverage 78.3% to 80.3%; pinball 0.1352 to 0.1351
cripto 2021-2025: coverage 80.9% to 81.2%; pinball 0.1659 to 0.1661
{'indice': [-0.046, -0.039, -0.046, -0.026, 0.024, 0.024, 0.023, 0.0, 0.02], 'cripto': [-0.047, -0.011, -0.003, -0.008, 0.026, 0.008, 0.015, -0.014, -0.062]}
```
On the index, the 2021-2025 coverage rises from 78.3 % to 80.3 %, almost the nominal value; the shifts are a few
hundredths in log (−0.046 at the 0.1 quantile, which lowers the tail). On crypto, which was already calibrated, it goes
from 80.9 % to 81.2 %. The **pinball loss barely changes** (0.1352 to 0.1351 and 0.1659 to 0.1661): the adjustment
corrects the average coverage, it does not add information. It is useful when what you will use is the interval (to
set a risk limit or the size of a position, lesson 4.2.2), not to improve the forecast. And it does not guarantee exact
coverage on time series, because the residuals are not exchangeable: lesson 5.7.3 covers this in detail.
## From quantiles of the log to the variance
For sizing or comparing with a GARCH you need tomorrow's **expected variance**, not the median of its log.
Exponentiating the median gives the median of the variance, which is lower than its mean when the distribution has a
right tail. With nine quantiles there is a direct approximation: exponentiate each quantile and average (a crude
quadrature of the expectation). The measure for comparing variance forecasts is **QLIKE**, the one in the level README
and in lesson 5.8.5: $\text{QLIKE} = RV/h - \log(RV/h) - 1$, robust to the noise of RV as a measure of the variance.
```python
def qlike(rv, h):
return np.mean(rv / h - np.log(rv / h) - 1)
for a, q in zs.items():
rv = np.exp(log_rv[a][orig])
mediana = np.exp(q[:, 4])
media_q = np.exp(q).mean(1) # mean of the quantiles, in levels
print(f"{a:<7} QLIKE with exp(median) {qlike(rv, mediana):.3f} "
f"with the mean of the quantiles {qlike(rv, media_q):.3f}")
```
```output
indice QLIKE with exp(median) 0.106 with the mean of the quantiles 0.103
cripto QLIKE with exp(median) 0.397 with the mean of the quantiles 0.364
```
On the index the difference is small (0.106 to 0.103); on crypto, with a wider distribution, the QLIKE falls from 0.397
to 0.364. The more uncertain the forecast, the further the mean moves from the median and the more it matters not to
confuse them. Lesson 5.8.5 compares these variance forecasts with a GARCH and a LightGBM.
## Covariates: who accepts them and what for
A **covariate** is another series that helps forecast the one you care about. There are three types, depending on what
is known when you forecast:
```mermaid
flowchart LR
P[Past] --> M[Model]
F[Known future] --> M
S[Static] --> M
M --> Y[Forecast]
```
**Past** covariates are only known up to today (volume, another market series); **known future** covariates are known
in advance for the horizon (the calendar, an auction, a scheduled release); **static** ones describe the series (the
sector, the asset type). In the versions installed in the course environment:
| Model | Covariates | How |
|---|---|---|
| Chronos-Bolt 2.3.2 | no | only the series |
| Chronos-2 (`Chronos2Pipeline`, same library) | past and known future, numerical and categorical | group attention, natively; weights not in the cache |
| TimesFM-2.5 (`timesfm` 3.0.2) | dynamic and static, numerical and categorical | XReg: in-context linear regression plus TimesFM, with JAX |
| TimeGPT (`nixtla` SDK) | future (`X_df`) and historical (`hist_exog_list`) | in the paid service |
**XReg**, the TimesFM-2.5 mechanism (added in October 2025, according to the project README), does not feed the
covariates into the transformer. In `"xreg + timesfm"` mode it fits, for each series, a linear regression of the target
on the covariates **within the context itself** and lets TimesFM forecast the residual; `"timesfm + xreg"` mode does it
the other way round. It requires compiling with `return_backcast=True` and uses JAX (installed in the environment).
Dynamic covariates are passed with the length of the context plus the horizon: their future values have to be known.
What covariate makes sense for tomorrow's volatility? The level README documents a **leverage effect** on the index:
today's log variance depends on the negative part of yesterday's shock (γ = 0.18). Today's return is known at the
close, so “the previous day's negative return” is a legitimate covariate for the next day. As a control, a
**placebo**: Gaussian noise with no relation at all to the series. With one origin every eight sessions (252 days):
```python
import timesfm
tfm = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
"google/timesfm-2.5-200m-pytorch", torch_compile=False)
tfm.compile(timesfm.ForecastConfig(
max_context=C, max_horizon=128, normalize_inputs=True,
use_continuous_quantile_head=True, fix_quantile_crossing=True,
force_flip_invariance=False, per_core_batch_size=64,
return_backcast=True)) # XReg requires it
a = "indice"
y, r = log_rv[a], rend[a]
neg = np.r_[0.0, np.minimum(r[:-1], 0)] / r.std() # neg[j] = r⁻ of j−1
ruido = np.random.default_rng(0).normal(size=len(y))
o = prueba[::8]
real = y[o]
contextos_tfm = [y[i - C:i] for i in o]
# forecast() pads the list it receives up to a multiple of the batch:
# pass it a copy
punto, _ = tfm.forecast(horizon=1, inputs=list(contextos_tfm))
e0 = (real - punto[:, -1]) ** 2 # with backcast, the last one is tomorrow
print(f"no covariates MSE {e0.mean():.4f}")
for nombre, x in (("negative return", neg),
("noise (placebo)", ruido)):
p, _ = tfm.forecast_with_covariates(
inputs=list(contextos_tfm),
dynamic_numerical_covariates={
nombre: [x[i - C:i + 1] for i in o]},
xreg_mode="xreg + timesfm")
e = (real - np.array([v[0] for v in p])) ** 2
d = e - e0
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{nombre:<20} MSE {e.mean():.4f} difference {d.mean():+.4f}"
f" (t = {t:+.1f})")
```
```output
no covariates MSE 0.1853
negative return MSE 0.1769 difference -0.0085 (t = -0.8)
noise (placebo) MSE 0.1850 difference -0.0003 (t = -1.2)
```
A detail in the code that costs a bug: `forecast()` **modifies the list it receives**, padding it with empty series up
to a multiple of the batch size. If the same list is reused in `forecast_with_covariates`, it no longer matches the
covariates. That is why it is passed copies.
The covariate with a real effect lowers the MSE from 0.1853 to 0.1769, by 4.5 %, but with **t = −0.8**: it cannot be
told apart from noise. The placebo leaves the error at 0.1850 (t = −1.2 on a difference of three ten-thousandths). The
effect exists in the data, but it is small: the variance shock is seen through the return with noise (the correlation
between the two is ρ = −0.45), tomorrow's RV has its own measurement error and the linear regression is estimated with
256 days in each context. Claiming the improvement would take many more test days; using it in production would also
take an economic reason.
Chronos-2 would do the same natively. Its code, with the input format from its documentation (one dictionary per
series, with past covariates the size of the context and future ones the size of the horizon):
```python skip
from chronos import Chronos2Pipeline
chronos2 = Chronos2Pipeline.from_pretrained("amazon/chronos-2",
device_map="cpu")
entradas = [{
"target": y[i - C:i],
"past_covariates": {"neg": neg[i - C:i]},
"future_covariates": {"neg": neg[i:i + 1]}, # known at the close
} for i in o]
cuantiles, media = chronos2.predict_quantiles(
entradas, prediction_length=1, quantile_levels=[0.1, 0.5, 0.9])
```
And TimeGPT, with a `DataFrame` of the future covariates (`X_df`), according to its documentation:
```python skip
import os
from nixtla import NixtlaClient
cliente = NixtlaClient(api_key=os.environ["NIXTLA_API_KEY"])
historia = pd.DataFrame({"unique_id": "indice", "ds": fechas[:-1],
"y": y[:-1], "neg": neg[:-1]})
futuro = pd.DataFrame({"unique_id": "indice", "ds": fechas[-1:],
"neg": neg[-1:]})
prevision = cliente.forecast(df=historia, X_df=futuro, h=1, freq="B",
level=[80])
```
## Fine-tuning: when it pays off
**Fine-tuning** trains the model a little further on your own data. There are three degrees: **full** fine-tuning
(all the weights), **adapters** (LoRA: small matrices added to the layers, with the rest frozen) and **the head** (only
the output layer). Full fine-tuning of Chronos-Bolt-small on CPU takes several seconds per training step with batches
of 32 windows on this machine, too much for the lesson; its code, with the quantile loss the model itself computes when
it receives the target, would be this:
```python skip
bolt.model.train()
opt = torch.optim.AdamW(bolt.model.parameters(), lr=1e-5)
for paso in range(1000):
contexto, objetivo = lote_de_ventanas() # (32, C) and (32, 1)
salida = bolt.model(context=contexto, target=objetivo)
opt.zero_grad()
salida.loss.backward()
opt.step()
bolt.model.eval()
```
Chronos-2 ships fine-tuning in the library (`Chronos2Pipeline.fit`, full or with LoRA, which returns a tuned copy),
in the form shown in its quickstart notebook; TimeGPT offers it in the service with `finetune_steps`:
```python skip
ajustado = chronos2.fit(inputs=entradas_entrenamiento,
prediction_length=1, num_steps=1000,
learning_rate=1e-5, batch_size=32)
prevision = cliente.forecast(df=historia, h=1, freq="B",
finetune_steps=10) # TimeGPT
```
What does fit in the lesson is tuning **the head**. In Chronos-Bolt it is a residual block (`ResidualBlock`) that turns
the decoder output into the nine quantiles of the 64 steps. With the rest of the model frozen, the decoder output for
each window is computed only once and training only touches the head: seconds of CPU. The tuning data are windows from
the six assets with the target day between 2009 and 2016; validation, 2017; the test, the same as in the previous
sections (2018-2025). No training window reaches the test.
```python
del tfm
m = bolt.model
cabeza = m.output_patch_embedding
print(cabeza.__class__.__name__,
f"{sum(p.numel() for p in cabeza.parameters()) / 1e6:.2f} M of",
f"{sum(p.numel() for p in m.parameters()) / 1e6:.1f} M",
"parameters")
rng = np.random.default_rng(0)
def ventanas(desde, hasta, n):
"""n windows per asset whose target day falls between those years."""
X, T = [], []
for a, y in log_rv.items():
ok = np.flatnonzero((fechas.year >= desde)
& (fechas.year <= hasta))
for i in rng.choice(ok[ok >= C], n, replace=False):
X.append(y[i - C:i])
T.append(y[i])
return (torch.tensor(np.stack(X), dtype=torch.float32),
torch.tensor(T, dtype=torch.float32))
@torch.no_grad()
def ocultos(X):
"""Output of the frozen decoder and scale of each window."""
H, L = [], []
for k in range(0, len(X), 256):
h, (loc, esc), emb, mask = m.encode(X[k:k + 256])
H.append(m.decode(emb, mask, h))
L.append(torch.cat([loc, esc], 1))
return torch.cat(H), torch.cat(L)
X_tr, T_tr = ventanas(2009, 2016, 200)
X_va, T_va = ventanas(2017, 2017, 50)
H_tr, L_tr = ocultos(X_tr)
H_va, L_va = ocultos(X_va)
print("training", tuple(H_tr.shape),
"validation", tuple(H_va.shape))
```
```output
ResidualBlock 2.53 M of 47.7 M parameters
training (1200, 1, 512) validation (300, 1, 512)
```
The loss is Chronos-Bolt's own (the pinball loss of the nine quantiles, summed and multiplied by 2) on the target
standardized with the mean and standard deviation of the context, as the model does. It trains for 200 steps with Adam
and checks validation every 50:
```python
import copy
tau_t = torch.tensor(m.chronos_config.quantiles)
def perdida(H, L, T):
"""Chronos-Bolt quantile loss at the first step,
with the target standardized like the context."""
q = cabeza(H).view(len(H), 9, -1)[:, :, 0]
z = (T[:, None] - L[:, :1]) / L[:, 1:]
return (2 * torch.abs((z - q) * ((z <= q).float() - tau_t))
).sum(1).mean()
original = copy.deepcopy(cabeza.state_dict())
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-4)
g = torch.Generator().manual_seed(0)
for paso in range(201):
if paso % 50 == 0:
with torch.no_grad():
print(f"step {paso:3d} training "
f"{perdida(H_tr, L_tr, T_tr).item():.3f} validation "
f"{perdida(H_va, L_va, T_va).item():.3f}")
lote = torch.randperm(len(H_tr), generator=g)[:256]
opt.zero_grad()
perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
opt.step()
```
```output
step 0 training 4.430 validation 4.717
step 50 training 4.328 validation 4.681
step 100 training 4.269 validation 4.682
step 150 training 4.223 validation 4.679
step 200 training 4.179 validation 4.684
```
*Fig. — Fine-tuning of Chronos-Bolt-small's output head (2.53 M of 47.7 M parameters) on the six assets' log RV: training (2009-2016) and validation (2017) loss.* Two lines over 200 steps. Training loss falls steadily from 4.430 to 4.179. Validation loss, highlighted, goes from 4.717 to 4.681 in the first 50 steps and stays flat (4.684 at the end): from there on the fit only memorizes the training windows.
The training loss keeps falling (4.430 to 4.179), but the validation loss only improves during the first 50 steps
(4.717 to 4.681) and then stays flat: from there on, the head memorizes the 1,200 windows. The number of steps was set
in advance; choosing it by looking at the test would be the selection from lesson 5.5.4. The test decides:
```python
ft = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
for a in ft:
real = log_rv[a][orig]
u0 = real[:, None] - zs[a]
u1 = real[:, None] - ft[a]
p0 = np.maximum(tau * u0, (tau - 1) * u0).mean(1)
p1 = np.maximum(tau * u1, (tau - 1) * u1).mean(1)
d = p1 - p0
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{a:<7} pinball {p0.mean():.4f} to {p1.mean():.4f} "
f"(t = {t:+.1f}); MSE {np.mean(u0[:, 4] ** 2):.4f} to "
f"{np.mean(u1[:, 4] ** 2):.4f}; coverage "
f"{cobertura(real, zs[a]):.1%} to "
f"{cobertura(real, ft[a]):.1%}")
cabeza.load_state_dict(original); # restores the published model
```
```output
indice pinball 0.1356 to 0.1369 (t = +1.4); MSE 0.1953 to 0.1979; coverage 77.6% to 76.4%
cripto pinball 0.1660 to 0.1675 (t = +1.4); MSE 0.3713 to 0.3671; coverage 80.8% to 79.3%
```
The fine-tuning slightly **worsens** the pinball loss on both assets (0.1356 to 0.1369 on the index, 0.1660 to 0.1675
on crypto, t = 1.4 in both) and lowers the coverage somewhat. The index MSE goes up and the crypto one goes down
(0.3713 to 0.3671): mixed signals within the noise. The conclusion is the honest one: on this series, the pretrained
model was already extracting almost all the information, and 1,200 windows are not enough to teach it something it
does not know without making it forget something else.
| Fine-tuning usually pays off if… | It usually does not pay off if… |
|---|---|
| the series looks little like the corpus (unusual units, frequency or dynamics) | the series has a simple, common structure (persistence, like volatility) |
| there is a lot of in-house data (thousands of series or years of high frequency) | there are few series and few years |
| the zero-shot forecast is clearly biased and the bias is stable | the zero-shot forecast is already close to the best estimated model |
| it is validated with a later test fixed in advance | steps, rate or layers are chosen by looking at the test |
## Exercises
1. Repeat the coverage by thirds of forecast volatility **after** the conformal adjustment, only on 2021-2025. Does the
adjustment fix the index thirds that were failing, or only the overall coverage?
**Solution.**
```python
for a, q in zs.items():
real, qa = log_rv[a][orig][~cal], (q + ajustes[a])[~cal]
q0 = q[~cal]
tercio = pd.qcut(q0[:, 4], 3, labels=["low", "medium", "high"])
fila = []
for t in ("low", "medium", "high"):
s = tercio == t
fila.append(f"{t} {cobertura(real[s], q0[s]):.1%} to "
f"{cobertura(real[s], qa[s]):.1%}")
print(f"{a:<7}", " ".join(fila))
```
```output
indice low 75.2% to 76.7% medium 76.1% to 78.5% high 83.7% to 85.6%
cripto low 82.9% to 82.4% medium 80.4% to 80.9% high 79.4% to 80.4%
```
The adjustment raises the coverage in all three index thirds (from 75.2 % to 76.7 % in the low one, from 76.1 % to
78.5 % in the medium one and from 83.7 % to 85.6 % in the high one), but it does not equalize them: the shift is the
same for every day, so it moves the bias without correcting its shape. After the adjustment, calm days are still below
80 % and volatile ones above. Correcting conditional coverage requires an adjustment that depends on the bucket
(calibrating by thirds, with three times less data in each) or a better model, not a global shift. On crypto, which
was already fine, everything stays between 80 % and 83 %.
2. Repeat the fine-tuning of the head with a learning rate ten times higher (`lr=1e-3`). What happens to the
validation loss?
**Solution.**
```python
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-3)
g = torch.Generator().manual_seed(0)
for paso in range(201):
if paso % 50 == 0:
with torch.no_grad():
print(f"step {paso:3d} training "
f"{perdida(H_tr, L_tr, T_tr).item():.3f} validation "
f"{perdida(H_va, L_va, T_va).item():.3f}")
lote = torch.randperm(len(H_tr), generator=g)[:256]
opt.zero_grad()
perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
opt.step()
cabeza.load_state_dict(original);
```
```output
step 0 training 4.430 validation 4.717
step 50 training 4.089 validation 4.699
step 100 training 3.849 validation 4.819
step 150 training 3.487 validation 4.949
step 200 training 3.160 validation 5.034
```
With a rate ten times higher, validation improves a little in the first 50 steps (4.717 to 4.699) and then **worsens**
steadily up to 5.034, above the starting point, while training drops from 4.430 to 3.160. It is overfitting in its
clearest form: the head learns the 1,200 windows by heart and loses what it knew. Without a validation set separated in
time, the training loss would have said the fine-tuning was going very well. At the end the cell restores the published
head.
3. You want to forecast tomorrow's volatility of a European stock with a covariate model, and you propose two: today's
S&P 500 close (the New York stock exchange closes at 22:00, Madrid time) and tomorrow's VIX at the European open.
You forecast at 17:35, after the European close. Which ones are legitimate and of what type?
**Solution.**
Neither, as stated. At 17:35 the S&P 500 is still open: today's close is not known, so as a covariate for “today” it is
a leak; what is legitimate is **yesterday's** close (past) or its value at 17:35 (past, timestamped at that time).
Tomorrow's VIX at the European open is in the future and not known in advance: it is not a known future covariate, but
what you want to forecast under another name. Genuinely known future covariates would be the company's earnings
calendar or scheduled macro releases (ECB, CPI), which are known days ahead. Any model that accepts covariates will
treat them however it is told; the responsibility for timestamping them lies with whoever prepares them (lesson
5.2.2).
## Self-assessment
1. With Chronos-Bolt on the index log RV, going from 16 to 512 sessions of context lowers the MSE by 9 % with t = 1.7, and from 64 to 512 there is no significant difference. On crypto, from 64 to 512 the improvement has t = 3.1. How do you choose the context?
- A. Always the maximum the model allows
- B. Always the minimum, because it is faster
- C. By measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy
- D. With the length of the longest season
*Answer C: By measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy.* The crypto series is noisier: the model needs more days to estimate its level. Past a certain point, old history adds nothing and the cost of attention keeps growing.
2. On 12.4 % of the days the index log RV falls below the forecast 0.1 quantile, and on 90.1 % below the 0.9 one. What is wrong with the forecast distribution?
- A. The lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated
- B. It is perfectly calibrated
- C. The upper tail is too narrow
- D. The model is biased upward in every quantile
*Answer A: The lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated.* With 1,006 days, the standard error of a 10 % frequency is 0.9 points: 12.4 % is more than two standard errors away. At 0.9 the observed frequency is 90.1 %, right at the nominal value.
3. A per-quantile conformal adjustment, estimated on 2018-2020, takes the 10-90 % coverage of the index in 2021-2025 from 78.3 % to 80.3 %, but the pinball loss goes from 0.1352 to 0.1351. What is gained?
- A. A much better forecast
- B. Nothing: coverage gets worse
- C. A more accurate median
- D. Intervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves
*Answer D: Intervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves.* The adjustment only shifts each quantile by a few hundredths in log. It corrects the average coverage bias, not the model's information. Lesson 5.7.3 explains why, on time series, it does not guarantee exact coverage.
4. With TimesFM-2.5 and XReg, the covariate “yesterday's negative return” lowers the index MSE from 0.1853 to 0.1769 with t = −0.8, and a noise placebo barely changes it. What do you conclude?
- A. That the covariate improves the model by 4.5 % and you should use it
- B. That with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily
- C. That XReg does not work
- D. That the placebo shows the model is overfitted
*Answer B: That with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily.* An improvement with t = −0.8 is not evidence. The placebo serves the opposite purpose: checking that adding a covariate with no information neither worsens things much nor often “improves” them by chance. Claiming the improvement would require a much larger sample.
5. Fine-tuning Chronos-Bolt's head lowers the training loss from 4.430 to 4.179, the validation loss from 4.717 to 4.684 and, in 2018-2025, the index pinball loss goes from 0.1356 to 0.1369. What happened?
- A. The fine-tuning worked
- B. There is a mistake in the evaluation
- C. The model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series
- D. It needed more training steps
*Answer C: The model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series.* It is the classic overfitting pattern, with a model of 2.5 million trainable parameters and 1,200 windows. Fine-tuning helps when the series looks little like the pretraining corpus or there is a lot of in-house data; neither is the case here.
## Key points
- Context length is measured: on the index it makes no difference beyond 32 sessions (16 versus 512: t = 1.7); on crypto, which is noisier, it improves up to 256-512 (64 versus 512: t = 3.1).
- Zero-shot quantiles are almost calibrated: on the index the lower tail is narrow (12.4 % below the 0.1 quantile) and on high-volatility days the coverage changes. A per-quantile conformal adjustment restores the nominal coverage (78.3 % to 80.3 %) without changing the pinball loss.
- The expected variance is not the exponential of the median of the log: the mean of the quantiles in levels lowers the QLIKE (crypto: 0.397 to 0.364).
- Covariates in the installed versions: Chronos-Bolt no; Chronos-2 yes (past, future and categorical; weights not cached); TimesFM-2.5 yes, with XReg (in-context linear regression and TimesFM on the residual). On `vol5`, neither the covariate with a real effect nor the placebo changes anything significantly.
- Fine-tuning Chronos-Bolt's head fits in seconds of CPU, but it overfits: validation stalls and out of sample the pinball loss worsens by 1 % (t = 1.4).
## Resources
### Research
- [Uses of Volatility and Current Volatility - VIX1D](https://pythonparatrading.com/en/uses-of-volatility/) · Research-0003 · 5 min
### Documentation
- [Chronos-2: quickstart notebook (covariates and fine-tuning)](https://github.com/amazon-science/chronos-forecasting/blob/main/notebooks/chronos-2-quickstart.ipynb) · github.com
- [google-research/timesfm (GitHub): covariates with XReg in TimesFM 2.5](https://github.com/google-research/timesfm) · github.com
- [Nixtla: exogenous variables in TimeGPT](https://www.nixtla.io/docs/forecasting/exogenous-variables/numeric_features) · nixtla.io
### Books
- [Romano, Y., Patterson, E. and Candès, E. (2019): Conformalized Quantile Regression. arXiv:1905.03222](https://arxiv.org/abs/1905.03222) · arxiv.org
### Data
- [Level 5 data: returns and realized variance of six assets (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet
Previous lesson: [5.8.2 · Foundation model families: architecture, data and license](https://pythonparatrading.com/en/python-trading-course/level-5/chronos-timesfm-moirai-kronos-families/)
Next lesson: [5.8.4 · Foundation models in finance: what the evidence says](https://pythonparatrading.com/en/python-trading-course/level-5/foundation-models-finance-evidence/)