Lesson 5.8.3 · 75 min
In practice: context, quantiles, covariates and fine-tuning
Forecasting with a foundation model is a single call, but using it well takes several decisions: how much history to give it, how to read and correct its quantiles, when to add covariates and whether it is worth tuning it on your own data. This lesson takes each decision on the level 5 realized volatility and measures it out of sample, with paired differences. Honest spoiler: almost everything that looks like an obvious improvement here improves nothing.
- Module
- 5.8 · Time series foundation models
- Duration
- 75 min
- Prerequisites
- 5.8.2 · Foundation model families: architecture, data and license, 5.7.3 · Probabilistic models: quantile loss, DeepAR and intervals, 5.4.3 · Walk-forward with retraining: windows and frequency
- Author
- Jesús Cuesta
- Updated
Level 5 lessons
5.1 · ML fundamentals applied to finance
5.2 · Feature engineering
5.3 · Labelling and sampling
5.4 · Validation for financial series
5.5 · Classical models
- 5.5.1 Regularized regressions: Ridge, Lasso, Elastic Net and logistic
- 5.5.2 Decision trees, Random Forest and bagging
- 5.5.3 Gradient boosting: XGBoost, LightGBM and CatBoost
- 5.5.4 Optuna and the risk of overfitting the validation
- 5.5.5 Unsupervised learning: PCA, clustering and autoencoders
- 5.5.6 Learning to rank for stock selection
5.6 · Deep learning with Keras 3
5.7 · Deep learning with PyTorch
- 5.7.1 PyTorch: tensors, autograd and the training loop
- 5.7.2 Time series architectures: TCN, N-BEATS, PatchTST and iTransformer
- 5.7.3 Probabilistic models: quantile loss, DeepAR and intervals
- 5.7.4 Graph neural networks for asset relationships
- 5.7.5 End-to-end portfolio optimization: differentiable layers and Sharpe loss
5.8 · Time series foundation models
- 5.8.1 Time series foundation models: pretrain once, forecast without training
- 5.8.2 Foundation model families: architecture, data and license
- 5.8.3 In practice: context, quantiles, covariates and fine-tuning
- 5.8.4 Foundation models in finance: what the evidence says
- 5.8.5 Lab: foundation models vs GARCH, HAR and LightGBM
5.9 · Reinforcement learning
5.10 · Interpretability, robustness and model risk
The decisions the call does not make
Lessons 5.8.1 and 5.8.2 used the models almost with their default options. In practice several things have to be decided, and each one can be measured with an out-of-sample test:
Mermaid source
flowchart TB A[Transform] --> B[Context] B --> C[Quantiles] C --> D[Recalibrate] D --> E[Covariates] E --> F[Fine-tuning] F --> G[Honest test]
The test bench is still the one from the previous lessons: the log of the realized variance from vol5, with a
one-day forecast and a rolling origin over 2018-2025. The transformation is already decided (the log: almost Gaussian
and unbounded). The rest is decided here.
import numpy as np
import pandas as pd
import torch
from chronos import BaseChronosPipeline
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
vol = pd.read_parquet("vol5_diario.parquet")
log_rv, rend = {}, {}
for a in vol["activo"].unique():
d = vol[vol["activo"] == a].set_index("fecha")
log_rv[a] = np.log(d["rv_5min"].to_numpy())
rend[a] = d["rendimiento"].to_numpy()
fechas = pd.DatetimeIndex(d.index)
prueba = np.flatnonzero(fechas.year >= 2018)
bolt = BaseChronosPipeline.from_pretrained(
"amazon/chronos-bolt-small", device_map="cpu",
dtype=torch.float32)
def zero_shot(y, origenes, contexto, lote=256):
"""Tomorrow's 0.1…0.9 quantiles with `contexto` sessions."""
salida = []
for k in range(0, len(origenes), lote):
v = np.stack([y[i - contexto:i] for i in origenes[k:k + lote]])
q, _ = bolt.predict_quantiles(
inputs=torch.tensor(v, dtype=torch.float32),
prediction_length=1)
salida.append(q[:, 0, :].numpy().astype(float))
return np.concatenate(salida)
print(sorted(log_rv), len(prueba), "test sessions")['accion', 'bono', 'cripto', 'divisa', 'indice', 'oro'] 2011 test sessionsHow much history to give it
Chronos-Bolt accepts up to 2,048 sessions of context and TimesFM-2.5 up to 16,384. More context gives more information about the level and dynamics of the series, but it costs time (more patches) and can bring in history that no longer looks like the present. The cell sweeps from 16 to 1,024 sessions with one origin every 16 (126 days per asset) and compares each context with the 512 one through the paired difference.
orig = prueba[::16]
contextos = [16, 32, 64, 128, 256, 512, 1024]
ecm, err = {}, {}
for a in ("indice", "cripto"):
real = log_rv[a][orig]
for c in contextos:
e = (real - zero_shot(log_rv[a], orig, c)[:, 4]) ** 2
ecm[(a, c)], err[(a, c)] = e.mean(), e
tabla = pd.Series(ecm).unstack(0).round(3)
tabla.index.name = "context"
print(tabla.to_string())
for a in ("indice", "cripto"):
for c in (16, 64):
d = err[(a, c)] - err[(a, 512)]
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{a}: {c} − 512 sessions {d.mean():+.4f} (t = {t:+.1f})") cripto indice
context
16 0.429 0.240
32 0.412 0.220
64 0.417 0.228
128 0.402 0.223
256 0.393 0.221
512 0.387 0.219
1024 0.390 0.218
indice: 16 − 512 sessions +0.0205 (t = +1.7)
indice: 64 − 512 sessions +0.0092 (t = +1.1)
cripto: 16 − 512 sessions +0.0413 (t = +3.1)
cripto: 64 − 512 sessions +0.0296 (t = +3.1)The two assets behave differently. On the index, the error drops from 0.240 with 16 sessions to 0.220 with 32 and from there it moves between 0.218 and 0.228 with no clear order; not even 16 versus 512 is significant (t = 1.7). On crypto, the error falls steadily from 0.429 to 0.387 with 512 sessions, and the difference against 16 or 64 sessions has t = 3.1. The reason is noise: crypto RV, with twelve jumps a year on average and a large measurement error, needs more days for the model to estimate its level; on the index, with no jumps, a month and a half is enough. With 1,024 sessions neither gains anything. In practice, 256-512 sessions is a reasonable choice for daily volatility, and the test has to be repeated on each series.
Quantiles: do they cover what they promise?
Chronos-Bolt returns nine quantiles. A quantile is calibrated if the realized value falls below it on a fraction of the days. Lesson 5.7.3 showed how to measure it with trained models; the difference here is that the model has not seen the series, so its calibration is whatever it brings from pretraining. With one origin every two sessions (1,006 days per asset) and a context of 256:
C = 256
tau = np.arange(1, 10) / 10
orig = prueba[::2]
zs = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
filas = {}
for a, q in zs.items():
real = log_rv[a][orig]
filas[a] = (real[:, None] <= q).mean(0)
frec = pd.DataFrame(filas, index=tau).round(3)
frec.index.name = "nominal"
print(frec.T.to_string())nominal 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
indice 0.124 0.235 0.332 0.428 0.507 0.600 0.690 0.786 0.901
cripto 0.103 0.206 0.296 0.394 0.480 0.583 0.685 0.796 0.912The header figure plots the difference between the observed and the nominal frequency, with the ±2 standard error band (with 1,006 days, the standard error of a 50 % frequency is ). On crypto the nine quantiles fall inside or close to the band: calibration is good. On the index, the lower tail is too narrow: 12.4 % of the days fall below the 0.1 quantile and 33.2 % below the 0.3 one, when they should be 10 % and 30 %. The upper tail is fine (90.1 % at 0.9). Put another way: the model does not expect days as calm as those of the synthetic index.
Overall coverage can hide problems that only show up when you condition, as lesson 5.7.3 showed. The test is split into thirds by the forecast median (a value known the day before):
def cobertura(real, q):
return np.mean((real >= q[:, 0]) & (real <= q[:, 8]))
for a, q in zs.items():
real = log_rv[a][orig]
tercio = pd.qcut(q[:, 4], 3, labels=["low", "medium", "high"])
cob = {t: cobertura(real[tercio == t], q[tercio == t])
for t in ("low", "medium", "high")}
print(f"{a:<7} total {cobertura(real, q):.1%} " + " ".join(
f"{t} {v:.1%}" for t, v in cob.items()))indice total 77.6% low 76.2% medium 75.8% high 80.9%
cripto total 80.8% low 80.4% medium 83.3% high 78.8%On the index the 10 to 90 % interval covers 76.2 % and 75.8 % in the low and medium volatility thirds, and 80.9 % in the high one: it is on normal days that it fails, consistent with the narrow lower tail. On crypto the pattern is the opposite and milder (83.3 % in the middle third, 78.8 % in the high one). With about 335 days per third, the standard error of each coverage is about 2.2 points, so only the index deviations in the low and medium thirds are clearly outside.
Recalibrating: a per-quantile conformal adjustment
If the quantiles are biased in a stable way, they can be corrected with a calibration period. The simplest version of conformalized quantile regression (Romano, Patterson and Candès, 2019) shifts each quantile by the empirical quantile of its residuals: with the residuals from the calibration period, the corrected quantile is . It is calibrated on 2018-2020 and measured on 2021-2025, without looking at the test:
def pinball(real, q):
u = real[:, None] - q
return np.maximum(tau * u, (tau - 1) * u).mean()
cal = fechas.year[orig] <= 2020
ajustes = {}
for a, q in zs.items():
real = log_rv[a][orig]
# per-quantile shift: nominal frequency in 2018-2020
res = real[cal, None] - q[cal]
delta = np.array([np.quantile(res[:, j], tau[j])
for j in range(9)])
qa = q + delta
ajustes[a] = delta
c0 = cobertura(real[~cal], q[~cal])
c1 = cobertura(real[~cal], qa[~cal])
print(f"{a:<7} 2021-2025: coverage {c0:.1%} to {c1:.1%}; pinball "
f"{pinball(real[~cal], q[~cal]):.4f} to "
f"{pinball(real[~cal], qa[~cal]):.4f}")
print({a: d.round(3).tolist() for a, d in ajustes.items()})indice 2021-2025: coverage 78.3% to 80.3%; pinball 0.1352 to 0.1351
cripto 2021-2025: coverage 80.9% to 81.2%; pinball 0.1659 to 0.1661
{'indice': [-0.046, -0.039, -0.046, -0.026, 0.024, 0.024, 0.023, 0.0, 0.02], 'cripto': [-0.047, -0.011, -0.003, -0.008, 0.026, 0.008, 0.015, -0.014, -0.062]}On the index, the 2021-2025 coverage rises from 78.3 % to 80.3 %, almost the nominal value; the shifts are a few hundredths in log (−0.046 at the 0.1 quantile, which lowers the tail). On crypto, which was already calibrated, it goes from 80.9 % to 81.2 %. The pinball loss barely changes (0.1352 to 0.1351 and 0.1659 to 0.1661): the adjustment corrects the average coverage, it does not add information. It is useful when what you will use is the interval (to set a risk limit or the size of a position, lesson 4.2.2), not to improve the forecast. And it does not guarantee exact coverage on time series, because the residuals are not exchangeable: lesson 5.7.3 covers this in detail.
From quantiles of the log to the variance
For sizing or comparing with a GARCH you need tomorrow's expected variance, not the median of its log. Exponentiating the median gives the median of the variance, which is lower than its mean when the distribution has a right tail. With nine quantiles there is a direct approximation: exponentiate each quantile and average (a crude quadrature of the expectation). The measure for comparing variance forecasts is QLIKE, the one in the level README and in lesson 5.8.5: , robust to the noise of RV as a measure of the variance.
def qlike(rv, h):
return np.mean(rv / h - np.log(rv / h) - 1)
for a, q in zs.items():
rv = np.exp(log_rv[a][orig])
mediana = np.exp(q[:, 4])
media_q = np.exp(q).mean(1) # mean of the quantiles, in levels
print(f"{a:<7} QLIKE with exp(median) {qlike(rv, mediana):.3f} "
f"with the mean of the quantiles {qlike(rv, media_q):.3f}")indice QLIKE with exp(median) 0.106 with the mean of the quantiles 0.103
cripto QLIKE with exp(median) 0.397 with the mean of the quantiles 0.364On the index the difference is small (0.106 to 0.103); on crypto, with a wider distribution, the QLIKE falls from 0.397 to 0.364. The more uncertain the forecast, the further the mean moves from the median and the more it matters not to confuse them. Lesson 5.8.5 compares these variance forecasts with a GARCH and a LightGBM.
Covariates: who accepts them and what for
A covariate is another series that helps forecast the one you care about. There are three types, depending on what is known when you forecast:
Mermaid source
flowchart LR P[Past] --> M[Model] F[Known future] --> M S[Static] --> M M --> Y[Forecast]
Past covariates are only known up to today (volume, another market series); known future covariates are known in advance for the horizon (the calendar, an auction, a scheduled release); static ones describe the series (the sector, the asset type). In the versions installed in the course environment:
| Model | Covariates | How |
|---|---|---|
| Chronos-Bolt 2.3.2 | no | only the series |
Chronos-2 (Chronos2Pipeline, same library) | past and known future, numerical and categorical | group attention, natively; weights not in the cache |
TimesFM-2.5 (timesfm 3.0.2) | dynamic and static, numerical and categorical | XReg: in-context linear regression plus TimesFM, with JAX |
TimeGPT (nixtla SDK) | future (X_df) and historical (hist_exog_list) | in the paid service |
XReg, the TimesFM-2.5 mechanism (added in October 2025, according to the project README), does not feed the
covariates into the transformer. In "xreg + timesfm" mode it fits, for each series, a linear regression of the target
on the covariates within the context itself and lets TimesFM forecast the residual; "timesfm + xreg" mode does it
the other way round. It requires compiling with return_backcast=True and uses JAX (installed in the environment).
Dynamic covariates are passed with the length of the context plus the horizon: their future values have to be known.
What covariate makes sense for tomorrow's volatility? The level README documents a leverage effect on the index: today's log variance depends on the negative part of yesterday's shock (γ = 0.18). Today's return is known at the close, so “the previous day's negative return” is a legitimate covariate for the next day. As a control, a placebo: Gaussian noise with no relation at all to the series. With one origin every eight sessions (252 days):
import timesfm
tfm = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
"google/timesfm-2.5-200m-pytorch", torch_compile=False)
tfm.compile(timesfm.ForecastConfig(
max_context=C, max_horizon=128, normalize_inputs=True,
use_continuous_quantile_head=True, fix_quantile_crossing=True,
force_flip_invariance=False, per_core_batch_size=64,
return_backcast=True)) # XReg requires it
a = "indice"
y, r = log_rv[a], rend[a]
neg = np.r_[0.0, np.minimum(r[:-1], 0)] / r.std() # neg[j] = r⁻ of j−1
ruido = np.random.default_rng(0).normal(size=len(y))
o = prueba[::8]
real = y[o]
contextos_tfm = [y[i - C:i] for i in o]
# forecast() pads the list it receives up to a multiple of the batch:
# pass it a copy
punto, _ = tfm.forecast(horizon=1, inputs=list(contextos_tfm))
e0 = (real - punto[:, -1]) ** 2 # with backcast, the last one is tomorrow
print(f"no covariates MSE {e0.mean():.4f}")
for nombre, x in (("negative return", neg),
("noise (placebo)", ruido)):
p, _ = tfm.forecast_with_covariates(
inputs=list(contextos_tfm),
dynamic_numerical_covariates={
nombre: [x[i - C:i + 1] for i in o]},
xreg_mode="xreg + timesfm")
e = (real - np.array([v[0] for v in p])) ** 2
d = e - e0
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{nombre:<20} MSE {e.mean():.4f} difference {d.mean():+.4f}"
f" (t = {t:+.1f})")no covariates MSE 0.1853
negative return MSE 0.1769 difference -0.0085 (t = -0.8)
noise (placebo) MSE 0.1850 difference -0.0003 (t = -1.2)A detail in the code that costs a bug: forecast() modifies the list it receives, padding it with empty series up
to a multiple of the batch size. If the same list is reused in forecast_with_covariates, it no longer matches the
covariates. That is why it is passed copies.
The covariate with a real effect lowers the MSE from 0.1853 to 0.1769, by 4.5 %, but with t = −0.8: it cannot be told apart from noise. The placebo leaves the error at 0.1850 (t = −1.2 on a difference of three ten-thousandths). The effect exists in the data, but it is small: the variance shock is seen through the return with noise (the correlation between the two is ρ = −0.45), tomorrow's RV has its own measurement error and the linear regression is estimated with 256 days in each context. Claiming the improvement would take many more test days; using it in production would also take an economic reason.
Chronos-2 would do the same natively. Its code, with the input format from its documentation (one dictionary per series, with past covariates the size of the context and future ones the size of the horizon):
from chronos import Chronos2Pipeline
chronos2 = Chronos2Pipeline.from_pretrained("amazon/chronos-2",
device_map="cpu")
entradas = [{
"target": y[i - C:i],
"past_covariates": {"neg": neg[i - C:i]},
"future_covariates": {"neg": neg[i:i + 1]}, # known at the close
} for i in o]
cuantiles, media = chronos2.predict_quantiles(
entradas, prediction_length=1, quantile_levels=[0.1, 0.5, 0.9])And TimeGPT, with a DataFrame of the future covariates (X_df), according to its documentation:
import os
from nixtla import NixtlaClient
cliente = NixtlaClient(api_key=os.environ["NIXTLA_API_KEY"])
historia = pd.DataFrame({"unique_id": "indice", "ds": fechas[:-1],
"y": y[:-1], "neg": neg[:-1]})
futuro = pd.DataFrame({"unique_id": "indice", "ds": fechas[-1:],
"neg": neg[-1:]})
prevision = cliente.forecast(df=historia, X_df=futuro, h=1, freq="B",
level=[80])Fine-tuning: when it pays off
Fine-tuning trains the model a little further on your own data. There are three degrees: full fine-tuning (all the weights), adapters (LoRA: small matrices added to the layers, with the rest frozen) and the head (only the output layer). Full fine-tuning of Chronos-Bolt-small on CPU takes several seconds per training step with batches of 32 windows on this machine, too much for the lesson; its code, with the quantile loss the model itself computes when it receives the target, would be this:
bolt.model.train()
opt = torch.optim.AdamW(bolt.model.parameters(), lr=1e-5)
for paso in range(1000):
contexto, objetivo = lote_de_ventanas() # (32, C) and (32, 1)
salida = bolt.model(context=contexto, target=objetivo)
opt.zero_grad()
salida.loss.backward()
opt.step()
bolt.model.eval()Chronos-2 ships fine-tuning in the library (Chronos2Pipeline.fit, full or with LoRA, which returns a tuned copy),
in the form shown in its quickstart notebook; TimeGPT offers it in the service with finetune_steps:
ajustado = chronos2.fit(inputs=entradas_entrenamiento,
prediction_length=1, num_steps=1000,
learning_rate=1e-5, batch_size=32)
prevision = cliente.forecast(df=historia, h=1, freq="B",
finetune_steps=10) # TimeGPTWhat does fit in the lesson is tuning the head. In Chronos-Bolt it is a residual block (ResidualBlock) that turns
the decoder output into the nine quantiles of the 64 steps. With the rest of the model frozen, the decoder output for
each window is computed only once and training only touches the head: seconds of CPU. The tuning data are windows from
the six assets with the target day between 2009 and 2016; validation, 2017; the test, the same as in the previous
sections (2018-2025). No training window reaches the test.
del tfm
m = bolt.model
cabeza = m.output_patch_embedding
print(cabeza.__class__.__name__,
f"{sum(p.numel() for p in cabeza.parameters()) / 1e6:.2f} M of",
f"{sum(p.numel() for p in m.parameters()) / 1e6:.1f} M",
"parameters")
rng = np.random.default_rng(0)
def ventanas(desde, hasta, n):
"""n windows per asset whose target day falls between those years."""
X, T = [], []
for a, y in log_rv.items():
ok = np.flatnonzero((fechas.year >= desde)
& (fechas.year <= hasta))
for i in rng.choice(ok[ok >= C], n, replace=False):
X.append(y[i - C:i])
T.append(y[i])
return (torch.tensor(np.stack(X), dtype=torch.float32),
torch.tensor(T, dtype=torch.float32))
@torch.no_grad()
def ocultos(X):
"""Output of the frozen decoder and scale of each window."""
H, L = [], []
for k in range(0, len(X), 256):
h, (loc, esc), emb, mask = m.encode(X[k:k + 256])
H.append(m.decode(emb, mask, h))
L.append(torch.cat([loc, esc], 1))
return torch.cat(H), torch.cat(L)
X_tr, T_tr = ventanas(2009, 2016, 200)
X_va, T_va = ventanas(2017, 2017, 50)
H_tr, L_tr = ocultos(X_tr)
H_va, L_va = ocultos(X_va)
print("training", tuple(H_tr.shape),
"validation", tuple(H_va.shape))ResidualBlock 2.53 M of 47.7 M parameters
training (1200, 1, 512) validation (300, 1, 512)The loss is Chronos-Bolt's own (the pinball loss of the nine quantiles, summed and multiplied by 2) on the target standardized with the mean and standard deviation of the context, as the model does. It trains for 200 steps with Adam and checks validation every 50:
import copy
tau_t = torch.tensor(m.chronos_config.quantiles)
def perdida(H, L, T):
"""Chronos-Bolt quantile loss at the first step,
with the target standardized like the context."""
q = cabeza(H).view(len(H), 9, -1)[:, :, 0]
z = (T[:, None] - L[:, :1]) / L[:, 1:]
return (2 * torch.abs((z - q) * ((z <= q).float() - tau_t))
).sum(1).mean()
original = copy.deepcopy(cabeza.state_dict())
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-4)
g = torch.Generator().manual_seed(0)
for paso in range(201):
if paso % 50 == 0:
with torch.no_grad():
print(f"step {paso:3d} training "
f"{perdida(H_tr, L_tr, T_tr).item():.3f} validation "
f"{perdida(H_va, L_va, T_va).item():.3f}")
lote = torch.randperm(len(H_tr), generator=g)[:256]
opt.zero_grad()
perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
opt.step()step 0 training 4.430 validation 4.717
step 50 training 4.328 validation 4.681
step 100 training 4.269 validation 4.682
step 150 training 4.223 validation 4.679
step 200 training 4.179 validation 4.684The training loss keeps falling (4.430 to 4.179), but the validation loss only improves during the first 50 steps (4.717 to 4.681) and then stays flat: from there on, the head memorizes the 1,200 windows. The number of steps was set in advance; choosing it by looking at the test would be the selection from lesson 5.5.4. The test decides:
ft = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
for a in ft:
real = log_rv[a][orig]
u0 = real[:, None] - zs[a]
u1 = real[:, None] - ft[a]
p0 = np.maximum(tau * u0, (tau - 1) * u0).mean(1)
p1 = np.maximum(tau * u1, (tau - 1) * u1).mean(1)
d = p1 - p0
t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
print(f"{a:<7} pinball {p0.mean():.4f} to {p1.mean():.4f} "
f"(t = {t:+.1f}); MSE {np.mean(u0[:, 4] ** 2):.4f} to "
f"{np.mean(u1[:, 4] ** 2):.4f}; coverage "
f"{cobertura(real, zs[a]):.1%} to "
f"{cobertura(real, ft[a]):.1%}")
cabeza.load_state_dict(original); # restores the published modelindice pinball 0.1356 to 0.1369 (t = +1.4); MSE 0.1953 to 0.1979; coverage 77.6% to 76.4%
cripto pinball 0.1660 to 0.1675 (t = +1.4); MSE 0.3713 to 0.3671; coverage 80.8% to 79.3%The fine-tuning slightly worsens the pinball loss on both assets (0.1356 to 0.1369 on the index, 0.1660 to 0.1675 on crypto, t = 1.4 in both) and lowers the coverage somewhat. The index MSE goes up and the crypto one goes down (0.3713 to 0.3671): mixed signals within the noise. The conclusion is the honest one: on this series, the pretrained model was already extracting almost all the information, and 1,200 windows are not enough to teach it something it does not know without making it forget something else.
| Fine-tuning usually pays off if… | It usually does not pay off if… |
|---|---|
| the series looks little like the corpus (unusual units, frequency or dynamics) | the series has a simple, common structure (persistence, like volatility) |
| there is a lot of in-house data (thousands of series or years of high frequency) | there are few series and few years |
| the zero-shot forecast is clearly biased and the bias is stable | the zero-shot forecast is already close to the best estimated model |
| it is validated with a later test fixed in advance | steps, rate or layers are chosen by looking at the test |
Exercises
- Repeat the coverage by thirds of forecast volatility after the conformal adjustment, only on 2021-2025. Does the adjustment fix the index thirds that were failing, or only the overall coverage?
Solution
for a, q in zs.items():
real, qa = log_rv[a][orig][~cal], (q + ajustes[a])[~cal]
q0 = q[~cal]
tercio = pd.qcut(q0[:, 4], 3, labels=["low", "medium", "high"])
fila = []
for t in ("low", "medium", "high"):
s = tercio == t
fila.append(f"{t} {cobertura(real[s], q0[s]):.1%} to "
f"{cobertura(real[s], qa[s]):.1%}")
print(f"{a:<7}", " ".join(fila))indice low 75.2% to 76.7% medium 76.1% to 78.5% high 83.7% to 85.6%
cripto low 82.9% to 82.4% medium 80.4% to 80.9% high 79.4% to 80.4%The adjustment raises the coverage in all three index thirds (from 75.2 % to 76.7 % in the low one, from 76.1 % to 78.5 % in the medium one and from 83.7 % to 85.6 % in the high one), but it does not equalize them: the shift is the same for every day, so it moves the bias without correcting its shape. After the adjustment, calm days are still below 80 % and volatile ones above. Correcting conditional coverage requires an adjustment that depends on the bucket (calibrating by thirds, with three times less data in each) or a better model, not a global shift. On crypto, which was already fine, everything stays between 80 % and 83 %.
- Repeat the fine-tuning of the head with a learning rate ten times higher (
lr=1e-3). What happens to the validation loss?
Solution
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-3)
g = torch.Generator().manual_seed(0)
for paso in range(201):
if paso % 50 == 0:
with torch.no_grad():
print(f"step {paso:3d} training "
f"{perdida(H_tr, L_tr, T_tr).item():.3f} validation "
f"{perdida(H_va, L_va, T_va).item():.3f}")
lote = torch.randperm(len(H_tr), generator=g)[:256]
opt.zero_grad()
perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
opt.step()
cabeza.load_state_dict(original);step 0 training 4.430 validation 4.717
step 50 training 4.089 validation 4.699
step 100 training 3.849 validation 4.819
step 150 training 3.487 validation 4.949
step 200 training 3.160 validation 5.034With a rate ten times higher, validation improves a little in the first 50 steps (4.717 to 4.699) and then worsens steadily up to 5.034, above the starting point, while training drops from 4.430 to 3.160. It is overfitting in its clearest form: the head learns the 1,200 windows by heart and loses what it knew. Without a validation set separated in time, the training loss would have said the fine-tuning was going very well. At the end the cell restores the published head.
- You want to forecast tomorrow's volatility of a European stock with a covariate model, and you propose two: today's S&P 500 close (the New York stock exchange closes at 22:00, Madrid time) and tomorrow's VIX at the European open. You forecast at 17:35, after the European close. Which ones are legitimate and of what type?
Solution
Neither, as stated. At 17:35 the S&P 500 is still open: today's close is not known, so as a covariate for “today” it is a leak; what is legitimate is yesterday's close (past) or its value at 17:35 (past, timestamped at that time). Tomorrow's VIX at the European open is in the future and not known in advance: it is not a known future covariate, but what you want to forecast under another name. Genuinely known future covariates would be the company's earnings calendar or scheduled macro releases (ECB, CPI), which are known days ahead. Any model that accepts covariates will treat them however it is told; the responsibility for timestamping them lies with whoever prepares them (lesson 5.2.2).
Each answer is folded under its question
Self-assessment
With Chronos-Bolt on the index log RV, going from 16 to 512 sessions of context lowers the MSE by 9 % with t = 1.7, and from 64 to 512 there is no significant difference. On crypto, from 64 to 512 the improvement has t = 3.1. How do you choose the context?
- AAlways the maximum the model allows
- BAlways the minimum, because it is faster
- CBy measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy
- DWith the length of the longest season
Show answer
Answer C. By measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy
The crypto series is noisier: the model needs more days to estimate its level. Past a certain point, old history adds nothing and the cost of attention keeps growing.
On 12.4 % of the days the index log RV falls below the forecast 0.1 quantile, and on 90.1 % below the 0.9 one. What is wrong with the forecast distribution?
- AThe lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated
- BIt is perfectly calibrated
- CThe upper tail is too narrow
- DThe model is biased upward in every quantile
Show answer
Answer A. The lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated
With 1,006 days, the standard error of a 10 % frequency is 0.9 points: 12.4 % is more than two standard errors away. At 0.9 the observed frequency is 90.1 %, right at the nominal value.
A per-quantile conformal adjustment, estimated on 2018-2020, takes the 10-90 % coverage of the index in 2021-2025 from 78.3 % to 80.3 %, but the pinball loss goes from 0.1352 to 0.1351. What is gained?
- AA much better forecast
- BNothing: coverage gets worse
- CA more accurate median
- DIntervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves
Show answer
Answer D. Intervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves
The adjustment only shifts each quantile by a few hundredths in log. It corrects the average coverage bias, not the model's information. Lesson 5.7.3 explains why, on time series, it does not guarantee exact coverage.
With TimesFM-2.5 and XReg, the covariate “yesterday's negative return” lowers the index MSE from 0.1853 to 0.1769 with t = −0.8, and a noise placebo barely changes it. What do you conclude?
- AThat the covariate improves the model by 4.5 % and you should use it
- BThat with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily
- CThat XReg does not work
- DThat the placebo shows the model is overfitted
Show answer
Answer B. That with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily
An improvement with t = −0.8 is not evidence. The placebo serves the opposite purpose: checking that adding a covariate with no information neither worsens things much nor often “improves” them by chance. Claiming the improvement would require a much larger sample.
Fine-tuning Chronos-Bolt's head lowers the training loss from 4.430 to 4.179, the validation loss from 4.717 to 4.684 and, in 2018-2025, the index pinball loss goes from 0.1356 to 0.1369. What happened?
- AThe fine-tuning worked
- BThere is a mistake in the evaluation
- CThe model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series
- DIt needed more training steps
Show answer
Answer C. The model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series
It is the classic overfitting pattern, with a model of 2.5 million trainable parameters and 1,200 windows. Fine-tuning helps when the series looks little like the pretraining corpus or there is a lot of in-house data; neither is the case here.
Key points
- Context length is measured: on the index it makes no difference beyond 32 sessions (16 versus 512: t = 1.7); on crypto, which is noisier, it improves up to 256-512 (64 versus 512: t = 3.1).
- Zero-shot quantiles are almost calibrated: on the index the lower tail is narrow (12.4 % below the 0.1 quantile) and on high-volatility days the coverage changes. A per-quantile conformal adjustment restores the nominal coverage (78.3 % to 80.3 %) without changing the pinball loss.
- The expected variance is not the exponential of the median of the log: the mean of the quantiles in levels lowers the QLIKE (crypto: 0.397 to 0.364).
- Covariates in the installed versions: Chronos-Bolt no; Chronos-2 yes (past, future and categorical; weights not cached); TimesFM-2.5 yes, with XReg (in-context linear regression and TimesFM on the residual). On
vol5, neither the covariate with a real effect nor the placebo changes anything significantly. - Fine-tuning Chronos-Bolt's head fits in seconds of CPU, but it overfits: validation stalls and out of sample the pinball loss worsens by 1 % (t = 1.4).