Lesson 5.8.3 · 75 min

In practice: context, quantiles, covariates and fine-tuning

Forecasting with a foundation model is a single call, but using it well takes several decisions: how much history to give it, how to read and correct its quantiles, when to add covariates and whether it is worth tuning it on your own data. This lesson takes each decision on the level 5 realized volatility and measures it out of sample, with paired differences. Honest spoiler: almost everything that looks like an obvious improvement here improves nothing.

Fig. — Calibration of zero-shot Chronos-Bolt's nine quantiles on tomorrow's log RV, 2018-2025 (1,006 days): observed frequency below each quantile minus its nominal level
Level 5 lessons
  1. 5.1 · ML fundamentals applied to finance

    1. 5.1.1 Why financial machine learning is different
    2. 5.1.2 Supervised, unsupervised and reinforcement learning; scikit-learn pipelines
    3. 5.1.3 Economic metrics for trading models: beyond accuracy
  2. 5.2 · Feature engineering

    1. 5.2.1 Price, volume and microstructure features
    2. 5.2.2 Point-in-time fundamentals, macro and alternative data
    3. 5.2.3 Fractional differentiation for features
    4. 5.2.4 Leakage-free normalization and cross-sectional rank
    5. 5.2.5 Feature importance: MDI, MDA, SFI and SHAP
  3. 5.3 · Labelling and sampling

    1. 5.3.1 Fixed-horizon labeling and its problems
    2. 5.3.2 Triple-barrier method: labels that mimic a trade
    3. 5.3.3 Meta-labeling: one model for the side and another for the size
    4. 5.3.4 Label uniqueness, overlap weights and the sequential bootstrap
    5. 5.3.5 Event-based sampling: the symmetric CUSUM filter
  4. 5.4 · Validation for financial series

    1. 5.4.1 Why random k-fold fails on financial data
    2. 5.4.2 Purged k-fold with embargo and CPCV for ML models
    3. 5.4.3 Walk-forward with retraining: windows and frequency
    4. 5.4.4 Backtesting ML signals with costs and Deflated Sharpe
  5. 5.5 · Classical models

    1. 5.5.1 Regularized regressions: Ridge, Lasso, Elastic Net and logistic
    2. 5.5.2 Decision trees, Random Forest and bagging
    3. 5.5.3 Gradient boosting: XGBoost, LightGBM and CatBoost
    4. 5.5.4 Optuna and the risk of overfitting the validation
    5. 5.5.5 Unsupervised learning: PCA, clustering and autoencoders
    6. 5.5.6 Learning to rank for stock selection
  6. 5.6 · Deep learning with Keras 3

    1. 5.6.1 Keras 3: one model, several compute engines
    2. 5.6.2 An MLP on a stock panel: regularize and compare
    3. 5.6.3 1D CNN, LSTM and GRU: windows, scaling and leaks
    4. 5.6.4 Callbacks, a Sharpe loss and the model in production
  7. 5.7 · Deep learning with PyTorch

    1. 5.7.1 PyTorch: tensors, autograd and the training loop
    2. 5.7.2 Time series architectures: TCN, N-BEATS, PatchTST and iTransformer
    3. 5.7.3 Probabilistic models: quantile loss, DeepAR and intervals
    4. 5.7.4 Graph neural networks for asset relationships
    5. 5.7.5 End-to-end portfolio optimization: differentiable layers and Sharpe loss
  8. 5.8 · Time series foundation models

    1. 5.8.1 Time series foundation models: pretrain once, forecast without training
    2. 5.8.2 Foundation model families: architecture, data and license
    3. 5.8.3 In practice: context, quantiles, covariates and fine-tuning
    4. 5.8.4 Foundation models in finance: what the evidence says
    5. 5.8.5 Lab: foundation models vs GARCH, HAR and LightGBM
  9. 5.9 · Reinforcement learning

    1. 5.9.1 MDP, Q-learning, PPO and Gymnasium environments
    2. 5.9.2 RL for execution, market making and dynamic hedging
    3. 5.9.3 Why reinforcement learning overfits in directional trading
  10. 5.10 · Interpretability, robustness and model risk

    1. 5.10.1 SHAP, partial dependence and the stability of feature importances
    2. 5.10.2 Data drift and concept drift: detecting them and what to do
    3. 5.10.3 Ensembles, signal combination and synthetic data

The decisions the call does not make

Lessons 5.8.1 and 5.8.2 used the models almost with their default options. In practice several things have to be decided, and each one can be measured with an out-of-sample test:

TransformContextQuantilesRecalibrateCovariatesFine-tuningHonest test
Mermaid source
flowchart TB
  A[Transform] --> B[Context]
  B --> C[Quantiles]
  C --> D[Recalibrate]
  D --> E[Covariates]
  E --> F[Fine-tuning]
  F --> G[Honest test]

The test bench is still the one from the previous lessons: the log of the realized variance from vol5, with a one-day forecast and a rolling origin over 2018-2025. The transformation is already decided (the log: almost Gaussian and unbounded). The rest is decided here.

IN [01]python
import numpy as np
import pandas as pd
import torch
from chronos import BaseChronosPipeline

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
vol = pd.read_parquet("vol5_diario.parquet")
log_rv, rend = {}, {}
for a in vol["activo"].unique():
    d = vol[vol["activo"] == a].set_index("fecha")
    log_rv[a] = np.log(d["rv_5min"].to_numpy())
    rend[a] = d["rendimiento"].to_numpy()
fechas = pd.DatetimeIndex(d.index)
prueba = np.flatnonzero(fechas.year >= 2018)
bolt = BaseChronosPipeline.from_pretrained(
    "amazon/chronos-bolt-small", device_map="cpu",
    dtype=torch.float32)

def zero_shot(y, origenes, contexto, lote=256):
    """Tomorrow's 0.1…0.9 quantiles with `contexto` sessions."""
    salida = []
    for k in range(0, len(origenes), lote):
        v = np.stack([y[i - contexto:i] for i in origenes[k:k + lote]])
        q, _ = bolt.predict_quantiles(
            inputs=torch.tensor(v, dtype=torch.float32),
            prediction_length=1)
        salida.append(q[:, 0, :].numpy().astype(float))
    return np.concatenate(salida)

print(sorted(log_rv), len(prueba), "test sessions")
OUT [01]Data and model
['accion', 'bono', 'cripto', 'divisa', 'indice', 'oro'] 2011 test sessions

How much history to give it

Chronos-Bolt accepts up to 2,048 sessions of context and TimesFM-2.5 up to 16,384. More context gives more information about the level and dynamics of the series, but it costs time (more patches) and can bring in history that no longer looks like the present. The cell sweeps from 16 to 1,024 sessions with one origin every 16 (126 days per asset) and compares each context with the 512 one through the paired difference.

IN [02]python
orig = prueba[::16]
contextos = [16, 32, 64, 128, 256, 512, 1024]
ecm, err = {}, {}
for a in ("indice", "cripto"):
    real = log_rv[a][orig]
    for c in contextos:
        e = (real - zero_shot(log_rv[a], orig, c)[:, 4]) ** 2
        ecm[(a, c)], err[(a, c)] = e.mean(), e
tabla = pd.Series(ecm).unstack(0).round(3)
tabla.index.name = "context"
print(tabla.to_string())
for a in ("indice", "cripto"):
    for c in (16, 64):
        d = err[(a, c)] - err[(a, 512)]
        t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
        print(f"{a}: {c} − 512 sessions {d.mean():+.4f} (t = {t:+.1f})")
OUT [02]Context length: from 16 to 1,024 sessions
         cripto  indice
context
16        0.429   0.240
32        0.412   0.220
64        0.417   0.228
128       0.402   0.223
256       0.393   0.221
512       0.387   0.219
1024      0.390   0.218
indice: 16 − 512 sessions +0.0205 (t = +1.7)
indice: 64 − 512 sessions +0.0092 (t = +1.1)
cripto: 16 − 512 sessions +0.0413 (t = +3.1)
cripto: 64 − 512 sessions +0.0296 (t = +3.1)
Fig. — One-day mean squared error of zero-shot Chronos-Bolt by context length, relative to 512 sessions (126 origins in 2018-2025 per asset)

The two assets behave differently. On the index, the error drops from 0.240 with 16 sessions to 0.220 with 32 and from there it moves between 0.218 and 0.228 with no clear order; not even 16 versus 512 is significant (t = 1.7). On crypto, the error falls steadily from 0.429 to 0.387 with 512 sessions, and the difference against 16 or 64 sessions has t = 3.1. The reason is noise: crypto RV, with twelve jumps a year on average and a large measurement error, needs more days for the model to estimate its level; on the index, with no jumps, a month and a half is enough. With 1,024 sessions neither gains anything. In practice, 256-512 sessions is a reasonable choice for daily volatility, and the test has to be repeated on each series.

Quantiles: do they cover what they promise?

Chronos-Bolt returns nine quantiles. A quantile qτq_\tau is calibrated if the realized value falls below it on a fraction τ\tau of the days. Lesson 5.7.3 showed how to measure it with trained models; the difference here is that the model has not seen the series, so its calibration is whatever it brings from pretraining. With one origin every two sessions (1,006 days per asset) and a context of 256:

IN [03]python
C = 256
tau = np.arange(1, 10) / 10
orig = prueba[::2]
zs = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
filas = {}
for a, q in zs.items():
    real = log_rv[a][orig]
    filas[a] = (real[:, None] <= q).mean(0)
frec = pd.DataFrame(filas, index=tau).round(3)
frec.index.name = "nominal"
print(frec.T.to_string())
OUT [03]Quantiles on every even day of the test
nominal    0.1    0.2    0.3    0.4    0.5    0.6    0.7    0.8    0.9
indice   0.124  0.235  0.332  0.428  0.507  0.600  0.690  0.786  0.901
cripto   0.103  0.206  0.296  0.394  0.480  0.583  0.685  0.796  0.912

The header figure plots the difference between the observed and the nominal frequency, with the ±2 standard error band (with 1,006 days, the standard error of a 50 % frequency is 0.25/1,006≈0.016\sqrt{0.25/1{,}006} \approx 0.016). On crypto the nine quantiles fall inside or close to the band: calibration is good. On the index, the lower tail is too narrow: 12.4 % of the days fall below the 0.1 quantile and 33.2 % below the 0.3 one, when they should be 10 % and 30 %. The upper tail is fine (90.1 % at 0.9). Put another way: the model does not expect days as calm as those of the synthetic index.

Overall coverage can hide problems that only show up when you condition, as lesson 5.7.3 showed. The test is split into thirds by the forecast median (a value known the day before):

IN [04]python
def cobertura(real, q):
    return np.mean((real >= q[:, 0]) & (real <= q[:, 8]))

for a, q in zs.items():
    real = log_rv[a][orig]
    tercio = pd.qcut(q[:, 4], 3, labels=["low", "medium", "high"])
    cob = {t: cobertura(real[tercio == t], q[tercio == t])
           for t in ("low", "medium", "high")}
    print(f"{a:<7} total {cobertura(real, q):.1%}  " + "  ".join(
        f"{t} {v:.1%}" for t, v in cob.items()))
OUT [04]Conditional coverage: by thirds of forecast volatility
indice  total 77.6%  low 76.2%  medium 75.8%  high 80.9%
cripto  total 80.8%  low 80.4%  medium 83.3%  high 78.8%

On the index the 10 to 90 % interval covers 76.2 % and 75.8 % in the low and medium volatility thirds, and 80.9 % in the high one: it is on normal days that it fails, consistent with the narrow lower tail. On crypto the pattern is the opposite and milder (83.3 % in the middle third, 78.8 % in the high one). With about 335 days per third, the standard error of each coverage is about 2.2 points, so only the index deviations in the low and medium thirds are clearly outside.

Recalibrating: a per-quantile conformal adjustment

If the quantiles are biased in a stable way, they can be corrected with a calibration period. The simplest version of conformalized quantile regression (Romano, Patterson and Candès, 2019) shifts each quantile by the empirical quantile of its residuals: with the residuals e=y−qτe = y - q_\tau from the calibration period, the corrected quantile is qτ+Qτ(e)q_\tau + Q_\tau(e). It is calibrated on 2018-2020 and measured on 2021-2025, without looking at the test:

IN [05]python
def pinball(real, q):
    u = real[:, None] - q
    return np.maximum(tau * u, (tau - 1) * u).mean()

cal = fechas.year[orig] <= 2020
ajustes = {}
for a, q in zs.items():
    real = log_rv[a][orig]
    # per-quantile shift: nominal frequency in 2018-2020
    res = real[cal, None] - q[cal]
    delta = np.array([np.quantile(res[:, j], tau[j])
                      for j in range(9)])
    qa = q + delta
    ajustes[a] = delta
    c0 = cobertura(real[~cal], q[~cal])
    c1 = cobertura(real[~cal], qa[~cal])
    print(f"{a:<7} 2021-2025: coverage {c0:.1%} to {c1:.1%}; pinball "
          f"{pinball(real[~cal], q[~cal]):.4f} to "
          f"{pinball(real[~cal], qa[~cal]):.4f}")
print({a: d.round(3).tolist() for a, d in ajustes.items()})
OUT [05]Recalibrate on 2018-2020 and measure on 2021-2025
indice  2021-2025: coverage 78.3% to 80.3%; pinball 0.1352 to 0.1351
cripto  2021-2025: coverage 80.9% to 81.2%; pinball 0.1659 to 0.1661
{'indice': [-0.046, -0.039, -0.046, -0.026, 0.024, 0.024, 0.023, 0.0, 0.02], 'cripto': [-0.047, -0.011, -0.003, -0.008, 0.026, 0.008, 0.015, -0.014, -0.062]}

On the index, the 2021-2025 coverage rises from 78.3 % to 80.3 %, almost the nominal value; the shifts are a few hundredths in log (−0.046 at the 0.1 quantile, which lowers the tail). On crypto, which was already calibrated, it goes from 80.9 % to 81.2 %. The pinball loss barely changes (0.1352 to 0.1351 and 0.1659 to 0.1661): the adjustment corrects the average coverage, it does not add information. It is useful when what you will use is the interval (to set a risk limit or the size of a position, lesson 4.2.2), not to improve the forecast. And it does not guarantee exact coverage on time series, because the residuals are not exchangeable: lesson 5.7.3 covers this in detail.

From quantiles of the log to the variance

For sizing or comparing with a GARCH you need tomorrow's expected variance, not the median of its log. Exponentiating the median gives the median of the variance, which is lower than its mean when the distribution has a right tail. With nine quantiles there is a direct approximation: exponentiate each quantile and average (a crude quadrature of the expectation). The measure for comparing variance forecasts is QLIKE, the one in the level README and in lesson 5.8.5: QLIKE=RV/h−log⁡(RV/h)−1\text{QLIKE} = RV/h - \log(RV/h) - 1, robust to the noise of RV as a measure of the variance.

IN [06]python
def qlike(rv, h):
    return np.mean(rv / h - np.log(rv / h) - 1)

for a, q in zs.items():
    rv = np.exp(log_rv[a][orig])
    mediana = np.exp(q[:, 4])
    media_q = np.exp(q).mean(1)       # mean of the quantiles, in levels
    print(f"{a:<7} QLIKE with exp(median) {qlike(rv, mediana):.3f}   "
          f"with the mean of the quantiles {qlike(rv, media_q):.3f}")
OUT [06]From quantiles of the log to the expected variance
indice  QLIKE with exp(median) 0.106   with the mean of the quantiles 0.103
cripto  QLIKE with exp(median) 0.397   with the mean of the quantiles 0.364

On the index the difference is small (0.106 to 0.103); on crypto, with a wider distribution, the QLIKE falls from 0.397 to 0.364. The more uncertain the forecast, the further the mean moves from the median and the more it matters not to confuse them. Lesson 5.8.5 compares these variance forecasts with a GARCH and a LightGBM.

Covariates: who accepts them and what for

A covariate is another series that helps forecast the one you care about. There are three types, depending on what is known when you forecast:

PastModelKnown futureStaticForecast
Mermaid source
flowchart LR
  P[Past] --> M[Model]
  F[Known future] --> M
  S[Static] --> M
  M --> Y[Forecast]

Past covariates are only known up to today (volume, another market series); known future covariates are known in advance for the horizon (the calendar, an auction, a scheduled release); static ones describe the series (the sector, the asset type). In the versions installed in the course environment:

ModelCovariatesHow
Chronos-Bolt 2.3.2noonly the series
Chronos-2 (Chronos2Pipeline, same library)past and known future, numerical and categoricalgroup attention, natively; weights not in the cache
TimesFM-2.5 (timesfm 3.0.2)dynamic and static, numerical and categoricalXReg: in-context linear regression plus TimesFM, with JAX
TimeGPT (nixtla SDK)future (X_df) and historical (hist_exog_list)in the paid service

XReg, the TimesFM-2.5 mechanism (added in October 2025, according to the project README), does not feed the covariates into the transformer. In "xreg + timesfm" mode it fits, for each series, a linear regression of the target on the covariates within the context itself and lets TimesFM forecast the residual; "timesfm + xreg" mode does it the other way round. It requires compiling with return_backcast=True and uses JAX (installed in the environment). Dynamic covariates are passed with the length of the context plus the horizon: their future values have to be known.

What covariate makes sense for tomorrow's volatility? The level README documents a leverage effect on the index: today's log variance depends on the negative part of yesterday's shock (γ = 0.18). Today's return is known at the close, so “the previous day's negative return” is a legitimate covariate for the next day. As a control, a placebo: Gaussian noise with no relation at all to the series. With one origin every eight sessions (252 days):

IN [07]python
import timesfm

tfm = timesfm.TimesFM_2p5_200M_torch.from_pretrained(
    "google/timesfm-2.5-200m-pytorch", torch_compile=False)
tfm.compile(timesfm.ForecastConfig(
    max_context=C, max_horizon=128, normalize_inputs=True,
    use_continuous_quantile_head=True, fix_quantile_crossing=True,
    force_flip_invariance=False, per_core_batch_size=64,
    return_backcast=True))       # XReg requires it

a = "indice"
y, r = log_rv[a], rend[a]
neg = np.r_[0.0, np.minimum(r[:-1], 0)] / r.std()   # neg[j] = r⁻ of j−1
ruido = np.random.default_rng(0).normal(size=len(y))
o = prueba[::8]
real = y[o]
contextos_tfm = [y[i - C:i] for i in o]
# forecast() pads the list it receives up to a multiple of the batch:
# pass it a copy
punto, _ = tfm.forecast(horizon=1, inputs=list(contextos_tfm))
e0 = (real - punto[:, -1]) ** 2      # with backcast, the last one is tomorrow
print(f"no covariates        MSE {e0.mean():.4f}")
for nombre, x in (("negative return", neg),
                  ("noise (placebo)", ruido)):
    p, _ = tfm.forecast_with_covariates(
        inputs=list(contextos_tfm),
        dynamic_numerical_covariates={
            nombre: [x[i - C:i + 1] for i in o]},
        xreg_mode="xreg + timesfm")
    e = (real - np.array([v[0] for v in p])) ** 2
    d = e - e0
    t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
    print(f"{nombre:<20} MSE {e.mean():.4f}  difference {d.mean():+.4f}"
          f" (t = {t:+.1f})")
OUT [07]TimesFM-2.5 with covariates (XReg)
no covariates        MSE 0.1853
negative return      MSE 0.1769  difference -0.0085 (t = -0.8)
noise (placebo)      MSE 0.1850  difference -0.0003 (t = -1.2)

A detail in the code that costs a bug: forecast() modifies the list it receives, padding it with empty series up to a multiple of the batch size. If the same list is reused in forecast_with_covariates, it no longer matches the covariates. That is why it is passed copies.

The covariate with a real effect lowers the MSE from 0.1853 to 0.1769, by 4.5 %, but with t = −0.8: it cannot be told apart from noise. The placebo leaves the error at 0.1850 (t = −1.2 on a difference of three ten-thousandths). The effect exists in the data, but it is small: the variance shock is seen through the return with noise (the correlation between the two is ρ = −0.45), tomorrow's RV has its own measurement error and the linear regression is estimated with 256 days in each context. Claiming the improvement would take many more test days; using it in production would also take an economic reason.

Chronos-2 would do the same natively. Its code, with the input format from its documentation (one dictionary per series, with past covariates the size of the context and future ones the size of the horizon):

IN [08]python
from chronos import Chronos2Pipeline

chronos2 = Chronos2Pipeline.from_pretrained("amazon/chronos-2",
                                            device_map="cpu")
entradas = [{
    "target": y[i - C:i],
    "past_covariates": {"neg": neg[i - C:i]},
    "future_covariates": {"neg": neg[i:i + 1]},   # known at the close
} for i in o]
cuantiles, media = chronos2.predict_quantiles(
    entradas, prediction_length=1, quantile_levels=[0.1, 0.5, 0.9])

And TimeGPT, with a DataFrame of the future covariates (X_df), according to its documentation:

IN [09]python
import os
from nixtla import NixtlaClient

cliente = NixtlaClient(api_key=os.environ["NIXTLA_API_KEY"])
historia = pd.DataFrame({"unique_id": "indice", "ds": fechas[:-1],
                         "y": y[:-1], "neg": neg[:-1]})
futuro = pd.DataFrame({"unique_id": "indice", "ds": fechas[-1:],
                       "neg": neg[-1:]})
prevision = cliente.forecast(df=historia, X_df=futuro, h=1, freq="B",
                             level=[80])

Fine-tuning: when it pays off

Fine-tuning trains the model a little further on your own data. There are three degrees: full fine-tuning (all the weights), adapters (LoRA: small matrices added to the layers, with the rest frozen) and the head (only the output layer). Full fine-tuning of Chronos-Bolt-small on CPU takes several seconds per training step with batches of 32 windows on this machine, too much for the lesson; its code, with the quantile loss the model itself computes when it receives the target, would be this:

IN [10]python
bolt.model.train()
opt = torch.optim.AdamW(bolt.model.parameters(), lr=1e-5)
for paso in range(1000):
    contexto, objetivo = lote_de_ventanas()   # (32, C) and (32, 1)
    salida = bolt.model(context=contexto, target=objetivo)
    opt.zero_grad()
    salida.loss.backward()
    opt.step()
bolt.model.eval()

Chronos-2 ships fine-tuning in the library (Chronos2Pipeline.fit, full or with LoRA, which returns a tuned copy), in the form shown in its quickstart notebook; TimeGPT offers it in the service with finetune_steps:

IN [11]python
ajustado = chronos2.fit(inputs=entradas_entrenamiento,
                        prediction_length=1, num_steps=1000,
                        learning_rate=1e-5, batch_size=32)
prevision = cliente.forecast(df=historia, h=1, freq="B",
                             finetune_steps=10)          # TimeGPT

What does fit in the lesson is tuning the head. In Chronos-Bolt it is a residual block (ResidualBlock) that turns the decoder output into the nine quantiles of the 64 steps. With the rest of the model frozen, the decoder output for each window is computed only once and training only touches the head: seconds of CPU. The tuning data are windows from the six assets with the target day between 2009 and 2016; validation, 2017; the test, the same as in the previous sections (2018-2025). No training window reaches the test.

IN [12]python
del tfm
m = bolt.model
cabeza = m.output_patch_embedding
print(cabeza.__class__.__name__,
      f"{sum(p.numel() for p in cabeza.parameters()) / 1e6:.2f} M of",
      f"{sum(p.numel() for p in m.parameters()) / 1e6:.1f} M",
      "parameters")
rng = np.random.default_rng(0)

def ventanas(desde, hasta, n):
    """n windows per asset whose target day falls between those years."""
    X, T = [], []
    for a, y in log_rv.items():
        ok = np.flatnonzero((fechas.year >= desde)
                            & (fechas.year <= hasta))
        for i in rng.choice(ok[ok >= C], n, replace=False):
            X.append(y[i - C:i])
            T.append(y[i])
    return (torch.tensor(np.stack(X), dtype=torch.float32),
            torch.tensor(T, dtype=torch.float32))

@torch.no_grad()
def ocultos(X):
    """Output of the frozen decoder and scale of each window."""
    H, L = [], []
    for k in range(0, len(X), 256):
        h, (loc, esc), emb, mask = m.encode(X[k:k + 256])
        H.append(m.decode(emb, mask, h))
        L.append(torch.cat([loc, esc], 1))
    return torch.cat(H), torch.cat(L)

X_tr, T_tr = ventanas(2009, 2016, 200)
X_va, T_va = ventanas(2017, 2017, 50)
H_tr, L_tr = ocultos(X_tr)
H_va, L_va = ocultos(X_va)
print("training", tuple(H_tr.shape),
      "validation", tuple(H_va.shape))
OUT [12]Fine-tuning Chronos-Bolt's head: data
ResidualBlock 2.53 M of 47.7 M parameters
training (1200, 1, 512) validation (300, 1, 512)

The loss is Chronos-Bolt's own (the pinball loss of the nine quantiles, summed and multiplied by 2) on the target standardized with the mean and standard deviation of the context, as the model does. It trains for 200 steps with Adam and checks validation every 50:

IN [13]python
import copy

tau_t = torch.tensor(m.chronos_config.quantiles)

def perdida(H, L, T):
    """Chronos-Bolt quantile loss at the first step,
    with the target standardized like the context."""
    q = cabeza(H).view(len(H), 9, -1)[:, :, 0]
    z = (T[:, None] - L[:, :1]) / L[:, 1:]
    return (2 * torch.abs((z - q) * ((z <= q).float() - tau_t))
            ).sum(1).mean()

original = copy.deepcopy(cabeza.state_dict())
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-4)
g = torch.Generator().manual_seed(0)
for paso in range(201):
    if paso % 50 == 0:
        with torch.no_grad():
            print(f"step {paso:3d}  training "
                  f"{perdida(H_tr, L_tr, T_tr).item():.3f}  validation "
                  f"{perdida(H_va, L_va, T_va).item():.3f}")
    lote = torch.randperm(len(H_tr), generator=g)[:256]
    opt.zero_grad()
    perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
    opt.step()
OUT [13]Fine-tuning the head: training
step   0  training 4.430  validation 4.717
step  50  training 4.328  validation 4.681
step 100  training 4.269  validation 4.682
step 150  training 4.223  validation 4.679
step 200  training 4.179  validation 4.684
Fig. — Fine-tuning of Chronos-Bolt-small's output head (2.53 M of 47.7 M parameters) on the six assets' log RV: training (2009-2016) and validation (2017) loss

The training loss keeps falling (4.430 to 4.179), but the validation loss only improves during the first 50 steps (4.717 to 4.681) and then stays flat: from there on, the head memorizes the 1,200 windows. The number of steps was set in advance; choosing it by looking at the test would be the selection from lesson 5.5.4. The test decides:

IN [14]python
ft = {a: zero_shot(log_rv[a], orig, C) for a in ("indice", "cripto")}
for a in ft:
    real = log_rv[a][orig]
    u0 = real[:, None] - zs[a]
    u1 = real[:, None] - ft[a]
    p0 = np.maximum(tau * u0, (tau - 1) * u0).mean(1)
    p1 = np.maximum(tau * u1, (tau - 1) * u1).mean(1)
    d = p1 - p0
    t = d.mean() / d.std(ddof=1) * np.sqrt(len(d))
    print(f"{a:<7} pinball {p0.mean():.4f} to {p1.mean():.4f} "
          f"(t = {t:+.1f}); MSE {np.mean(u0[:, 4] ** 2):.4f} to "
          f"{np.mean(u1[:, 4] ** 2):.4f}; coverage "
          f"{cobertura(real, zs[a]):.1%} to "
          f"{cobertura(real, ft[a]):.1%}")
cabeza.load_state_dict(original);      # restores the published model
OUT [14]Fine-tuned versus zero-shot in 2018-2025
indice  pinball 0.1356 to 0.1369 (t = +1.4); MSE 0.1953 to 0.1979; coverage 77.6% to 76.4%
cripto  pinball 0.1660 to 0.1675 (t = +1.4); MSE 0.3713 to 0.3671; coverage 80.8% to 79.3%

The fine-tuning slightly worsens the pinball loss on both assets (0.1356 to 0.1369 on the index, 0.1660 to 0.1675 on crypto, t = 1.4 in both) and lowers the coverage somewhat. The index MSE goes up and the crypto one goes down (0.3713 to 0.3671): mixed signals within the noise. The conclusion is the honest one: on this series, the pretrained model was already extracting almost all the information, and 1,200 windows are not enough to teach it something it does not know without making it forget something else.

Fine-tuning usually pays off if…It usually does not pay off if…
the series looks little like the corpus (unusual units, frequency or dynamics)the series has a simple, common structure (persistence, like volatility)
there is a lot of in-house data (thousands of series or years of high frequency)there are few series and few years
the zero-shot forecast is clearly biased and the bias is stablethe zero-shot forecast is already close to the best estimated model
it is validated with a later test fixed in advancesteps, rate or layers are chosen by looking at the test

Exercises

  1. Repeat the coverage by thirds of forecast volatility after the conformal adjustment, only on 2021-2025. Does the adjustment fix the index thirds that were failing, or only the overall coverage?
Solution
IN [15]python
for a, q in zs.items():
    real, qa = log_rv[a][orig][~cal], (q + ajustes[a])[~cal]
    q0 = q[~cal]
    tercio = pd.qcut(q0[:, 4], 3, labels=["low", "medium", "high"])
    fila = []
    for t in ("low", "medium", "high"):
        s = tercio == t
        fila.append(f"{t} {cobertura(real[s], q0[s]):.1%} to "
                    f"{cobertura(real[s], qa[s]):.1%}")
    print(f"{a:<7}", "  ".join(fila))
OUT [15]Coverage by thirds after recalibrating
indice  low 75.2% to 76.7%  medium 76.1% to 78.5%  high 83.7% to 85.6%
cripto  low 82.9% to 82.4%  medium 80.4% to 80.9%  high 79.4% to 80.4%

The adjustment raises the coverage in all three index thirds (from 75.2 % to 76.7 % in the low one, from 76.1 % to 78.5 % in the medium one and from 83.7 % to 85.6 % in the high one), but it does not equalize them: the shift is the same for every day, so it moves the bias without correcting its shape. After the adjustment, calm days are still below 80 % and volatile ones above. Correcting conditional coverage requires an adjustment that depends on the bucket (calibrating by thirds, with three times less data in each) or a better model, not a global shift. On crypto, which was already fine, everything stays between 80 % and 83 %.

  1. Repeat the fine-tuning of the head with a learning rate ten times higher (lr=1e-3). What happens to the validation loss?
Solution
IN [16]python
opt = torch.optim.Adam(cabeza.parameters(), lr=1e-3)
g = torch.Generator().manual_seed(0)
for paso in range(201):
    if paso % 50 == 0:
        with torch.no_grad():
            print(f"step {paso:3d}  training "
                  f"{perdida(H_tr, L_tr, T_tr).item():.3f}  validation "
                  f"{perdida(H_va, L_va, T_va).item():.3f}")
    lote = torch.randperm(len(H_tr), generator=g)[:256]
    opt.zero_grad()
    perdida(H_tr[lote], L_tr[lote], T_tr[lote]).backward()
    opt.step()
cabeza.load_state_dict(original);
OUT [16]Fine-tuning with lr = 1e-3
step   0  training 4.430  validation 4.717
step  50  training 4.089  validation 4.699
step 100  training 3.849  validation 4.819
step 150  training 3.487  validation 4.949
step 200  training 3.160  validation 5.034

With a rate ten times higher, validation improves a little in the first 50 steps (4.717 to 4.699) and then worsens steadily up to 5.034, above the starting point, while training drops from 4.430 to 3.160. It is overfitting in its clearest form: the head learns the 1,200 windows by heart and loses what it knew. Without a validation set separated in time, the training loss would have said the fine-tuning was going very well. At the end the cell restores the published head.

  1. You want to forecast tomorrow's volatility of a European stock with a covariate model, and you propose two: today's S&P 500 close (the New York stock exchange closes at 22:00, Madrid time) and tomorrow's VIX at the European open. You forecast at 17:35, after the European close. Which ones are legitimate and of what type?
Solution

Neither, as stated. At 17:35 the S&P 500 is still open: today's close is not known, so as a covariate for “today” it is a leak; what is legitimate is yesterday's close (past) or its value at 17:35 (past, timestamped at that time). Tomorrow's VIX at the European open is in the future and not known in advance: it is not a known future covariate, but what you want to forecast under another name. Genuinely known future covariates would be the company's earnings calendar or scheduled macro releases (ECB, CPI), which are known days ahead. Any model that accepts covariates will treat them however it is told; the responsibility for timestamping them lies with whoever prepares them (lesson 5.2.2).

Each answer is folded under its question

Self-assessment

  1. With Chronos-Bolt on the index log RV, going from 16 to 512 sessions of context lowers the MSE by 9 % with t = 1.7, and from 64 to 512 there is no significant difference. On crypto, from 64 to 512 the improvement has t = 3.1. How do you choose the context?

    • AAlways the maximum the model allows
    • BAlways the minimum, because it is faster
    • CBy measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy
    • DWith the length of the longest season
    Show answer

    Answer C. By measuring out of sample on each series: the useful history depends on the memory of the series and on its noise; here, about 256-512 sessions are enough, and more context costs time without gaining accuracy

    The crypto series is noisier: the model needs more days to estimate its level. Past a certain point, old history adds nothing and the cost of attention keeps growing.

  2. On 12.4 % of the days the index log RV falls below the forecast 0.1 quantile, and on 90.1 % below the 0.9 one. What is wrong with the forecast distribution?

    • AThe lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated
    • BIt is perfectly calibrated
    • CThe upper tail is too narrow
    • DThe model is biased upward in every quantile
    Show answer

    Answer A. The lower tail is too narrow (calm days are calmer than forecast) and the upper tail is well calibrated

    With 1,006 days, the standard error of a 10 % frequency is 0.9 points: 12.4 % is more than two standard errors away. At 0.9 the observed frequency is 90.1 %, right at the nominal value.

  3. A per-quantile conformal adjustment, estimated on 2018-2020, takes the 10-90 % coverage of the index in 2021-2025 from 78.3 % to 80.3 %, but the pinball loss goes from 0.1352 to 0.1351. What is gained?

    • AA much better forecast
    • BNothing: coverage gets worse
    • CA more accurate median
    • DIntervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves
    Show answer

    Answer D. Intervals with the promised coverage, useful for sizing or setting limits; the distribution barely changes, so the score hardly moves

    The adjustment only shifts each quantile by a few hundredths in log. It corrects the average coverage bias, not the model's information. Lesson 5.7.3 explains why, on time series, it does not guarantee exact coverage.

  4. With TimesFM-2.5 and XReg, the covariate “yesterday's negative return” lowers the index MSE from 0.1853 to 0.1769 with t = −0.8, and a noise placebo barely changes it. What do you conclude?

    • AThat the covariate improves the model by 4.5 % and you should use it
    • BThat with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily
    • CThat XReg does not work
    • DThat the placebo shows the model is overfitted
    Show answer

    Answer B. That with 252 days it cannot be told apart from noise: the leverage effect exists in the data (γ = 0.18), but it is small next to the RV error and the in-context linear regression estimates it very noisily

    An improvement with t = −0.8 is not evidence. The placebo serves the opposite purpose: checking that adding a covariate with no information neither worsens things much nor often “improves” them by chance. Claiming the improvement would require a much larger sample.

  5. Fine-tuning Chronos-Bolt's head lowers the training loss from 4.430 to 4.179, the validation loss from 4.717 to 4.684 and, in 2018-2025, the index pinball loss goes from 0.1356 to 0.1369. What happened?

    • AThe fine-tuning worked
    • BThere is a mistake in the evaluation
    • CThe model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series
    • DIt needed more training steps
    Show answer

    Answer C. The model memorizes the training windows: validation stalls at 50 steps and out of sample the result gets slightly worse. The pretrained model was already extracting almost all the information in this series

    It is the classic overfitting pattern, with a model of 2.5 million trainable parameters and 1,200 windows. Fine-tuning helps when the series looks little like the pretraining corpus or there is a lot of in-house data; neither is the case here.

Key points

  • Context length is measured: on the index it makes no difference beyond 32 sessions (16 versus 512: t = 1.7); on crypto, which is noisier, it improves up to 256-512 (64 versus 512: t = 3.1).
  • Zero-shot quantiles are almost calibrated: on the index the lower tail is narrow (12.4 % below the 0.1 quantile) and on high-volatility days the coverage changes. A per-quantile conformal adjustment restores the nominal coverage (78.3 % to 80.3 %) without changing the pinball loss.
  • The expected variance is not the exponential of the median of the log: the mean of the quantiles in levels lowers the QLIKE (crypto: 0.397 to 0.364).
  • Covariates in the installed versions: Chronos-Bolt no; Chronos-2 yes (past, future and categorical; weights not cached); TimesFM-2.5 yes, with XReg (in-context linear regression and TimesFM on the residual). On vol5, neither the covariate with a real effect nor the placebo changes anything significantly.
  • Fine-tuning Chronos-Bolt's head fits in seconds of CPU, but it overfits: validation stalls and out of sample the pinball loss worsens by 1 % (t = 1.4).

Resources