Time series foundation models: pretrain once, forecast without training.

From one model per series to one model for all

Up to here, every model in the course has been estimated with the data of its problem: a GARCH with the returns of one asset (lesson 2.1.4), a HAR or a network with the volatility of six assets (lesson 5.7.2). A time series foundation model reverses the order. Someone trains it once, on a huge corpus of series from very different domains (energy, traffic, sales, weather, web visits, finance, synthetic series), publishes the weights, and anyone can then use them on a series the model has never seen, without retraining. That is called zero-shot forecasting: the new series only enters as context, in the forward pass, and the weights do not change.

flowchart TB
  A[Massive corpus] --> B[Pretraining]
  B --> C[Published weights]
  C --> D[Context]
  D --> E[Zero-shot]
  E --> F[Fine-tuning]

The idea comes from language models: a transformer that has seen enough texts completes a new one without anyone training it for that text. In time series, the bet is that there are regularities common to almost all of them (trends, seasonalities, persistence, level shifts, structured noise) and that a large model can learn them and recognize them in the context it receives. The corpus figures say it all:

Lesson 5.8.2 covers the families with their architectures, sizes and licenses. This lesson sticks to what they have in common: how they turn a series into something a transformer can read and what you get when you use them without training.

The series: the index's realized variance

The test bench is the one from lessons 5.7.2 and 5.7.3: the indice asset of vol5_diario.parquet, 4,529 sessions from 2008 to 2025, with its realized variance for each day (rv_5min, in squared return). Its logarithm is forecast, which is almost Gaussian and is the scale on which the generator works.

import numpy as np
import pandas as pd

vol = pd.read_parquet("vol5_diario.parquet")
ind = vol[vol["activo"] == "indice"].set_index("fecha")
log_rv = np.log(ind["rv_5min"].to_numpy())
vol_anual = np.sqrt(252 * ind["rv_5min"].to_numpy())
fechas = pd.DatetimeIndex(ind.index)
print(f"{len(log_rv)} sessions, {fechas[0].date()} to "
      f"{fechas[-1].date()}")
print(f"log RV: mean {log_rv.mean():.2f}, std. {log_rv.std():.2f}")
print(f"annual vol.: median {np.median(vol_anual):.1%}, "
      f"p99 {np.quantile(vol_anual, 0.99):.1%}")

4529 sessions, 2008-01-02 to 2025-12-31
log RV: mean -9.75, std. 0.64
annual vol.: median 12.1%, p99 25.6%

The log RV sits around −9.75 with a standard deviation of 0.64: a series far from zero and with little relative variation. That detail matters right away.

From numbers to tokens: the Chronos recipe

A language transformer works with a finite vocabulary. Chronos (Ansari et al., 2024) does the minimum to make a series look like one: it scales the context by dividing it by the mean of its absolute values and quantizes the result into uniform bins. With a context $x_1, \dots, x_C$:

According to the paper and the models' config.json, the bins are 4,093 equally spaced centers between −15 and +15 (a vocabulary of 4,096 with the special padding and end tokens), the context is 512 observations and the horizon, 64. The model (a T5 with 20 to 710 million parameters) is trained with cross-entropy to predict the next token, like a language model, and the probabilistic forecast comes from sampling token trajectories and undoing the scaling. Each bin measures $30/4{,}092 \approx 0.0073$ in scaled units.

The chronos-forecasting library includes the tokenizer, which needs no weights: the configuration of amazon/chronos-t5-small is enough. The cell reproduces it by hand and applies it to two versions of the same information, the annual volatility and the logarithm of the variance:

import torch
from chronos import ChronosConfig

# The configuration of amazon/chronos-t5-small (the repo's config.json)
cfg = ChronosConfig(
    tokenizer_class="MeanScaleUniformBins",
    tokenizer_kwargs={"low_limit": -15.0, "high_limit": 15.0},
    n_tokens=4096, n_special_tokens=2, pad_token_id=0,
    eos_token_id=1, use_eos_token=True, model_type="seq2seq",
    context_length=512, prediction_length=64, num_samples=20,
    temperature=1.0, top_k=50, top_p=1.0)
tok = cfg.create_tokenizer()
centros = np.linspace(-15, 15, 4096 - 2 - 1)

def tokens_a_mano(x):
    """Scale by the mean of |x| and round to the nearest center.
    Ids 0 and 1 are PAD and EOS; the bins start at 3."""
    escala = np.abs(x).mean()
    ids = np.abs(x[:, None] / escala - centros).argmin(1) + 3
    return ids, centros[ids - 3] * escala

for nombre, x in (("annual vol", vol_anual[-512:]),
                  ("log RV", log_rv[-512:])):
    ids, x_rec = tokens_a_mano(x)
    ids_chr, _, escala = tok.context_input_transform(
        torch.tensor(x[None, :], dtype=torch.float32))
    igual = (ids == ids_chr[0, :512].numpy()).mean()
    print(f"{nombre:<10} scale {escala.item():8.4f}  "
          f"distinct tokens {len(np.unique(ids)):4d}  "
          f"max. error {np.abs(x - x_rec).max():.4f}  "
          f"same as chronos {igual:.0%}")

annual vol scale   0.1405  distinct tokens  168  max. error 0.0005  same as chronos 100%
log RV     scale   9.5590  distinct tokens   46  max. error 0.0350  same as chronos 100%

The library's tokenizer and the cell's agree. The comparison between the two series shows the weakness of mean scaling. The annual volatility (positive, with a mean of 0.14) uses 168 distinct bins and the rounding costs at most 0.0005, five hundredths of a volatility point. The logarithm of the variance, on the other hand, is always around −9.6: divided by its mean absolute value it is almost always close to −1, it only uses 46 bins and the rounding reaches 0.035 in log, 3.6% of the variance. Mean scaling does not subtract the mean, so a series far from zero loses resolution. Exercise 2 measures it with the standardized series.

Patches: what Chronos-Bolt and TimesFM do

The second generation drops the one-token-per-value approach. Chronos-Bolt standardizes the context (subtracts the mean and divides by the standard deviation, what the code calls InstanceNorm), cuts it into patches of 16 observations and projects each patch to a vector with a small network; at the end it adds a regression token. The T5 encoder reads those vectors and the decoder produces at once the 0.1…0.9 quantiles of the next 64 steps (direct multi-step forecasting), without sampling. TimesFM does something similar with patches of 32 and a decoder-only transformer (lesson 5.8.2). It is the same idea as PatchTST in lesson 5.7.2, at pretraining scale.

flowchart LR
  S[Context] --> M[Mean scaling]
  M --> Q[One token per value]
  S --> N[Standardize]
  N --> P[Patches of 16]
  Q --> T[Transformer]
  P --> T

The consequence is about cost. Attention compares each token with all the others, so its cost grows with the square of the number of tokens:

The Chronos-Bolt-small weights are in the ml environment's cache, pinned by commit (environment README). The cell loads them without network access and checks the hand-made patches against the model's:

from chronos import BaseChronosPipeline

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
bolt = BaseChronosPipeline.from_pretrained(
    "amazon/chronos-bolt-small", device_map="cpu",
    dtype=torch.float32)
cc = bolt.model.chronos_config
print(f"max. context {cc.context_length}, "
      f"patch {cc.input_patch_size}, "
      f"horizon {cc.prediction_length}, quantiles {cc.quantiles}")

x = torch.tensor(log_rv[-512:], dtype=torch.float32)[None, :]
z = (x - x.mean()) / x.std(correction=0)       # standardize
parches = z.unfold(-1, 16, 16)                    # (1, 32, 16)
z_bolt, _ = bolt.model.instance_norm(x)
print("patches:", tuple(parches.shape),
      "| equal to the model's:",
      torch.allclose(parches, bolt.model.patch(z_bolt), atol=1e-5))
print(f"standardized values: from {z.min():.2f} to {z.max():.2f}")

max. context 2048, patch 16, horizon 64, quantiles [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
patches: (1, 32, 16) | equal to the model's: True
standardized values: from -2.45 to 2.57

Maximum context of 2,048 observations, patches of 16, a horizon of 64 steps and nine quantiles. Standardized, the log RV spans −2.45 to 2.57: the resolution problem of mean scaling disappears, because here there is no rounding (each patch enters with its real values) and the mean is subtracted first.

A first forecast

With the model loaded, forecasting is one call: it is given the context (the last 512 sessions) and the horizon. The chosen origin is Friday, March 6, 2020, the day before one of the shocks planted in the data (the README documents +1.8 in log-variance on March 9). The output is converted to annual volatility to make it easier to read:

i0 = fechas.get_loc(pd.Timestamp("2020-03-06")) + 1
q, _ = bolt.predict_quantiles(
    inputs=torch.tensor(log_rv[i0 - 512:i0], dtype=torch.float32),
    prediction_length=5, quantile_levels=[0.1, 0.5, 0.9])
a_vol = lambda l: np.sqrt(252 * np.exp(l))       # log RV to annual vol.
tabla = pd.DataFrame(
    a_vol(q[0].numpy().astype(float)) * 100,
    index=fechas[i0:i0 + 5].date, columns=["q10", "q50", "q90"])
tabla["realized"] = vol_anual[i0:i0 + 5] * 100
tabla.round(1)

             q10   q50   q90  realized
2020-03-09  12.0  15.5  20.2      52.8
2020-03-10  12.0  16.0  21.4      28.6
2020-03-11  12.2  16.5  22.1      31.0
2020-03-12  12.4  16.9  22.6      24.9
2020-03-13  12.5  17.1  22.9      23.3

The model forecasts a median volatility of 15.5% for Monday, with a 10 to 90% band from 12.0 to 20.2%, and a slight rise over the following days. Monday's realized volatility is 52.8%: the shock was not in the past of the series and nobody could have forecast it from it. From Tuesday on the realized value (28.6%, 31.0%…) stays above the band, because this forecast is five days ahead from Friday and is not updated. The figure at the top shows what happens when the forecast is redone each day with the new data point: the band jumps after the shock and chases the realized value.

One-day zero-shot with a rolling origin

A single forecast evaluates nothing. The honest evaluation is that of lesson 5.4.3 without the retraining: for each session from 2018 to 2025, the next day is forecast with the previous 512 sessions and compared with what happened. The model is never fitted, so there is no possible leak through training; the only rule is that the context ends on the day before. The baselines are the naive forecast (yesterday's log RV) and the mean of the last 22 days.

prueba = np.flatnonzero(fechas.year >= 2018)
C = 512

def zero_shot(serie, origenes, contexto=C, lote=256):
    """Quantiles 0.1…0.9 for tomorrow at each origin (no retraining)."""
    salida = []
    for k in range(0, len(origenes), lote):
        ventanas = np.stack([serie[i - contexto:i]
                             for i in origenes[k:k + lote]])
        q, _ = bolt.predict_quantiles(
            inputs=torch.tensor(ventanas, dtype=torch.float32),
            prediction_length=1)
        salida.append(q[:, 0, :].numpy().astype(float))
    return np.concatenate(salida)

q_bolt = zero_shot(log_rv, prueba)
real = log_rv[prueba]
previsiones = {
    "naive (yesterday)": log_rv[prueba - 1],
    "22-day mean": np.array([log_rv[i - 22:i].mean()
                             for i in prueba]),
    "Chronos-Bolt (median)": q_bolt[:, 4],
}

def metricas(f):
    e2 = (real - f) ** 2
    return {"MSE": e2.mean(), "R²": np.corrcoef(real, f)[0, 1] ** 2}

tabla = pd.DataFrame({k: metricas(f) for k, f in previsiones.items()})
tabla.T.round(3)

                         MSE     R²
naive (yesterday)      0.233  0.503
22-day mean            0.237  0.418
Chronos-Bolt (median)  0.181  0.551

Chronos-Bolt, without ever having seen the series, gives a mean squared error (MSE) of 0.181 against 0.233 for the naive forecast and 0.237 for the 22-day mean, and an R² of 0.551 (the R² is that of the regression of the log RV on the forecast, the same measure as in the README). To put it in context: the README publishes, for the same period, an R² of 0.57 for a HAR re-estimated every January and of 0.61 for the oracle, the exact expectation of the variance with the information of the day before. The foundation model sits between the naive baseline and the HAR, closer to the HAR. Whether that is enough is decided by the lab of lesson 5.8.5, with GARCH and LightGBM in the same test.

The forecast being compared is the median (the 0.5 quantile), not the mean. In log it makes no difference because the distribution is almost symmetric; in variance it would (the exponential of the median of the log understates the mean of the variance), and lesson 5.8.3 deals with it.

Is the improvement larger than the noise?

The error differences are read with the day-by-day paired difference, as in lesson 5.4.3: the hard days are hard for every model, and the difference removes that common noise.

def pareada(a, b):
    """Mean difference of squared errors (a − b) and its t."""
    d = (real - previsiones[a]) ** 2 - (real - previsiones[b]) ** 2
    return d.mean(), d.mean() / (d.std(ddof=1) / np.sqrt(len(d)))

for ref in ("naive (yesterday)", "22-day mean"):
    dif, t = pareada("Chronos-Bolt (median)", ref)
    print(f"Bolt − {ref:<17} {dif:+.4f}  t = {t:+.1f}")
dentro = (real >= q_bolt[:, 0]) & (real <= q_bolt[:, 8])
print(f"coverage of the 10-90% interval: {dentro.mean():.1%} "
      f"({len(real)} days)")

Bolt − naive (yesterday) -0.0516  t = -9.8
Bolt − 22-day mean       -0.0557  t = -10.4
coverage of the 10-90% interval: 76.4% (2011 days)

Both improvements are far above the noise: t = −9.8 against the naive forecast and −10.4 against the 22-day mean. The test days overlap in the context but not in the error being measured (each day is forecast once), so the standard error of the difference is reasonable; if the errors had autocorrelation it would have to be corrected (Newey-West), but with a t of −10 a very strong autocorrelation would be needed to change the conclusion. The 10 to 90% interval covers 76.4% of the days, not 80%: the quantiles are somewhat narrow. Lesson 5.8.3 measures the calibration quantile by quantile and corrects it.

The result holds year by year:

anio = fechas.year[prueba]
por_anio = pd.DataFrame({k: (real - f) ** 2 for k, f in
                         previsiones.items()}).groupby(anio).mean()
por_anio.round(3)

       naive (yesterday)  22-day mean  Chronos-Bolt (median)
fecha
2018               0.202        0.163                  0.147
2019               0.229        0.194                  0.166
2020               0.288        0.285                  0.224
2021               0.214        0.301                  0.181
2022               0.247        0.261                  0.197
2023               0.201        0.216                  0.159
2024               0.233        0.224                  0.182
2025               0.249        0.251                  0.194

Chronos-Bolt has the lowest error in all eight years. The worst is 2020 for all of them (the year of the March shocks). The 22-day mean suffers more in 2021: for weeks it drags along the days of extreme volatility of the previous year, which the foundation model discounts sooner.

What zero-shot does not do

A good zero-shot volatility forecast says nothing about other things, and it is worth being clear about that before going on:

Lesson 5.8.2 compares the families (Chronos, Chronos-Bolt, Chronos-2, TimesFM-2.5, Moirai-2.0, TiRex, TimeGPT and Kronos) and puts TimesFM-2.5 next to Chronos-Bolt; 5.8.3 goes into the practical decisions (context length, quantiles, covariates and fine-tuning).

Exercises

1. Repeat the zero-shot evaluation with the cripto asset (use one origin out of every four to make it fast). Does Chronos-Bolt beat both baselines? Compare its R² with that of the README's HAR for that asset (0.40).

vc = vol[vol["activo"] == "cripto"].set_index("fecha")
log_rv_c = np.log(vc["rv_5min"].to_numpy())
orig = prueba[::4]
q_c = zero_shot(log_rv_c, orig)
real_c = log_rv_c[orig]
for nombre, f in (("naive", log_rv_c[orig - 1]),
                  ("mean 22", np.array([log_rv_c[i - 22:i].mean()
                                        for i in orig])),
                  ("Bolt", q_c[:, 4])):
    print(f"{nombre:<9} MSE {np.mean((real_c - f) ** 2):.3f}  "
          f"R² {np.corrcoef(real_c, f)[0, 1] ** 2:.3f}")

naive     MSE 0.652  R² 0.252
mean 22   MSE 0.427  R² 0.328
Bolt      MSE 0.395  R² 0.393

2. Standardize the log RV context (subtract the mean and divide by the standard deviation) before tokenizing it with the Chronos recipe. How many bins does it use now? What does it tell you about the mean scaling of the original Chronos?

x = log_rv[-512:]
z = (x - x.mean()) / x.std()
for nombre, serie in (("raw log RV", x), ("z-scored log RV", z)):
    ids, rec = tokens_a_mano(serie)
    print(f"{nombre:<18} distinct tokens {len(np.unique(ids)):4d}")

raw log RV         distinct tokens   46
z-scored log RV    distinct tokens  353

3. A paper evaluates in 2025 a foundation model on daily returns of S&P 500 stocks from 2015 to 2023, and the model was pretrained on a corpus published in 2024 that includes public financial series. What problem does the evaluation have? Why do the course data not have it?