--- title: "5.8.4 Evidence on foundation models in finance" description: "What 2025-2026 studies say about Chronos, TimesFM and Moirai on returns and volatility, plus an in-house test with Diebold-Mariano." url: "https://pythonparatrading.com/en/python-trading-course/level-5/foundation-models-finance-evidence/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/foundation-models-finance-evidence.png" --- # 5.8.4 · Foundation models in finance: what the evidence says > A foundation model that wins a ranking need not beat forecasting zero. This lesson reads two recent studies on returns closely, summarizes the evidence on volatility and repeats the test with Chronos-Bolt and TimesFM-2.5 on a series whose return is unpredictable by construction, with Diebold-Mariano and a multiple-testing correction. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.8 · Time series foundation models - Duration: 60 min - Prerequisites: [5.8.1 · Time series foundation models: pretrain once, forecast without training](https://pythonparatrading.com/en/python-trading-course/level-5/what-are-time-series-foundation-models/), [5.4.4 · Backtesting ML signals with costs and Deflated Sharpe](https://pythonparatrading.com/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe/), [5.1.3 · Economic metrics for trading models: beyond accuracy](https://pythonparatrading.com/en/python-trading-course/level-5/economic-metrics-for-trading-models/) ## By the end of this lesson - Read a study of foundation models in finance, separating ranking, improvement over a naive baseline, significance and economic value. - Summarize with their exact figures the 2026 benchmark (arXiv:2606.27100) and the study by Rahimikia, Ni and Wang (arXiv:2511.18578), and what their own authors say about their limits. - Implement the Diebold-Mariano test with the Harvey, Leybourne and Newbold correction and correct a family of tests with Holm. - Check on data with a known answer that Chronos-Bolt and TimesFM-2.5 do not predict an unpredictable return, and that the same model does capture structure in volatility. - Recognize the illusion of evaluating price forecasts in levels. ## The hard question: tomorrow's return Lessons 5.8.1 to 5.8.3 explained what a time series foundation model is, what families there are (Chronos, TimesFM, Moirai, Kronos…) and how to use them without retraining. This lesson asks something else: **what is known about their performance in finance**, and specifically on the task that matters most to anyone who trades, forecasting returns. It is the toughest possible test for a pretrained model. Daily returns have an extremely low signal-to-noise ratio, fat tails, regime changes and almost no autocorrelation (lesson 1.6.2). A model that has learned patterns from thousands of traffic, energy or sales series finds little to recognize in them, and whatever little there is gets arbitraged away by other participants. The honest question is not “which model wins?” but **“does any of them beat forecasting zero, with what statistical confidence and with what value after costs?”**. A claim that a model “predicts” has to travel this whole chain: ```mermaid flowchart TB A[Forecast] --> B[Loss and baseline] B --> C[Robust test] C --> D[Multiple testing] D --> E[Costs and capacity] E --> F[Conclusion] ``` Each link has its trap. The **baseline** must be the naive one (zero return or historical mean), not another complex model. The **loss** decides what gets rewarded: squared error punishes extreme days, absolute error targets the median. The **test** has to account for the fact that the losses of consecutive days or overlapping horizons are not independent. With many models, assets and variants, something comes out significant by chance (lesson 5.4.4). And a slightly better forecast may not pay the **costs** of trading it (lesson 5.1.3). ## The 2026 benchmark: rankings versus the random walk The syllabus cites a benchmark posted on arXiv on June 25, 2026: “Pretrained Time-Series Foundation Models for Financial Return Forecasting”, by Miquel Noguer i Alonso and Rodolfo Pereira Franklin (Artificial Intelligence Finance Institute), arXiv:2606.27100. What follows comes from the abstract page and the PDF of version 1, read on 2026-10-23. The design: | Element | Value in the paper | |---|---| | Assets | AAPL, AMZN, GOOG, JPM and META; daily adjusted closes from Yahoo Finance (2014-09-15 to 2026-02-15) | | Targets | Simple and log return (10 tasks: 5 stocks × 2 targets) | | Pretrained models (no retraining) | TimeGPT, TimeGPT-LH, TimesFM-2.5, Moirai-2.0, Chronos (v1) and Chronos-2 | | Models trained per stock | NBEATS, NHITS, PatchTST, iTransformer and KAN (training 2014-09 to 2024-08) | | Evaluation | 10 rolling-origin windows between August 2024 and February 2026; 20-session horizon; context of 512 for all | | Metric | Mean absolute error (MAE); skill = 1 − MAE / MAE of the zero-return random walk | | Test | One-sided Diebold-Mariano with the Harvey, Leybourne and Newbold correction; p not corrected for multiple testing | The results the syllabus repeats are in the paper as stated: the pretrained models win 8 of the 10 tasks, and **Moirai-2.0 and TimesFM-2.5 have the best average ranks, 2.9 and 3.1** (table 4; they are followed by Chronos with 3.4, Chronos-2 with 3.9, iTransformer with 4.3 and KAN with 5.4). iTransformer, trained from scratch, wins the two META tasks. And the abstract puts it bluntly: “gains over the random-walk benchmark are small and sparse”. *Fig. — 2026 benchmark (arXiv:2606.27100): improvement in mean absolute error of each task's winning model against forecasting a zero return, 20 sessions ahead with a 512 context.* Ten horizontal bars, one per task (five stocks, linear and log returns). Six winners have negative skill, between −2.9 and −5.2%: the best model in the task is worse than forecasting zero for AAPL, JPM and META. The positive ones are AMZN (8.6 and 8.2%) and GOOG (22.9 and 22.8%, highlighted, Moirai-2.0). The figure plots table 5 of the paper: the skill of the **winner** of each task. In six of the ten (AAPL, JPM and META, on both targets) the best model in the comparison has a **larger** absolute error than forecasting a zero return: skill from −2.9 to −5.2 %. The four improvements are in AMZN (8.6 and 8.2 %) and GOOG (22.9 and 22.8 %). The one-sided Diebold-Mariano test (table 7) only rejects the hypothesis of “equal to or worse than the random walk” in two cases, Chronos on AMZN and Moirai-2.0 on GOOG, both with p = 0.0421; for TimesFM-2.5 on AAPL and JPM, p = 0.9999. In table 4, each of the four best-ranked pretrained models has positive skill in 4 of the 10 tasks; the two trained from scratch, in none. Three readings that the paper itself supports: - **Ranking is not predictability.** Ordering eleven models tells you which is the least bad, not whether any of them is useful. The abstract phrases it like this: “strong rankings need not imply economically meaningful predictability in noisy markets”. - **Significance is exploratory.** The tests cover several models, stocks and targets and the p-values are reported uncorrected; the authors ask readers to take the significant cases “as exploratory evidence rather than as family-level discoveries” and acknowledge that many differences among the top of the ranking are within the noise floor of the test window (about 200 sessions). - **It is not a trading test.** The paper evaluates forecast error, not usefulness: calibration, turnover, costs and capacity are left out, and the authors state explicitly that their results should not be read as evidence of alpha. ## The other study: strategies with thousands of stocks The syllabus cites “another study” in which strategies built on these models perform worse than specific reference models. It is “Re(Visiting) Time Series Foundation Models in Finance”, by Eghbal Rahimikia (University of Manchester), Hao Ni (UCL) and Weiguan Wang (Shanghai University), arXiv:2511.18578, posted on November 23, 2025; the figures come from the PDF of version 1. Its design is very different from the previous one: daily excess returns of about 10,000 US stocks (more than 18 million observations), out-of-sample evaluation from 2001 to 2023 with models re-estimated every year, windows of 5, 21, 252 and 512 sessions, and **decile long-short portfolios**, equally weighted and rebalanced daily, built from each model's forecasts. The reference models are linear (OLS, Lasso, Ridge, Elastic Net, PCR), tree-based (XGBoost, CatBoost, LightGBM) and two small networks; the best is CatBoost. *Fig. — Rahimikia, Ni and Wang (arXiv:2511.18578): daily long-short portfolios of U.S. stocks (2001-2023) built on each model's forecasts; top, 512-session window without costs, with Chronos and TimesFM zero-shot; bottom, the best benchmark model with costs.* Five horizontal bars. Without costs and with a 512 window: CatBoost 6.46; Chronos large zero-shot 2.92; TimesFM 500 million zero-shot −0.18. With costs, mean of the four windows: CatBoost with 20 basis points −3.62 (highlighted) and with 40, −13.59. The results of the no-retraining (zero-shot) mode use the public versions of Chronos (v1, from tiny to large) and of TimesFM (version 1 with 200 million parameters and version 2 with 500 million), not Chronos-Bolt or TimesFM-2.5. With a 512-session window and no costs, the best configuration, Chronos large, gives an annualized return of 20.17 % and a Sharpe of 2.92; the best TimesFM, −1.47 % and −0.18; CatBoost with the same window, 47.25 % and 6.46. In goodness of fit, with a 512 window, the out-of-sample R² of Chronos large is −1.37 % and that of the 500-million TimesFM, −2.80 %; CatBoost's, averaged over the four windows, −0.10 %. Fine-tuning on financial data does not close the gap. What does narrow it is **pretraining from scratch** on returns, year by year and without seeing the future: Chronos small pretrained this way reaches 36.84 % and a Sharpe of 5.42. The figure adds what the study shows once costs are included: at 20 basis points or more, **every** long-short portfolio turns to a negative Sharpe, whatever the model. CatBoost's average Sharpe over the four windows falls from 6.44 without costs to −3.62 with 20 basis points and to −13.59 with 40. A Sharpe of 6 without costs on a portfolio rebuilt every day over thousands of stocks is a statistical result (the signal exists), not a strategy. | | 2026 benchmark | Rahimikia, Ni and Wang (2025) | |---|---|---| | Universe | 5 large stocks | ≈ 10,000 US stocks (and 7 more markets) | | Test period | Aug. 2024 to Feb. 2026 (10 windows) | 2001-2023 | | Horizon | 20 sessions | 1 session | | Foundation models | TimeGPT, TimesFM-2.5, Moirai-2.0, Chronos, Chronos-2 | Chronos v1, TimesFM v1 and v2, and twelve more | | References | Networks trained per stock and random walk | Linear, tree and network models on the whole panel | | Measure | Absolute error, skill, Diebold-Mariano | R², hit rate, portfolios, Sharpe with and without costs | | Authors' conclusion | Useful as a starting point; not a reliable source of alpha | Weak without retraining; financial pretraining helps | A methodological detail of this second study applies to any work with public models: the authors warn (note 17) that, depending on the data each model was pretrained on, **there may be look-ahead bias**: if the pretraining corpus contains the same financial series from the test period, the model has seen the future. That is why they pretrain their own models with no overlap. In this course's synthetic data the problem does not exist: none of these series was in the pretraining of Chronos-Bolt or TimesFM-2.5. ## Volatility: where there is structure The syllabus conclusion (“useful for volatility, covariates and series with structure”) has support, with nuances. Two recent papers, of which only what their arXiv abstracts say is cited here: - **Brini (2026)**, arXiv:2607.05291, compares nine foundation models without retraining with eight econometric specifications (the HAR family included) on 50 equity, currency and futures assets: “Foundation models do not deliver a uniform gain”. Averaging by asset against a well-specified Log-HAR, only one small model (Tiny Time Mixers) beats it at every horizon, and narrowly; a Mincer-Zarnowitz recalibration shows that much of the short-term advantage is a matter of scale, not dynamics; and the simple average of that model and the Log-HAR enters the Model Confidence Set for 98-100 % of the assets. - **Goel, Pasricha, Magris and Kanniainen (2025)**, arXiv:2505.11163[^1], evaluate TimesFM on volatility: without retraining it is “a reasonable baseline”, and with incremental fine-tuning it outperforms traditional models under Diebold-Mariano and Giacomini-White tests. Volatility has what returns lack: persistence, long memory and a recognizable shape (clustering, mean reversion) that resembles that of many series in the pretraining corpus. That is why a foundation model forecasts it reasonably well without tuning, and for the same reason a four-parameter HAR is a very tough rival. Lesson 5.8.5 measures this with a full lab. ## The in-house test: returns unpredictable by construction The studies use real data, in which nobody knows how much predictability there is. The course's synthetic data allow something no paper can do: test the models where the answer is known. In `vol5_diario.parquet` (README of the level 5 data, “Control” section) the conditional mean of the return is **constant**: $E[r_t \mid t-1] = \mu$ exactly, with fat tails and skewness. No model can beat the historical mean except by chance. It is the perfect control for a foundation model: if it “finds” something, it is measuring noise. The protocol mimics the benchmark's, with a rolling origin and no leakage: at each origin $t$ the model receives the 512 previous sessions (from $t-512$ to $t-1$) and forecasts $r_t$. So that the lesson runs in a couple of minutes on CPU, **one in every 40 sessions** from 2018 to 2025 is taken on the six assets; real research would use every day and more series. ```mermaid flowchart LR A["512 sessions"] --> B[Untuned model] B --> C["Forecast of r_t"] C --> D["Loss vs base"] ``` ```python import numpy as np import pandas as pd d = pd.read_parquet("vol5_diario.parquet") ACT = list(d["activo"].unique()) g = d.groupby("activo", sort=False) r = {a: x["rendimiento"].to_numpy() for a, x in g} rv = {a: x["rv_5min"].to_numpy() for a, x in g} fechas = pd.DatetimeIndex(d.loc[d["activo"] == "indice", "fecha"]) CTX = 512 # context: 512 sessions corte = fechas.searchsorted(pd.Timestamp("2018-01-02")) orig = np.arange(corte, len(fechas), 40) # one in every 40 sessions act = np.repeat(ACT, len(orig)) y = np.concatenate([r[a][orig] for a in ACT]) ctx = [r[a][t - CTX:t].astype("float32") for a in ACT for t in orig] media = np.concatenate([[r[a][:t].mean() for t in orig] for a in ACT]) print(f"{len(orig)} origins per asset, {len(y)} forecasts, " f"{fechas[orig[0]].date()} to {fechas[orig[-1]].date()}") ``` ```output 51 origins per asset, 306 forecasts, 2018-01-02 to 2025-12-16 ``` There are two baselines. The benchmark's, the **random walk** on the price, which amounts to forecasting a zero return. And the one from the predictability literature (Campbell and Thompson, 2008; Welch and Goyal, 2008[^2]), the **historical mean** of returns up to the previous day. Both models are used without retraining (how to load and configure them is in lesson 5.8.3), with the median as the point forecast. ```python import torch torch.manual_seed(0) torch.use_deterministic_algorithms(True) from chronos import BaseChronosPipeline import timesfm chronos = BaseChronosPipeline.from_pretrained( "amazon/chronos-bolt-small", device_map="cpu", dtype=torch.float32) q, _ = chronos.predict_quantiles( inputs=[torch.tensor(c) for c in ctx], prediction_length=1, quantile_levels=[0.1, 0.5, 0.9]) f_ch = q[:, 0, 1].numpy().astype(float) # forecast median tfm = timesfm.TimesFM_2p5_200M_torch.from_pretrained( "google/timesfm-2.5-200m-pytorch", torch_compile=False) tfm.compile(timesfm.ForecastConfig( max_context=CTX, max_horizon=16, normalize_inputs=True, use_continuous_quantile_head=True, per_core_batch_size=len(ctx))) punto, _ = tfm.forecast(horizon=1, inputs=ctx) f_tf = punto[:, 0].astype(float) prev = {"zero": np.zeros_like(y), "mean": media, "chronos": f_ch, "timesfm": f_tf} pd.DataFrame({k: [np.std(v) / np.std(y), 100 * np.mean(v > 0)] for k, v in prev.items()}, index=["std / std of r", "% forecasts > 0"]).round(3) ``` ```output zero mean chronos timesfm std / std of r 0.0 0.028 0.072 0.112 % forecasts > 0 0.0 83.333 62.745 68.301 ``` `per_core_batch_size=len(ctx)` makes TimesFM process the 306 series at once: the default value, 1, passes them one by one and on CPU multiplies the time. The two models' forecasts are small, as they should be: their standard deviation is 7.2 % (Chronos) and 11.2 % (TimesFM) of that of the return, more than that of the historical mean (2.8 %) but very far from what would be needed to explain anything. Both lean upward (63 % and 68 % of the forecasts are positive), like the mean, which is positive at 83 % of the origins because five of the six assets have μ > 0. ### Skill against forecasting zero The skill is the benchmark's: $1 - \text{MAE}_{\text{model}} / \text{MAE}_{\text{zero}}$, and its version with the mean squared error (MSE). With the historical mean as the base and the MSE, it is the out-of-sample $R^2$ of Campbell and Thompson: $$ R^2_{oos} = 1 - \frac{\sum_t (r_t - \hat r_t)^2}{\sum_t (r_t - \overline{r}_{t-1})^2} $$ ```python def skill(f, base, m, perdida): """1 - mean loss of the model / mean loss of the base, in %.""" L = (lambda p: (y[m] - p[m]) ** 2) if perdida == "mse" else ( lambda p: np.abs(y[m] - p[m])) return 100 * (1 - L(f).mean() / L(base).mean()) filas = {} for a in ACT: m = act == a filas[a] = {f"{k}_{p}": skill(prev[k], prev["zero"], m, p) for k in ["mean", "chronos", "timesfm"] for p in ["mse", "mae"]} t = pd.DataFrame(filas).T t.loc["asset mean"] = t.mean() t.round(2) ``` ```output mean_mse mean_mae chronos_mse chronos_mae timesfm_mse timesfm_mae indice -1.02 0.31 -6.55 -2.45 -1.72 -0.16 accion 0.12 1.26 -0.44 0.76 1.07 -0.76 bono 0.08 0.15 -3.63 -1.75 -6.33 -2.83 oro 0.94 -0.35 -0.38 -2.50 2.94 -0.43 divisa -0.79 -0.53 3.88 2.03 2.38 0.89 cripto 1.01 0.43 2.28 1.86 -0.88 -1.12 asset mean 0.06 0.21 -0.81 -0.34 -0.42 -0.74 ``` The table looks a lot like the benchmark's: skill of a few points, sometimes positive and sometimes negative. Chronos ranges from −6.55 % (index, MSE) to +3.88 % (currency); TimesFM, from −6.33 % (bond) to +2.94 % (gold). Averaged over the six assets, both lose against forecasting zero with both losses (from −0.34 to −0.81 %), and the historical mean comes out even (+0.06 and +0.21 %). If this table came from real data, someone could write that “TimesFM improves 2.9 % on gold” or “Chronos 3.9 % on currencies”. Here we know there is nothing to improve: those figures are the size of the noise with 51 forecasts. ## Diebold-Mariano and the multiple-testing correction To know whether a difference in losses is noise you need a test. The Diebold and Mariano (1995) test compares the losses of two forecasts of the same target, $d_t = L(e^{(1)}_t) - L(e^{(2)}_t)$, and tests whether their mean is zero with a variance that allows for autocorrelation (the long-run variance, $\hat\sigma^2_\infty$, with autocovariances up to $h-1$ for a horizon $h$): $$ DM = \frac{\overline{d}}{\sqrt{\hat\sigma^2_\infty / n}},\qquad DM^* = DM\,\sqrt{\frac{n + 1 - 2h + h(h-1)/n}{n}} $$ $DM^*$ is the Harvey, Leybourne and Newbold (1997) small-sample correction, which is compared with a Student's t with $n-1$ degrees of freedom; it is the one the 2026 benchmark uses. Here $h = 1$ and the alternative hypothesis is one-sided: the model has a lower loss than the base ($DM^* < 0$). ```python from scipy import stats def dm(l_mod, l_base, h=1): """Diebold-Mariano with the Harvey, Leybourne and Newbold correction. One-sided H1: the model has a lower loss.""" dd = l_mod - l_base n = len(dd) u = dd - dd.mean() var = u @ u / n for k in range(1, h): # autocovariances up to h-1 var += 2 * (u[k:] @ u[:-k]) / n t = dd.mean() / np.sqrt(var / n) t *= np.sqrt((n + 1 - 2 * h + h * (h - 1) / n) / n) return t, stats.t.cdf(t, df=n - 1) pruebas = [] for a in ACT: m = act == a for k in ["chronos", "timesfm"]: for b in ["zero", "mean"]: for p, L in [("mse", np.square), ("mae", np.abs)]: t_, p_ = dm(L(y[m] - prev[k][m]), L(y[m] - prev[b][m])) pruebas.append((a, k, b, p, t_, p_)) pr = pd.DataFrame(pruebas, columns=["asset", "model", "base", "loss", "t", "p"]) print(len(pr), "tests; p < 0.05:", (pr["p"] < 0.05).sum()) pr.sort_values("p").head(4).round(3) ``` ```output 48 tests; p < 0.05: 1 asset model base loss t p 34 divisa chronos mean mse -1.704 0.047 32 divisa chronos zero mse -1.551 0.064 41 cripto chronos zero mae -1.512 0.068 43 cripto chronos mean mae -1.486 0.072 ``` Six assets, two models, two baselines and two losses make **48 tests**, and one comes out “significant”: Chronos against the mean on the currency, with the squared error, t = −1.704 and p = 0.047. It is the asset with μ = 0 and no dependence at all between return and volatility (ρ = 0 in the README): the least predictable of the six. With 48 tests at 5 %, about 2.4 rejections are expected even if nothing works (somewhat fewer here, because the tests share data and are correlated). The figure shows all 48: *Fig. — The lesson's 48 Diebold-Mariano tests (six assets, squared and absolute error) on the return forecasts of the two foundation models against forecasting zero and against the historical mean.* Four rows of twelve marks between −1.7 and 1.8. Only one mark (Chronos against the mean on the currency, squared error, t = −1.70, highlighted) falls left of the uncorrected one-sided 5% threshold (−1.68); none comes close to the Holm threshold with 48 tests (−3.25). Most are positive: the models lose more than the baselines. Most of the statistics are **positive**: the models lose more than the baselines, which is what you expect when a forecast adds variance without information. The standard way to control the family-wise error is to adjust the p-values. Holm (1979) sorts them and compares the smallest with $0.05/48$, the next with $0.05/47$, and so on; it is uniformly more powerful than Bonferroni and does not assume independence. ```python from statsmodels.stats.multitest import multipletests rech, p_holm, _, _ = multipletests(pr["p"], alpha=0.05, method="holm") print("rejections with Holm:", rech.sum(), "| minimum adjusted p:", round(p_holm.min(), 3)) ``` ```output rejections with Holm: 0 | minimum adjusted p: 1.0 ``` No rejection survives: the minimum adjusted p is 1 (0.047 × 48 exceeds 1 and is truncated). It is the same logic as the Deflated Sharpe of lesson 5.4.4 applied to forecast tests. And it helps to read the benchmark: two rejections with p = 0.0421 within a family of models, stocks and targets would not survive Holm either, as its authors acknowledge when they ask for them to be read as exploratory. ## The illusion of levels A mistake that shows up in demos of foundation models on financial data: feeding the model the **price** and measuring accuracy on the price. The cell builds a price from each asset's returns, asks Chronos for tomorrow's price and compares it with two things: the sample mean of the price (the usual R²) and yesterday's price (the random walk). Then it translates the forecast into an implied return, $\hat P_t / P_{t-1} - 1$. ```python precio = {a: 100 * np.cumprod(1 + r[a]) for a in ACT} ctx_p = [precio[a][t - CTX:t].astype("float32") for a in ACT for t in orig] qp, _ = chronos.predict_quantiles( inputs=[torch.tensor(c) for c in ctx_p], prediction_length=1, quantile_levels=[0.5]) p_ch = qp[:, 0, 0].numpy().astype(float) p_hoy = np.concatenate([precio[a][orig] for a in ACT]) p_ayer = np.concatenate([precio[a][orig - 1] for a in ACT]) def r2(obs, f, base): sse, sst = np.sum((obs - f) ** 2), np.sum((obs - base) ** 2) return 100 * (1 - sse / sst) filas = {} for a in ACT: m = act == a r_impl = p_ch[m] / p_ayer[m] - 1 # implied return filas[a] = { "R2_price": r2(p_hoy[m], p_ch[m], p_hoy[m].mean()), "MAPE_chronos": 100 * np.mean(np.abs(p_ch[m] / p_hoy[m] - 1)), "MAPE_yesterday": 100 * np.mean(np.abs(p_ayer[m] / p_hoy[m] - 1)), "R2_impl_ret": r2(y[m], r_impl, media[m])} pd.DataFrame(filas).T.round(2) ``` ```output R2_price MAPE_chronos MAPE_yesterday R2_impl_ret indice 99.82 0.82 0.74 -22.38 accion 99.77 1.02 0.92 -0.48 bono 99.37 0.34 0.31 -42.84 oro 99.84 0.59 0.45 -40.46 divisa 98.94 0.38 0.36 -16.07 cripto 99.27 2.92 2.70 -17.85 ``` The R² of the price ranges from 98.94 to 99.84 %, and a mean absolute percentage error (MAPE) of 0.3 to 2.9 % sounds precise. Both figures are an illusion: the price is so persistent that **repeating yesterday's** gives a lower MAPE than Chronos on all six assets (0.74 versus 0.82 % on the index, 2.70 versus 2.92 % on crypto). The return implied by the forecast loses against the historical mean on all of them, with an R² from −0.48 to −42.84 %: by forecasting the level, the model adds noise to yesterday's price. The header figure puts this row next to the others. ## The same model where there is structure The final contrast uses the same Chronos-Bolt, the same origins and the same context, but with the log of the realized variance instead of the return. In `vol5` that series follows a HAR in logs with persistence 0.92: there is a lot of structure to learn from the past. The base is again the historical mean, now of the log RV. ```python ctx_v = [np.log(rv[a][t - CTX:t]).astype("float32") for a in ACT for t in orig] qv, _ = chronos.predict_quantiles( inputs=[torch.tensor(c) for c in ctx_v], prediction_length=1, quantile_levels=[0.5]) v_ch = qv[:, 0, 0].numpy().astype(float) lv = np.log(np.concatenate([rv[a][orig] for a in ACT])) lv_media = np.concatenate([[np.log(rv[a][:t]).mean() for t in orig] for a in ACT]) r2_v = {a: r2(lv[act == a], v_ch[act == a], lv_media[act == a]) for a in ACT} r2_r = {a: r2(y[act == a], f_ch[act == a], media[act == a]) for a in ACT} pd.DataFrame({"R2_log_RV": r2_v, "R2_return": r2_r}).round(1) ``` ```output R2_log_RV R2_return indice 67.7 -5.5 accion 24.1 -0.6 bono 48.6 -3.7 oro 56.0 -1.3 divisa 29.2 4.6 cripto 44.8 1.3 ``` With volatility, the same model with no tuning at all explains between 24.1 % (stock) and 67.7 % (index) of the variance of the log RV out of sample; with the return, between −5.5 and +4.6 %. The header figure summarizes the four rows with the average of the six assets: 99.5 % for the price (illusion), 45.1 % for the log RV (real structure), −0.9 % for the return (nothing) and −23.3 % for the return implied by the price (worse than nothing). A high R² on volatility does not yet say whether the model is **good**: it has to be compared with what a HAR or a GARCH would do on the same dates and with the right loss. That is the lab in lesson 5.8.5. ## How to read a study of foundation models in finance | Question | Red flag | In this lesson's studies | |---|---|---| | Against what baseline? | Only against other complex models | The benchmark includes the random walk; Rahimikia et al., linear and tree models | | What target? | Prices in levels | Returns in both | | What loss and what test? | No test, or a test that ignores dependence | Diebold-Mariano with HLN in the first | | How many tests? | Isolated p-values among dozens of combinations | Uncorrected p-values in the first, acknowledged by the authors | | How long is the test? | A single short window | ≈ 200 sessions versus 23 years | | Overlap with pretraining? | Public models on public series from the test period | Flagged and avoided in Rahimikia et al. | | Economic value? | Sharpe without costs | Negative with 20 basis points in Rahimikia et al. | ## Common mistakes - **Confusing ranking with predictability.** The best of eleven models can be worse than forecasting zero; in the 2026 benchmark it is in six of ten tasks. - **Evaluating prices in levels.** An R² of 99 % and a MAPE of 1 % say nothing if yesterday's price does as well or better. - **Keeping the only p < 0.05 out of 48.** In the lesson's control, the only “significant” one is on the asset whose mean is exactly constant. - **Forgetting the right baseline.** The random walk and the historical mean are hard to beat on returns; a model that is not compared with them has proven nothing. - **Reading the Sharpe of a daily portfolio without costs.** With daily turnover over thousands of stocks, 20 basis points turn a Sharpe of 6 into a negative one. - **Ignoring possible overlap.** A public model pretrained on financial series may have seen the test period of the real data. ## Exercises 1. Repeat the benchmark protocol with Chronos-Bolt: a 20-session horizon, mean absolute error over the 20 sessions of each origin and skill against forecasting zero, with the same origins (except the last one, which does not have 20 sessions left). Does the conclusion change? Since the origins are 40 sessions apart, the windows do not overlap and $h = 1$ in Diebold-Mariano with the mean loss of each origin is enough. **Solution.** ```python H = 20 o20 = orig[:-1] ctx20 = [r[a][t - CTX:t].astype("float32") for a in ACT for t in o20] q20, _ = chronos.predict_quantiles( inputs=[torch.tensor(c) for c in ctx20], prediction_length=H, quantile_levels=[0.5]) f20 = q20[:, :, 0].numpy().astype(float) # (origins, 20) y20 = np.array([r[a][t:t + H] for a in ACT for t in o20]) a20 = np.repeat(ACT, len(o20)) eam_m = np.abs(y20 - f20).mean(axis=1) # MAE per origin eam_0 = np.abs(y20).mean(axis=1) filas = {} for a in ACT: m = a20 == a t_, p_ = dm(eam_m[m], eam_0[m]) sk = 100 * (1 - eam_m[m].mean() / eam_0[m].mean()) filas[a] = {"skill_%": sk, "t": t_, "p": p_} pd.DataFrame(filas).T.round(3) ``` ```output skill_% t p indice 0.703 -1.137 0.131 accion 0.281 -0.743 0.231 bono -0.011 0.035 0.514 oro -0.930 2.250 0.986 divisa -0.408 2.396 0.990 cripto -0.227 0.615 0.729 ``` It does not change. With the benchmark's horizon and metric, Chronos's skill against forecasting zero ranges from −0.93 % (gold) to +0.70 % (index), and no one-sided p falls below 0.13. On gold and the currency the statistic is positive and large (t = 2.25 and 2.40): there the model is **worse** than forecasting zero, and a two-sided test would say so with p < 0.05. Adding up 20 sessions does not create information: the forecast at each step still cannot anticipate a return whose conditional mean is constant. With 50 origins per asset and absolute error, the noise is of the same order as in the benchmark. 2. Simulate the multiple-testing problem: generate 2,000 pure-noise “models” for the index (normal forecasts with the standard deviation of Chronos's forecasts on that asset) and test each one against the historical mean with the squared error. What fraction gives p < 0.05? What is the p of the best one? **Solution.** ```python rng = np.random.default_rng(5) m = act == "indice" ruido = rng.normal(0, f_ch[m].std(), size=(2000, m.sum())) l_base = (y[m] - media[m]) ** 2 ps = np.array([dm((y[m] - f) ** 2, l_base)[1] for f in ruido]) print(f"p < 0.05: {np.mean(ps < 0.05):.1%} | best p: {ps.min():.4f}" f" | with Holm: {multipletests(ps, method='holm')[0].sum()}") ``` ```output p < 0.05: 4.2% | best p: 0.0004 | with Holm: 0 ``` 4.2 % of the noise models “improve” on the mean with p < 0.05; slightly below the nominal 5 % because noise added to the forecast worsens the squared error in expectation. The best one has p = 0.0004, a result that would look compelling in a paper; with Holm over the 2,000, none survives. It is the extreme version of what happens when you test many models, contexts, quantiles and assets and publish the best: without the number of attempts, a small p means nothing (lessons 2.5.4 and 5.4.4). 3. No code: a paper claims that “a foundation model without retraining achieves a Sharpe of 3 in a daily long-short strategy with the 500 stocks of the S&P 500, 2020-2024, versus 0.4 for momentum”. What questions would you ask before believing it? **Solution.** First, those from the chain of evidence. Are there costs? With daily rebalancing of 500 stocks, turnover is huge and 20 basis points were enough to make a Sharpe of 6 negative in Rahimikia, Ni and Wang. Was the test period in the pretraining? Public models were pretrained on corpora that may include US stock prices from those years: that would be look-ahead bias. How many configurations (model, size, context, horizon, quantile, portfolio rule) were tried before this one? The Sharpe would have to be deflated by that number (lesson 5.4.4). Are the 500 stocks the index constituents at each date or today's (survivorship)? Is the difference with momentum significant in a paired test? And a cheap check you can do at home: run the same pipeline on a series with an unpredictable mean, like the `vol5` control, and see whether it also “wins”. [^1]: Goel, A., Pasricha, P., Magris, M. and Kanniainen, J. (2025): Foundation Time-Series AI Model for Realized Volatility Forecasting, arXiv:2505.11163, . [^2]: Welch, I. and Goyal, A. (2008): A Comprehensive Look at the Empirical Performance of Equity Premium Prediction. The Review of Financial Studies 21(4), 1455-1508, . ## Self-assessment 1. In the 2026 benchmark (arXiv:2606.27100), Moirai-2.0 and TimesFM-2.5 achieve the best average ranks (2.9 and 3.1) among eleven models. What can be concluded from that? - A. That both predict daily returns better than the random walk - B. That they win all ten tasks in the study - C. That they rank better than the other models in the comparison, which does not imply beating a zero forecast: in six of the ten tasks the winner has negative skill against the random walk - D. That their forecasts generate economic value after costs *Answer C: That they rank better than the other models in the comparison, which does not imply beating a zero forecast: in six of the ten tasks the winner has negative skill against the random walk.* The ranking compares models with each other. Table 5 of the paper shows that the winner for AAPL, JPM and META (on simple and log returns) has a larger absolute error than forecasting a zero return, and the Diebold-Mariano test only rejects in two cases. The authors themselves write that strong rankings do not imply economically meaningful predictability. 2. The lesson runs 48 Diebold-Mariano tests (6 assets, 2 models, 2 baselines, 2 losses) and one gives p = 0.047: Chronos against the historical mean on the currency. How should it be interpreted? - A. As an expected false positive: with 48 tests at 5 % it would be odd to get none, it does not survive Holm and on that series the conditional mean is constant by construction - B. As proof that Chronos captures something in currencies that the other assets do not have - C. As a sign that you should trade the currency with Chronos and monitor it - D. As a flaw in the test, which only works with more than 1,000 observations *Answer A: As an expected false positive: with 48 tests at 5 % it would be odd to get none, it does not survive Holm and on that series the conditional mean is constant by construction.* With 48 independent tests at 5 %, about 2.4 rejections are expected even if nothing works. The Holm correction takes the minimum adjusted p to 1. And the known answer of the generator says that E[r_t | t−1] = μ exactly: there is nothing to find. 3. Chronos-Bolt, fed the price instead of the return, gives an R² of 98.9 to 99.8 % on the six assets. Why is that not good news? - A. Because R² only applies to linear models - B. Because 51 origins per asset are too few to measure R² - C. Because the model is overfitted to the 2018-2025 data - D. Because the price is so persistent that any forecast close to yesterday's price has an R² of 99 %; the useful comparison is with yesterday's price, which Chronos does not beat in percentage error, and its implied return gives an R² of −23 % on average *Answer D: Because the price is so persistent that any forecast close to yesterday's price has an R² of 99 %; the useful comparison is with yesterday's price, which Chronos does not beat in percentage error, and its implied return gives an R² of −23 % on average.* The R² of a level is measured against the sample mean of the level, an absurd reference for a series with a unit root. Chronos's mean absolute percentage error is larger than that of repeating yesterday's price on all six assets, and the return implied by its forecast loses against the historical mean on all of them. 4. Rahimikia, Ni and Wang (arXiv:2511.18578) report a Sharpe of 6.46 for CatBoost's long-short portfolio without costs (512 window) and an average of −3.62 with 20 basis points. Which conclusion is correct? - A. That CatBoost is an excellent strategy if traded with a cheap broker - B. That a decile long-short portfolio rebalanced every day over thousands of stocks turns over so much that the Sharpe without costs says nothing about the net; and that the foundation models without retraining were already far behind before costs - C. That costs affect every strategy equally and do not change the comparison - D. That the study shows foundation models work better with costs *Answer B: That a decile long-short portfolio rebalanced every day over thousands of stocks turns over so much that the Sharpe without costs says nothing about the net; and that the foundation models without retraining were already far behind before costs.* The authors say that at 20 basis points or more every long-short portfolio turns to a negative Sharpe, whatever the model. Before costs, Chronos large without retraining gave 2.92 and the 500-million-parameter TimesFM −0.18, against 6.46 for CatBoost. 5. Given the evidence in this lesson, which use of a time series foundation model has the most support in finance? - A. Forecasting tomorrow's return of a stock and trading its sign - B. Replacing every econometric model without comparing them - C. Forecasting volatility or series with structure as one more baseline, compared with a HAR using formal tests, and with skepticism toward direct return prediction - D. Generating future prices to compute VaR *Answer C: Forecasting volatility or series with structure as one more baseline, compared with a HAR using formal tests, and with skepticism toward direct return prediction.* Volatility has structure (long memory, clustering) and the same Chronos-Bolt explains between 24 and 68 % of the variance of the log RV in the course data. Even so, in a study with 50 real assets the foundation models do not uniformly improve on the Log-HAR (Brini, 2026). On returns, the improvements are small, sparse and do not survive costs. ## Key points - A ranking among models is not an improvement over a naive baseline. In the 2026 benchmark, Moirai-2.0 and TimesFM-2.5 have the best average ranks (2.9 and 3.1), but the winner of six of the ten tasks loses against forecasting a zero return and Diebold-Mariano only rejects in two cases, with uncorrected p-values. - In the Rahimikia, Ni and Wang study, Chronos and TimesFM without retraining are far behind CatBoost in long-short portfolios (Sharpe 2.92 and −0.18 versus 6.46, without costs); with 20 basis points of cost every portfolio, CatBoost's included, turns to a negative Sharpe. - On a series with a constant conditional mean, Chronos-Bolt and TimesFM-2.5 produce forecasts with 7-11 % of the standard deviation of the return and negative average skill against forecasting zero; of 48 tests, one gives p < 0.05 and none survives Holm. - Forecasting prices in levels is an illusion: an R² of 99 % that does not beat yesterday's price, and an implied return with an R² of −23 % on average. - The same model explains between 24 and 68 % of the variance of the log realized variance: foundation models are useful where there is structure, and even there they have to be compared with a HAR (lesson 5.8.5). - To read a study: naive baseline, loss, a test that handles dependence, multiple-testing correction, overlap with pretraining, costs and capacity. ## Resources ### Books - [Noguer i Alonso, M. and Pereira Franklin, R. (2026): Pretrained Time-Series Foundation Models for Financial Return Forecasting (arXiv:2606.27100)](https://arxiv.org/abs/2606.27100) · arxiv.org - [Rahimikia, E., Ni, H. and Wang, W. (2025): Re(Visiting) Time Series Foundation Models in Finance (arXiv:2511.18578)](https://arxiv.org/abs/2511.18578) · arxiv.org - [Brini, A. (2026): Forecasting Realized Volatility with Time Series Foundation Models: A Comparison with Econometric Benchmarks (arXiv:2607.05291)](https://arxiv.org/abs/2607.05291) · arxiv.org - [Diebold, F. X. and Mariano, R. S. (1995): Comparing Predictive Accuracy. Journal of Business & Economic Statistics 13(3), 253-263](https://doi.org/10.1080/07350015.1995.10524599) · doi.org - [Harvey, D., Leybourne, S. and Newbold, P. (1997): Testing the equality of prediction mean squared errors. International Journal of Forecasting 13(2), 281-291](https://doi.org/10.1016/S0169-2070(96)00719-4) · doi.org - [Campbell, J. Y. and Thompson, S. B. (2008): Predicting Excess Stock Returns Out of Sample. The Review of Financial Studies 21(4), 1509-1531](https://doi.org/10.1093/rfs/hhm055) · doi.org ### Data - [vol5_diario.parquet: returns and realized variance of six assets (synthetic data)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet Previous lesson: [5.8.3 · In practice: context, quantiles, covariates and fine-tuning](https://pythonparatrading.com/en/python-trading-course/level-5/zero-shot-covariates-fine-tuning-practice/) Next lesson: [5.8.5 · Lab: foundation models vs GARCH, HAR and LightGBM](https://pythonparatrading.com/en/python-trading-course/level-5/foundation-models-vs-garch-lightgbm-lab/)