Lesson 4.2.3 · 75 min

VaR, Expected Shortfall and VaR backtesting

VaR sums up in one figure the loss that should almost never be exceeded, and Expected Shortfall says how much is lost when it is. This lesson computes both with six methods on a book whose true VaR is known every day, puts them through the Kupiec, Christoffersen and Acerbi-Székely tests and the Basel Committee traffic light, and measures how wrong each one is and when.

Fig. — One-day 99% VaR of the 10 million USD book, January 2010 to March 2011: the true one (known GARCH-t) against the 250-session historical, and the losses that exceed the historical

03/06

Module: 4.2 · Risk management and position sizing
Duration: 75 min
Prerequisites: 1.6.2 · Stylized facts: fat tails, clustering, autocorrelation and leverage, 2.1.4 · Volatility models: ARCH, GARCH, EGARCH and GJR with the arch library, 2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations, 4.2.2 · Kelly criterion and volatility targeting: how much to risk
Author: Jesús Cuesta
Updated: 04 Nov 2026

Level 4 lessons

What VaR measures and what it does not

The VaR (value at risk) of a book at a horizon $h$ and a confidence level $1-\alpha$ is the loss that is only exceeded with probability $\alpha$ . With $L$ the loss (positive when money is lost) over that horizon:

Z_2 = \sum_{t=1}^{T} \frac{X_t, I_t}{T,\alpha,\mathrm{ES}_t} + 1.

With a correct ES, $\mathrm{E}[Z_2] = 0$; if the model underestimates the tail, $Z_2 < 0$. The data README gives the same statistic with losses as positive numbers, with the sign flipped. For each method, the ES of its own model: the mean of the tail of 250 observations (historical), the normal formula $\sigma\,\varphi(z_\alpha)/\alpha$, the standardized t formula for the GARCH-t and the mean of the tail of the filtered residuals (FHS): :::cell{title="97.5% ES for each method"} ```python a = 0.025 za = stats.norm.ppf(a) es_norm = stats.norm.pdf(za) / a # ES of the standard normal def es_historico(x): cola = np.sort(x)[: int(np.ceil(a * len(x)))] return -cola.mean() tg = stats.t.ppf(a, gt["nu"]) es_tg = ((gt["nu"] + tg**2) / (gt["nu"] - 1) * stats.t.pdf(tg, gt["nu"]) / a * np.sqrt((gt["nu"] - 2) / gt["nu"])) sig_n = r.rolling(V).std().shift(1) cola_fhs = res_ewma.rolling(V).apply(es_historico, raw=True).shift(1) modelos = { "verdadero": (verdad["var_975"], verdad["es_975"]), "historico": (-r.rolling(V).quantile(a).shift(1) * N, r.rolling(V).apply(es_historico, raw=True) .shift(1) * N), "normal": (-sig_n * za * N, sig_n * es_norm * N), "ewma": (-sig_ewma.shift(1) * za * N, sig_ewma.shift(1) * es_norm * N), "fhs": (-sig_ewma.shift(1) * res_ewma.rolling(V).quantile(a) .shift(1) * N, sig_ewma.shift(1) * cola_fhs * N), "garch_t": ((-(gt["mu"] + gt["sigma"] * tg * np.sqrt((gt["nu"] - 2) / gt["nu"]))) * N, (-gt["mu"] + gt["sigma"] * es_tg) * N)} X = pyg["pyg"].loc[prueba] # P&L (negative = loss) def z2(X, var_a, es_a): """Acerbi-Székely (2014): ≈ 0 if the ES is correct; < 0 if not.""" I = (X + var_a < 0) return (X * I / (len(X) * a * es_a)).sum() + 1 filas = {} for m, (v, e) in modelos.items(): v, e = v.loc[prueba], e.loc[prueba] filas[m] = {"es_medio": e.mean(), "excesos_975_pct": 100 * (X + v < 0).mean(), "Z2": z2(X, v, e)} tabla_es = pd.DataFrame(filas).T tabla_es.round(3) ``` ```output es_medio excesos_975_pct Z2 verdadero 228952.309 2.490 0.004 historico 245963.360 3.703 -0.691 normal 216480.024 3.619 -0.835 ewma 197701.580 3.075 -0.401 fhs 238706.681 2.908 -0.229 garch_t 228438.812 2.573 -0.067 ``` ::: Acerbi and Székely give critical values for $Z_2$ that are almost constant across distributions, around −0.7 at 5% for one year of data. With the true model of the data, the exact critical value can be computed, also for the whole sample: :::cell{title="How bad is a “bad” Z2? Simulation with the true model"} ```python rng = np.random.default_rng(5) sig = verdad["sigma"].loc[prueba].values v_v = verdad["var_975"].loc[prueba].values e_v = verdad["es_975"].loc[prueba].values sim_total, sim_anual = [], [] for _ in range(1000): z = rng.standard_t(nu, len(sig)) * esc Xs = (mu + sig * z) * N I = Xs + v_v < 0 sim_total.append((Xs * I / (len(Xs) * a * e_v)).sum() + 1) k = 250 # a one-year window sim_anual.append((Xs[:k] * I[:k] / (k * a * e_v[:k])).sum() + 1) print(f"5% critical Z2 with {len(sig)} days: " f"{np.quantile(sim_total, 0.05):.3f}") print(f"5% critical Z2 with 250 days: " f"{np.quantile(sim_anual, 0.05):.3f}") ``` ```output 5% critical Z2 with 4780 days: -0.160 5% critical Z2 with 250 days: -0.691 ``` ::: With 250 days the critical value is −0.691, the one in the paper. With the 4,780 test days it is −0.160: with more data, smaller deviations are detected. The true model gives 0.004 and the GARCH-t −0.067: they pass. FHS (−0.229), EWMA (−0.401), historical (−0.691) and normal (−0.835) underestimate the tail and are rejected. The normal model has the worst tail: with a mean ES of 216,500 USD, 5% below the true one, it has 3.62% exceptions of its 97.5% VaR instead of 2.5%, and when it fails, it fails by more. The mean historical ES (246,000 USD) is above the true one and is still rejected, for the same reason as its VaR: it is high when it is not needed. ## What the backtest does not see - **Changing positions.** This book has a constant notional. In a real one, today's P&L comes from positions that yesterday's VaR did not know about if there was intraday trading: that is why MAR32 compares with the actual P&L and with the hypothetical one (yesterday's closing positions revalued), and counts the larger number of exceptions of the two. - **Scenarios that are not in the sample.** A 250-session historical VaR only knows the last year; a GARCH estimated in the calm does not know stress (the October 2008 Monte Carlo). Losses that have never happened are covered by the stress tests of lesson 4.2.4. - **Aggregation.** The VaR of a portfolio depends on correlations, which change in crises (4.2.4). A model that gets the VaR of each strategy right separately can get the VaR of the sum wrong. - **Using VaR as a limit.** If a system reduces positions when its VaR rises, the VaR affects the P&L that judges it. That is desirable, but it has to be taken into account when reading the backtest of a system that uses it. [^1]: Artzner, P., Delbaen, F., Eber, J.-M. and Heath, D. (1999): Coherent Measures of Risk. Mathematical Finance 9(3), 203-228, <https://doi.org/10.1111/1467-9965.00068>. Barone-Adesi, G., Giannopoulos, K. and Vosper, L. (1999): VaR without correlations for portfolios of derivative securities. Journal of Futures Markets 19(5), 583-602, <https://doi.org/10.1002/(SICI)1096-9934(199908)19:5%3C583::AID-FUT5%3E3.0.CO;2-S>. ## Exercises 1. **RiskMetrics for investors.** The RiskMetrics technical document proposed $\lambda = 0.94$ for daily trading data and $\lambda = 0.97$ for investment horizons. Compute the 99% EWMA VaR with 0.97 and with a standardized 5-degree t quantile instead of the normal one, and compare exceptions, Kupiec p and independence p with the normal EWMA with 0.94. ::::solucion :::cell{title="EWMA with λ = 0.97 and a t quantile"} ```python sig97 = np.sqrt((r**2).ewm(alpha=0.03, adjust=False).mean()) variantes = {"0.94 normal": -(sig_ewma * z01).shift(1) * N, "0.97 normal": -(sig97 * z01).shift(1) * N, "0.94 t5": -(sig_ewma * q01).shift(1) * N, "0.97 t5": -(sig97 * q01).shift(1) * N} filas = {} for k, v in variantes.items(): I = (L > v.loc[prueba]).astype(int).values filas[k] = {"excesos": I.sum(), "p_kupiec": kupiec(I)[1], "p_indep": christoffersen(I)[1]} pd.DataFrame(filas).T.round(3) ``` ```output excesos p_kupiec p_indep 0.94 normal 78.0 0.000 0.798 0.97 normal 77.0 0.000 0.524 0.94 t5 54.0 0.377 0.267 0.97 t5 48.0 0.977 0.324 ``` ::: Changing λ changes almost nothing (78 to 77 exceptions): the problem of the normal EWMA is not speed, it is the tail. With the 5-degree t quantile, the 0.94 EWMA drops to 54 exceptions (Kupiec p 0.377) and the 0.97 one to 48, exactly the expected number (p 0.977), and both still pass the independence test. An EWMA with fat tails gets close to the GARCH-t without estimating anything; here it helps that the chosen quantile is the true one of the innovations, something that would have to be estimated with real data. :::: 2. **The power of the traffic light.** Compute, with a 250-day binomial, the probability that a model whose exceptions truly occur with a 2% and a 3% probability ends up in green, amber and red. What does this say about a year in green? ::::solucion :::cell{title="Probability of each zone by the actual frequency of exceptions"} ```python zonas = {} for p_real in (0.01, 0.02, 0.03): bb = stats.binom(250, p_real) zonas[p_real] = {"green": bb.cdf(4), "amber": bb.cdf(9) - bb.cdf(4), "red": bb.sf(9)} pd.DataFrame(zonas).T.round(3) ``` ```output green amber red 0.01 0.892 0.108 0.000 0.02 0.439 0.531 0.030 0.03 0.128 0.651 0.221 ``` ::: A correct model ends up in green 89.2% of the years. One whose exceptions occur twice as often (2%) still ends up in green 43.9% of the years and only reaches red 3.0% of the time; with three times as many (3%), in green 12.8%. A year in green is compatible with a model that underestimates risk by half: the test has little power with 250 observations. A year in amber does not condemn a correct model either (10.8%). That is why results are accumulated over several years and looked at together with the clustering of exceptions and the ES. :::: 3. **A VaR for a proprietary system.** You have a system with three strategies and want a one-day 99% VaR to set a daily loss limit. Which method among those in the lesson would you choose, with what data would you estimate it and what would you check every month? :::solucion A model that reacts to recent volatility with fat tails: a GARCH-t or an FHS on the aggregate P&L of the system (not on each strategy separately, so that correlations come in on their own), re-estimated with the whole past every few months and computed at the close with data up to that close. The P&L must be that of the system with its positions of each day; if the strategies are new, that of their backtest, knowing that the backtest does not have the costs or the errors of live trading (lesson 4.5.3). Every month: number of exceptions and their clustering (Kupiec and Christoffersen with the whole history, not just the month), the 250-day traffic light, the ES Z2 and a list of each exception with its explanation, as MAR32.12 requires. And, separately, the stress tests of lesson 4.2.4 for what VaR does not see. :::

Each answer is folded under its question

Self-assessment

A one-day 99% VaR of 200,000 USD means that...
- AThe maximum possible loss tomorrow is 200,000 USD
- BWith a correct model, tomorrow's loss exceeds 200,000 USD with a probability of 1%; VaR does not say how much is lost that day
- C200,000 USD is lost one day out of every hundred
- DThe mean loss of the worst days is 200,000 USD
Show answer
Answer B. With a correct model, tomorrow's loss exceeds 200,000 USD with a probability of 1%; VaR does not say how much is lost that day
VaR is a quantile: it marks the boundary of the worst 1%, not its size. What lies beyond is measured by Expected Shortfall, the mean loss conditional on exceeding the VaR.
The 250-session historical VaR has 1.97% exceptions in the data and its mean error versus the true one is 105,800 USD, even though on average it is above the true one. Why does it fail?
- ABecause it uses the normal distribution
- BBecause the window is too long to have enough data
- CBecause it reacts late: it does not rise until the crisis losses enter its window and does not fall until they leave, a year later; exceptions cluster at the start of each crisis
- DBecause the book has a positive mean
Show answer
Answer C. Because it reacts late: it does not rise until the crisis losses enter its window and does not fall until they leave, a year later; exceptions cluster at the start of each crisis
A quantile of 250 equally weighted observations does not distinguish yesterday's volatility from that of eleven months ago. That is why the historical VaR fails Christoffersen's independence test (p 0.000): one exception makes the next one more likely.
A bank has 7 exceptions of its 99% VaR in the last 250 days. According to table 1 of MAR32, what happens?
- ANothing: it is in the green zone
- BIt is in the red zone and loses its internal model
- CThe supervisor forces it to switch to a historical VaR
- DIt is in the amber zone and its backtesting multiplier goes from 1.50 to 1.83
Show answer
Answer D. It is in the amber zone and its backtesting multiplier goes from 1.50 to 1.83
Green from 0 to 4 exceptions (1.50), amber from 5 to 9 (1.70, 1.76, 1.83, 1.88 and 1.92) and red with 10 or more (2.00). With a correct model the amber zone comes up 10.8% of the time: it does not prove that the model is wrong, but it makes it more expensive.
What does the Christoffersen test check that the Kupiec test does not?
- AThat exceptions do not cluster: the probability of an exception must not depend on whether there was one yesterday
- BThat the number of exceptions is the expected one
- CThat the mean loss of the exceptions is correct
- DThat VaR is larger than ES
Show answer
Answer A. That exceptions do not cluster: the probability of an exception must not depend on whether there was one yesterday
Kupiec only looks at frequency. A VaR can hit 1% on average and fail in streaks in every crisis. Christoffersen compares the probability of an exception after a day without one with the probability after a day with one.
With 250 days, the 5% critical value of the Acerbi and Székely Z2 comes out at −0.69 in the simulation; with 4,780 days, −0.16. What does it imply for FHS, which gives Z2 = −0.229 on the full sample?
- AThat it passes: it is above −0.69
- BThat with the full sample its ES is rejected at 5% (−0.229 < −0.16), although in a one-year window that value would not stand out
- CThat the FHS ES is too high
- DThat Z2 does not work for FHS
Show answer
Answer B. That with the full sample its ES is rejected at 5% (−0.229 < −0.16), although in a one-year window that value would not stand out
The critical value depends on the number of days: with more data, smaller deviations are significant. FHS underestimates the tail overall and, with 19 years, this is detected; with one, it is not.

Key points

VaR is a quantile of the loss and ES the mean of what lies beyond it. Basel uses the 99% VaR for backtesting (MAR32) and the 97.5% ES for capital (MAR33).
With the same book, the 250-session historical and normal VaRs have twice as many exceptions as they should, and the error of the historical one versus the true one is 105,800 USD on average: they react late and stay high for a year.
A GJR-GARCH-t estimated each January with the past stays 16,200 USD away from the true VaR on average, passes Kupiec and Christoffersen and spends 88% of the time in green, like the true model.
The square-root-of-10 rule fails when volatility reverts to the mean: the Monte Carlo of the true model gives 945,000 USD at 10 days in October 2008 versus 858,000 with the square root of 10.
The tests have little power with one year of data: a correct model falls into amber 10.8% of the time and one with twice the exceptions often stays in green. You have to look at long series, clustering and ES, not just the count.
The Acerbi-Székely Z2 tests ES with what is stored each day (VaR, ES and P&L): it rejects the historical, normal, EWMA and FHS models, and not the GARCH-t or the true one.

Resources

Documentation

Books

Data

Level 4 data: book P&L with its true VaR and ES (CSV)Download · riesgo4_pyg_verdad.csv

Back to level 4