--- title: "4.2.3 VaR, Expected Shortfall and VaR backtesting" description: "Historical, normal, EWMA, FHS, GARCH-t and Monte Carlo VaR versus the true VaR; Kupiec, Christoffersen, Basel traffic light and Acerbi-Székely." url: "https://pythonparatrading.com/en/python-trading-course/level-4/var-expected-shortfall-and-var-backtesting/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-4/var-expected-shortfall-and-var-backtesting.png" --- # 4.2.3 · VaR, Expected Shortfall and VaR backtesting > VaR sums up in one figure the loss that should almost never be exceeded, and Expected Shortfall says how much is lost when it is. This lesson computes both with six methods on a book whose true VaR is known every day, puts them through the Kupiec, Christoffersen and Acerbi-Székely tests and the Basel Committee traffic light, and measures how wrong each one is and when. - Course: [Level 4: Portfolios, risk, execution and live trading](https://pythonparatrading.com/en/python-trading-course/level-4/) - Module: 4.2 · Risk management and position sizing - Duration: 75 min - Prerequisites: [1.6.2 · Stylized facts: fat tails, clustering, autocorrelation and leverage](https://pythonparatrading.com/en/python-trading-course/level-1/stylized-facts-of-financial-markets/), [2.1.4 · Volatility models: ARCH, GARCH, EGARCH and GJR with the arch library](https://pythonparatrading.com/en/python-trading-course/level-2/volatility-models-arch-garch-egarch-gjr/), [2.5.5 · Monte Carlo: trade resampling, block bootstrap and signal permutations](https://pythonparatrading.com/en/python-trading-course/level-2/monte-carlo-bootstrap-permutation-tests/), [4.2.2 · Kelly criterion and volatility targeting: how much to risk](https://pythonparatrading.com/en/python-trading-course/level-4/kelly-criterion-and-volatility-targeting/) ## By the end of this lesson - Define the VaR and the Expected Shortfall for a horizon and a confidence level, and explain why ES says what VaR keeps quiet. - Compute the 99% VaR with historical, normal, EWMA, filtered historical simulation (FHS) and a GJR-GARCH-t estimated with `arch`, without look-ahead, and compare them with the true one. - Estimate a 10-day VaR by Monte Carlo with a GARCH and see when the square-root-of-10 rule fails. - Apply the Kupiec and Christoffersen tests and the Basel traffic light (MAR32) to a series of exceptions, knowing their power. - Test the 97.5% ES with the Acerbi and Székely Z2 statistic and compute its critical value by simulation. ## What VaR measures and what it does not The **VaR** (*value at risk*) of a book at a horizon $h$ and a confidence level $1-\alpha$ is the loss that is only exceeded with probability $\alpha$. With $L$ the loss (positive when money is lost) over that horizon: $$\mathrm{VaR}_\alpha = \inf\{\ell : P(L > \ell) \le \alpha\}, \qquad \mathrm{ES}_\alpha = \mathrm{E}\left[L \mid L \ge \mathrm{VaR}_\alpha\right].$$ VaR is a quantile: it says where the worst 1% begins, not how much is lost there. **Expected Shortfall** (ES, also CVaR or expected tail loss) is the mean of those losses. Two books with the same VaR can have very different tails: one that sells far out-of-the-money options has a small VaR and an enormous ES (lesson 3.6.1). In addition, VaR is not subadditive (the VaR of a portfolio can exceed the sum of the VaRs of its parts) and ES is: it is a coherent measure in the sense of Artzner, Delbaen, Eber and Heath (1999)[^1]. Regulation uses both. In the Basel framework for market risk, internal model capital is computed with a 97.5% ES (MAR33.3) and **backtesting**, the daily check of the model against the P&L, with a one-day 99% VaR (MAR32.5). A proprietary trading system does not need regulatory capital, but it does need both questions: how much can I lose on a bad day, and has my risk model been able to tell me. ```mermaid flowchart LR P[P&L up to yesterday] --> M[Model] M --> V[Today's VaR and ES] V --> C{Today's loss} C --> E[Exception: yes or no] E --> T[Tests] ``` The order matters: today's VaR is computed at yesterday's close and compared with today's loss. A VaR that uses today's loss to compute today's VaR passes any test and is useless. ## A book with a known true VaR `riesgo4_pyg.csv` is the daily P&L of a book with a constant notional of 10 million USD, 2006-2025. It is synthetic and has a known conditional distribution: a GJR-GARCH(1,1) with standardized Student's t innovations with 5 degrees of freedom, $a = 0.05$, $\gamma = 0.10$, $b = 0.88$, and a long-run variance that changes with the regime of the level's universe (calm, stress, inflation). `riesgo4_pyg_verdad.csv` gives the true volatility of each day, known at the previous close, and the true VaR and ES. The first cell checks where they come from: ```python import numpy as np import pandas as pd from scipy import stats N = 10_000_000 # constant notional (USD) pyg = pd.read_csv("riesgo4_pyg.csv", index_col="fecha", parse_dates=True) verdad = pd.read_csv("riesgo4_pyg_verdad.csv", index_col="fecha", parse_dates=True) r, perdida = pyg["rendimiento"], -pyg["pyg"] # loss > 0 # quantile and ES of the standardized Student's t (variance 1) nu, mu = 5, 0.0003 esc = np.sqrt((nu - 2) / nu) q01 = stats.t.ppf(0.01, nu) * esc q025 = stats.t.ppf(0.025, nu) * esc t025 = stats.t.ppf(0.025, nu) es_t = (nu + t025**2) / (nu - 1) * stats.t.pdf(t025, nu) / 0.025 print(f"q 1%: {q01:.4f} ES 2.5%: {es_t * esc:.4f}") var99 = -(mu + verdad["sigma"] * q01) * N es975 = (-mu + verdad["sigma"] * es_t * esc) * N print("Maximum difference with the file (USD):", round(max((var99 - verdad["var_99"]).abs().max(), (es975 - verdad["es_975"]).abs().max()), 2)) verdad.loc["2020-03-16":"2020-03-18"].round( {"sigma": 5, "var_95": 0, "var_975": 0, "var_99": 0, "es_975": 0}) ``` ```output q 1%: -2.6065 ES 2.5%: 2.7278 Maximum difference with the file (USD): 1.37 sigma var_95 var_975 var_99 es_975 fecha 2020-03-16 0.01155 177307.0 227016.0 298095.0 312112.0 2020-03-17 0.01205 185123.0 236988.0 311147.0 325772.0 2020-03-18 0.01174 180195.0 230700.0 302917.0 317159.0 ``` The 1% quantile of a 5-degree t standardized to variance 1 is −2.6065, 12% further from zero than the normal one (−2.3263), and its 2.5% ES, 2.7278. With $\mathrm{VaR}_{99} = -(\mu + \sigma_t q_{0.01})\cdot N$ the figures in the file are reproduced up to rounding differences of 1.37 USD. In the week of March 16, 2020, the true daily volatility is 1.2% and the 99% VaR, about 300,000 USD. Having the truth allows something that is impossible with real data: to measure the error of each method day by day, not just to count exceptions. ## Six VaRs with what was known each day The usual methods, all computed with data up to the previous close (`shift(1)`): - **Historical**: the 1% quantile of the last 250 returns. It assumes no distribution, but it gives yesterday's session the same weight as the one eleven months ago. - **Normal**: $-\hat\sigma_{250}\, z_{0.01}$ with the 250-session standard deviation. Assuming normality underestimates the tails. - **EWMA** (RiskMetrics): $\hat\sigma^2_t = \lambda\hat\sigma^2_{t-1} + (1-\lambda)r^2_{t-1}$ with $\lambda = 0.94$, the value from the RiskMetrics technical document for daily data, and a normal quantile. - **FHS** (filtered historical simulation, Barone-Adesi et al.): returns are divided by the EWMA volatility of the day before, the quantile of those residuals over 250 sessions is taken and rescaled with today's volatility. It combines the responsiveness of EWMA with the empirical shape of the tails. - **GJR-GARCH-t** with `arch` (lesson 2.1.4): re-estimated each January with the whole past and used with fixed parameters during the year. It is the true model, but with estimated parameters. - **True**: the one in the file, the reference. ```python import warnings from arch import arch_model V = 250 # one-year window z01 = stats.norm.ppf(0.01) lam = 0.94 # RiskMetrics, daily sig_ewma = np.sqrt((r**2).ewm(alpha=1 - lam, adjust=False).mean()) res_ewma = r / sig_ewma.shift(1) # filtered residuals var = pd.DataFrame({ "verdadero": verdad["var_99"], "historico": -r.rolling(V).quantile(0.01).shift(1) * N, "normal": -(r.rolling(V).std() * z01).shift(1) * N, "ewma": -(sig_ewma * z01).shift(1) * N, "fhs": -(sig_ewma.shift(1) * res_ewma.rolling(V).quantile(0.01).shift(1)) * N}) def garch_t(r, inicio=V): """GJR-GARCH(1,1)-t re-estimated each January with the whole past.""" y = 100 * r # arch prefers percentages out, params = [], {} for año in sorted(set(r.index.year)): n = int((r.index.year < año).sum()) if n < inicio: continue am = arch_model(y, mean="Constant", p=1, o=1, q=1, dist="t") with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) fit = am.fit(last_obs=n, disp="off") prev = fit.forecast(start=n - 1, reindex=False).variance dias = r.index[r.index.year == año] s = np.sqrt(prev["h.1"]).shift(1).reindex(dias) / 100 out.append(pd.DataFrame({"sigma": s, "nu": fit.params["nu"], "mu": fit.params["mu"] / 100})) params[año] = fit.params return pd.concat(out), params gt, params = garch_t(r) q_gt = stats.t.ppf(0.01, gt["nu"]) * np.sqrt((gt["nu"] - 2) / gt["nu"]) var["garch_t"] = -(gt["mu"] + gt["sigma"] * q_gt) * N print("2020 fit:", params[2020].round(3).to_dict()) prueba = gt.index # from 2007 to 2025 var = var.loc[prueba] L = perdida.loc[prueba] excesos = L.values[:, None] > var.values pd.DataFrame({"excesos": excesos.sum(axis=0), "pct": 100 * excesos.mean(axis=0), "var_medio": var.mean(), "error_medio_abs": (var.sub(var["verdadero"], axis=0) ).abs().mean(), "sesgo": var.sub(var["verdadero"], axis=0).mean()}, index=var.columns).round(2) ``` ```output 2020 fit: {'mu': 0.036, 'omega': 0.009, 'alpha[1]': 0.061, 'gamma[1]': 0.097, 'beta[1]': 0.887, 'nu': 4.71} excesos pct var_medio error_medio_abs sesgo verdadero 49 1.03 218634.57 0.00 0.00 historico 94 1.97 242182.61 105831.93 23548.03 normal 112 2.34 215419.30 94591.87 -3215.27 ewma 78 1.63 196732.87 36019.11 -21901.70 fhs 69 1.44 226313.08 45680.86 7678.51 garch_t 55 1.15 217830.13 16162.03 -804.45 ``` *Fig. — Exceedance frequency of the one-day 99% VaR for six methods on the same book, 2007-2025 (4,780 sessions); the dashed line is the 1% expected from a correct VaR.* Horizontal bars: True 1.03% (49); Historical 1.97% (94); Normal 2.34% (112); EWMA 0.94 1.63% (78); FHS 1.44% (69); GARCH-t 1.15% (55). Only the true VaR and the estimated GARCH-t (highlighted) stay close to 1%; the normal and historical ones have twice as many exceedances. With 4,780 test sessions (2007-2025), 48 exceptions of a correct 99% VaR are expected. The true one has 49. The estimated GARCH-t, 55 (1.15%), and it stays on average **16,200 USD** away from the true one, with almost zero bias: its 2020 parameters ($a = 0.061$, $\gamma = 0.097$, $b = 0.887$, $\nu = 4.7$) are close to the true ones. EWMA reacts just as fast (mean error of 36,000 USD) but with a normal quantile: 78 exceptions and a bias of −21,900 USD. FHS corrects the tail (69 exceptions). And the two fixed-window methods fail by a wide margin: the normal one has 112 exceptions and the historical one 94, with a mean error of 105,800 USD versus the true one **even though on average it is 23,500 USD above it**. Being above on average is not enough if you are high when it is not needed and low when it is. The header figure shows it in the 2010 stress episode. The true VaR goes from about 150,000 to 1.34 million USD between May and September. The historical one stays at 145,000-160,000 during May and June, rises in steps as the losses enter its window and piles up 15 exceptions in that stretch. Then it stays at 664,000 USD all winter, when the true risk has already gone back to half or less: a year of excess capital tied up behind each crisis. ## Monte Carlo: VaR over several days For one day, the VaR of a GARCH has a formula. For several days it does not: the volatility of the day after tomorrow depends on tomorrow's shock, and the sum of ten GARCH-t returns does not have a known distribution. The way out is to simulate many paths with the model and take the quantile of the sum. The usual alternative, the **square-root-of-10 rule** ($\mathrm{VaR}_{10} \approx \sqrt{10}\,\mathrm{VaR}_1$), assumes independent returns with the same volatility over the ten days. MAR33.4 starts from a 10-day ES to scale to longer liquidity horizons, so the question is not academic. The cell simulates 50,000 10-day paths from three dates, with the true model (with the long-run variance of that day's regime) and with the GARCH-t estimated that year: ```python def mc_suma(mu, omega, a, gam, b, nu, sig1, h=10, n=50_000, semilla=3): """Simulates h days of a GJR-GARCH-t starting from tomorrow's sigma.""" rng = np.random.default_rng(semilla) s2 = np.full(n, sig1**2) total = np.zeros(n) for _ in range(h): z = rng.standard_t(nu, n) * np.sqrt((nu - 2) / nu) eps = np.sqrt(s2) * z total += mu + eps s2 = omega + (a + gam * (eps < 0)) * eps**2 + b * s2 return total fac = pd.read_csv("universo4_regimenes.csv", parse_dates=["desde", "hasta"]) v_reg = {"calma": 0.008, "estres": 0.020, "inflacion": 0.012} filas = {} for dia in ["2017-06-01", "2019-11-01", "2008-10-15"]: t0 = pd.Timestamp(dia) reg = fac[(fac.desde <= t0) & (fac.hasta >= t0)].regimen.item() om_v = (1 - 0.05 - 0.88 - 0.1 / 2) * v_reg[reg] ** 2 sig_v = verdad.at[t0, "sigma"] # sigma of t0, known beforehand sig_g = gt.at[t0, "sigma"] p = params[t0.year] # that year's fit (in %) tot_v = mc_suma(mu, om_v, 0.05, 0.1, 0.88, 5, sig_v) tot_g = mc_suma(p["mu"] / 100, p["omega"] / 1e4, p["alpha[1]"], p["gamma[1]"], p["beta[1]"], p["nu"], sig_g) real = -r.loc[t0:].iloc[:10].sum() * N filas[dia] = {"regimen": reg, "raiz10_verdad": np.sqrt(10) * verdad.at[t0, "var_99"], "mc_verdad": -np.quantile(tot_v, 0.01) * N, "mc_garch": -np.quantile(tot_g, 0.01) * N, "perdida_real": real} tabla = pd.DataFrame(filas).T tabla.iloc[:, 1:] = tabla.iloc[:, 1:].astype(float).round(-3) tabla ``` ```output regimen raiz10_verdad mc_verdad mc_garch perdida_real 2017-06-01 calma 966000.0 949000.0 1066000.0 -545000.0 2019-11-01 calma 441000.0 452000.0 482000.0 -326000.0 2008-10-15 estres 858000.0 945000.0 597000.0 -358000.0 ``` - **In stress** (October 15, 2008), the true Monte Carlo gives 945,000 USD and the square root of 10, 858,000: with $\gamma > 0$ a negative shock raises the next day's volatility more than a positive one, and losses chain together. The square root of 10 falls short precisely when it matters most. - **In calm** the two are similar (966,000 versus 949,000 in June 2017; 441,000 versus 452,000 in November 2019). - **The estimated GARCH** with a single regime is wrong in both directions: in the calm its volatility reverts toward a long-run mean that mixes crises and calm, and it overestimates (1,066,000 and 482,000); in October 2008, with parameters estimated only with 2006-2007, it does not know the level of stress and gives 597,000, 37% less than the truth. On all three dates the book ended up making money over the following ten days (the `perdida_real` column is negative): a VaR is a statement about a distribution, and a single date neither confirms nor refutes it. That is why models are judged with many dates. ## Kupiec, Christoffersen and the clustering of exceptions Each day you record $I_t = 1$ if the loss exceeds the VaR. With a correct VaR, the $I_t$ are independent and equal 1 with probability $p = 1\%$. Two likelihood ratio tests, both $\chi^2$: $$LR_{uc} = -2\ln\frac{(1-p)^{n-x}p^{x}}{(1-\hat p)^{n-x}\hat p^{x}} \sim \chi^2_1, \qquad LR_{cc} = LR_{uc} + LR_{ind} \sim \chi^2_2.$$ The **Kupiec** test (unconditional coverage, $LR_{uc}$) compares the observed frequency $\hat p = x/n$ with $p$. The **Christoffersen** test (independence, $LR_{ind}$) fits a Markov chain to the $I_t$ and compares the probability of an exception after a day without one ($\pi_{01}$) with the probability after a day with one ($\pi_{11}$). Together they give conditional coverage: ```python from scipy.stats import chi2 def kupiec(I, p=0.01): """Unconditional coverage: is the frequency of exceptions p?""" n, x = len(I), int(I.sum()) ph = x / n lr = -2 * ((n - x) * np.log(1 - p) + x * np.log(p) - (n - x) * np.log(1 - ph) - x * np.log(ph)) return lr, chi2.sf(lr, 1) def christoffersen(I): """Independence: does one exception make the next one more likely?""" a, b = I[:-1], I[1:] n00 = ((a == 0) & (b == 0)).sum() n01 = ((a == 0) & (b == 1)).sum() n10 = ((a == 1) & (b == 0)).sum() n11 = ((a == 1) & (b == 1)).sum() p01, p11 = n01 / (n00 + n01), n11 / (n10 + n11) p1 = (n01 + n11) / (n00 + n01 + n10 + n11) ll0 = (n00 + n10) * np.log(1 - p1) + (n01 + n11) * np.log(p1) ll1 = (n00 * np.log(1 - p01) + n01 * np.log(p01) + n10 * np.log(1 - p11) + (n11 * np.log(p11) if n11 else 0)) lr = -2 * (ll0 - ll1) return lr, chi2.sf(lr, 1), p11 filas = {} for j, m in enumerate(var.columns): I = excesos[:, j].astype(int) lr_uc, p_uc = kupiec(I) lr_ind, p_ind, p11 = christoffersen(I) filas[m] = {"excesos": I.sum(), "p_kupiec": p_uc, "p_indep": p_ind, "p_exceso_tras_exceso": p11, "p_condicional": chi2.sf(lr_uc + lr_ind, 2)} pd.DataFrame(filas).T.round(3) ``` ```output excesos p_kupiec p_indep p_exceso_tras_exceso p_condicional verdadero 49.0 0.862 0.314 0.000 0.593 historico 94.0 0.000 0.000 0.096 0.000 normal 112.0 0.000 0.000 0.089 0.000 ewma 78.0 0.000 0.798 0.013 0.000 fhs 69.0 0.004 0.997 0.014 0.015 garch_t 55.0 0.307 0.164 0.036 0.225 ``` The true model passes all three (p of 0.862, 0.314 and 0.593) and never has two exceptions in a row. The estimated GARCH-t also passes (0.307, 0.164, 0.225). The historical and normal models fail both: they have too many exceptions and, after an exception, the probability of another one the next day is 9.6% and 8.9%, nine times what it should be. EWMA and FHS fail only Kupiec: they react well (independence 0.798 and 0.997) but with the wrong tail. ## The Basel traffic light The banking supervisor does not use p-values but a **traffic light** on the number of exceptions of the 99% VaR over the last 250 business days (MAR32.9, table 1). Each zone carries a backtesting capital multiplier: | Zone | Exceptions in 250 days | Multiplier | Probability with a correct model | |---|---|---|---| | Green | 0 to 4 | 1.50 | 89.2% | | Amber | 5, 6, 7, 8, 9 | 1.70, 1.76, 1.83, 1.88, 1.92 | 10.8% | | Red | 10 or more | 2.00 | 0.03% | In red, the supervisor increases the multiplier automatically or may withdraw the model (MAR32.15). The last column comes from a 250-day binomial with $p = 1\%$, in the cell: ```python conteo = pd.DataFrame(excesos, index=prueba, columns=var.columns).rolling(250).sum() conteo = conteo.dropna() zona = pd.cut(conteo.stack(), [-1, 4, 9, 250], labels=["green", "amber", "red"]) reparto = (zona.groupby(level=1, observed=False) .value_counts(normalize=True).unstack() .reindex(var.columns) * 100) print(reparto.round(1)) # with a correct model: binomial(250, 1%) b = stats.binom(250, 0.01) print(f"\nCorrect model: green {b.cdf(4):.1%}, " f"amber {b.cdf(9) - b.cdf(4):.1%}, red {b.sf(9):.2%}") def multiplicador(n): # MAR32, table 1 return {5: 1.70, 6: 1.76, 7: 1.83, 8: 1.88, 9: 1.92}.get(n, 1.50 if n < 5 else 2.00) peor = conteo["historico"].idxmax() print(f"\nWorst window of the historical VaR: 250 days to {peor.date()}") print(pd.DataFrame({"excesos": conteo.loc[peor].astype(int), "multiplicador": conteo.loc[peor].astype(int) .map(multiplicador)}).to_string()) ``` ```output green amber red verdadero 88.0 12.0 0.0 historico 54.2 36.3 9.5 normal 44.8 30.2 25.0 ewma 58.8 41.2 0.0 fhs 74.8 25.2 0.0 garch_t 88.2 11.8 0.0 Correct model: green 89.2%, amber 10.8%, red 0.03% Worst window of the historical VaR: 250 days to 2010-09-29 excesos multiplicador verdadero 4 1.50 historico 15 2.00 normal 16 2.00 ewma 6 1.76 fhs 5 1.70 garch_t 4 1.50 ``` *Fig. — Basel traffic light (MAR32): 99% VaR exceptions over the previous 250 days, for the book's historical and true VaR, 2007-2025.* Two step lines. The true VaR moves between 0 and 7 exceptions and spends 88% of the time in green (0 to 4), as expected from a correct model. The historical one (highlighted) rises after each crisis: it reaches 15 exceptions in 2010 and spends 9.5% of the time in red (10 or more) and 36.3% in amber. The true model spends 88.0% of the days in green and 12.0% in amber, which is what is expected of a correct model. The estimated GARCH-t, the same (88.2% and 11.8%). The historical model spends 9.5% of the time in red and the normal one, 25.0%. EWMA never reaches red but spends 41.2% of the time in amber. In the historical model's worst window, the 250 days to September 29, 2010, the historical model has 15 exceptions and the normal one 16, both with a multiplier of 2.00, while the true model and the GARCH-t have 4 (green). ## Expected Shortfall and its backtest For years it was repeated that ES could not be backtested because it is not “elicitable”. Acerbi and Székely (2014) showed that it can, with statistics that only require storing the VaR, the ES and the P&L each day. The second of them, with $X_t$ the P&L (negative when money is lost) and $I_t = 1[X_t + \mathrm{VaR}_t < 0]$ at the level $\alpha = 2.5\%$: $$ Z_2 = \sum_{t=1}^{T} \frac{X_t\, I_t}{T\,\alpha\,\mathrm{ES}_t} + 1. $$ With a correct ES, $\mathrm{E}[Z_2] = 0$; if the model underestimates the tail, $Z_2 < 0$. The data README gives the same statistic with losses as positive numbers, with the sign flipped. For each method, the ES of its own model: the mean of the tail of 250 observations (historical), the normal formula $\sigma\,\varphi(z_\alpha)/\alpha$, the standardized t formula for the GARCH-t and the mean of the tail of the filtered residuals (FHS): ```python a = 0.025 za = stats.norm.ppf(a) es_norm = stats.norm.pdf(za) / a # ES of the standard normal def es_historico(x): cola = np.sort(x)[: int(np.ceil(a * len(x)))] return -cola.mean() tg = stats.t.ppf(a, gt["nu"]) es_tg = ((gt["nu"] + tg**2) / (gt["nu"] - 1) * stats.t.pdf(tg, gt["nu"]) / a * np.sqrt((gt["nu"] - 2) / gt["nu"])) sig_n = r.rolling(V).std().shift(1) cola_fhs = res_ewma.rolling(V).apply(es_historico, raw=True).shift(1) modelos = { "verdadero": (verdad["var_975"], verdad["es_975"]), "historico": (-r.rolling(V).quantile(a).shift(1) * N, r.rolling(V).apply(es_historico, raw=True) .shift(1) * N), "normal": (-sig_n * za * N, sig_n * es_norm * N), "ewma": (-sig_ewma.shift(1) * za * N, sig_ewma.shift(1) * es_norm * N), "fhs": (-sig_ewma.shift(1) * res_ewma.rolling(V).quantile(a) .shift(1) * N, sig_ewma.shift(1) * cola_fhs * N), "garch_t": ((-(gt["mu"] + gt["sigma"] * tg * np.sqrt((gt["nu"] - 2) / gt["nu"]))) * N, (-gt["mu"] + gt["sigma"] * es_tg) * N)} X = pyg["pyg"].loc[prueba] # P&L (negative = loss) def z2(X, var_a, es_a): """Acerbi-Székely (2014): ≈ 0 if the ES is correct; < 0 if not.""" I = (X + var_a < 0) return (X * I / (len(X) * a * es_a)).sum() + 1 filas = {} for m, (v, e) in modelos.items(): v, e = v.loc[prueba], e.loc[prueba] filas[m] = {"es_medio": e.mean(), "excesos_975_pct": 100 * (X + v < 0).mean(), "Z2": z2(X, v, e)} tabla_es = pd.DataFrame(filas).T tabla_es.round(3) ``` ```output es_medio excesos_975_pct Z2 verdadero 228952.309 2.490 0.004 historico 245963.360 3.703 -0.691 normal 216480.024 3.619 -0.835 ewma 197701.580 3.075 -0.401 fhs 238706.681 2.908 -0.229 garch_t 228438.812 2.573 -0.067 ``` Acerbi and Székely give critical values for $Z_2$ that are almost constant across distributions, around −0.7 at 5% for one year of data. With the true model of the data, the exact critical value can be computed, also for the whole sample: ```python rng = np.random.default_rng(5) sig = verdad["sigma"].loc[prueba].values v_v = verdad["var_975"].loc[prueba].values e_v = verdad["es_975"].loc[prueba].values sim_total, sim_anual = [], [] for _ in range(1000): z = rng.standard_t(nu, len(sig)) * esc Xs = (mu + sig * z) * N I = Xs + v_v < 0 sim_total.append((Xs * I / (len(Xs) * a * e_v)).sum() + 1) k = 250 # a one-year window sim_anual.append((Xs[:k] * I[:k] / (k * a * e_v[:k])).sum() + 1) print(f"5% critical Z2 with {len(sig)} days: " f"{np.quantile(sim_total, 0.05):.3f}") print(f"5% critical Z2 with 250 days: " f"{np.quantile(sim_anual, 0.05):.3f}") ``` ```output 5% critical Z2 with 4780 days: -0.160 5% critical Z2 with 250 days: -0.691 ``` With 250 days the critical value is −0.691, the one in the paper. With the 4,780 test days it is −0.160: with more data, smaller deviations are detected. The true model gives 0.004 and the GARCH-t −0.067: they pass. FHS (−0.229), EWMA (−0.401), historical (−0.691) and normal (−0.835) underestimate the tail and are rejected. The normal model has the worst tail: with a mean ES of 216,500 USD, 5% below the true one, it has 3.62% exceptions of its 97.5% VaR instead of 2.5%, and when it fails, it fails by more. The mean historical ES (246,000 USD) is above the true one and is still rejected, for the same reason as its VaR: it is high when it is not needed. ## What the backtest does not see - **Changing positions.** This book has a constant notional. In a real one, today's P&L comes from positions that yesterday's VaR did not know about if there was intraday trading: that is why MAR32 compares with the actual P&L and with the hypothetical one (yesterday's closing positions revalued), and counts the larger number of exceptions of the two. - **Scenarios that are not in the sample.** A 250-session historical VaR only knows the last year; a GARCH estimated in the calm does not know stress (the October 2008 Monte Carlo). Losses that have never happened are covered by the stress tests of lesson 4.2.4. - **Aggregation.** The VaR of a portfolio depends on correlations, which change in crises (4.2.4). A model that gets the VaR of each strategy right separately can get the VaR of the sum wrong. - **Using VaR as a limit.** If a system reduces positions when its VaR rises, the VaR affects the P&L that judges it. That is desirable, but it has to be taken into account when reading the backtest of a system that uses it. [^1]: Artzner, P., Delbaen, F., Eber, J.-M. and Heath, D. (1999): Coherent Measures of Risk. Mathematical Finance 9(3), 203-228, . Barone-Adesi, G., Giannopoulos, K. and Vosper, L. (1999): VaR without correlations for portfolios of derivative securities. Journal of Futures Markets 19(5), 583-602, . ## Exercises 1. **RiskMetrics for investors.** The RiskMetrics technical document proposed $\lambda = 0.94$ for daily trading data and $\lambda = 0.97$ for investment horizons. Compute the 99% EWMA VaR with 0.97 and with a standardized 5-degree t quantile instead of the normal one, and compare exceptions, Kupiec p and independence p with the normal EWMA with 0.94. **Solution.** ```python sig97 = np.sqrt((r**2).ewm(alpha=0.03, adjust=False).mean()) variantes = {"0.94 normal": -(sig_ewma * z01).shift(1) * N, "0.97 normal": -(sig97 * z01).shift(1) * N, "0.94 t5": -(sig_ewma * q01).shift(1) * N, "0.97 t5": -(sig97 * q01).shift(1) * N} filas = {} for k, v in variantes.items(): I = (L > v.loc[prueba]).astype(int).values filas[k] = {"excesos": I.sum(), "p_kupiec": kupiec(I)[1], "p_indep": christoffersen(I)[1]} pd.DataFrame(filas).T.round(3) ``` ```output excesos p_kupiec p_indep 0.94 normal 78.0 0.000 0.798 0.97 normal 77.0 0.000 0.524 0.94 t5 54.0 0.377 0.267 0.97 t5 48.0 0.977 0.324 ``` Changing λ changes almost nothing (78 to 77 exceptions): the problem of the normal EWMA is not speed, it is the tail. With the 5-degree t quantile, the 0.94 EWMA drops to 54 exceptions (Kupiec p 0.377) and the 0.97 one to 48, exactly the expected number (p 0.977), and both still pass the independence test. An EWMA with fat tails gets close to the GARCH-t without estimating anything; here it helps that the chosen quantile is the true one of the innovations, something that would have to be estimated with real data. 2. **The power of the traffic light.** Compute, with a 250-day binomial, the probability that a model whose exceptions truly occur with a 2% and a 3% probability ends up in green, amber and red. What does this say about a year in green? **Solution.** ```python zonas = {} for p_real in (0.01, 0.02, 0.03): bb = stats.binom(250, p_real) zonas[p_real] = {"green": bb.cdf(4), "amber": bb.cdf(9) - bb.cdf(4), "red": bb.sf(9)} pd.DataFrame(zonas).T.round(3) ``` ```output green amber red 0.01 0.892 0.108 0.000 0.02 0.439 0.531 0.030 0.03 0.128 0.651 0.221 ``` A correct model ends up in green 89.2% of the years. One whose exceptions occur twice as often (2%) still ends up in green 43.9% of the years and only reaches red 3.0% of the time; with three times as many (3%), in green 12.8%. A year in green is compatible with a model that underestimates risk by half: the test has little power with 250 observations. A year in amber does not condemn a correct model either (10.8%). That is why results are accumulated over several years and looked at together with the clustering of exceptions and the ES. 3. **A VaR for a proprietary system.** You have a system with three strategies and want a one-day 99% VaR to set a daily loss limit. Which method among those in the lesson would you choose, with what data would you estimate it and what would you check every month? **Solution.** A model that reacts to recent volatility with fat tails: a GARCH-t or an FHS on the aggregate P&L of the system (not on each strategy separately, so that correlations come in on their own), re-estimated with the whole past every few months and computed at the close with data up to that close. The P&L must be that of the system with its positions of each day; if the strategies are new, that of their backtest, knowing that the backtest does not have the costs or the errors of live trading (lesson 4.5.3). Every month: number of exceptions and their clustering (Kupiec and Christoffersen with the whole history, not just the month), the 250-day traffic light, the ES Z2 and a list of each exception with its explanation, as MAR32.12 requires. And, separately, the stress tests of lesson 4.2.4 for what VaR does not see. ## Self-assessment 1. A one-day 99% VaR of 200,000 USD means that... - A. The maximum possible loss tomorrow is 200,000 USD - B. With a correct model, tomorrow's loss exceeds 200,000 USD with a probability of 1%; VaR does not say how much is lost that day - C. 200,000 USD is lost one day out of every hundred - D. The mean loss of the worst days is 200,000 USD *Answer B: With a correct model, tomorrow's loss exceeds 200,000 USD with a probability of 1%; VaR does not say how much is lost that day.* VaR is a quantile: it marks the boundary of the worst 1%, not its size. What lies beyond is measured by Expected Shortfall, the mean loss conditional on exceeding the VaR. 2. The 250-session historical VaR has 1.97% exceptions in the data and its mean error versus the true one is 105,800 USD, even though on average it is above the true one. Why does it fail? - A. Because it uses the normal distribution - B. Because the window is too long to have enough data - C. Because it reacts late: it does not rise until the crisis losses enter its window and does not fall until they leave, a year later; exceptions cluster at the start of each crisis - D. Because the book has a positive mean *Answer C: Because it reacts late: it does not rise until the crisis losses enter its window and does not fall until they leave, a year later; exceptions cluster at the start of each crisis.* A quantile of 250 equally weighted observations does not distinguish yesterday's volatility from that of eleven months ago. That is why the historical VaR fails Christoffersen's independence test (p 0.000): one exception makes the next one more likely. 3. A bank has 7 exceptions of its 99% VaR in the last 250 days. According to table 1 of MAR32, what happens? - A. Nothing: it is in the green zone - B. It is in the red zone and loses its internal model - C. The supervisor forces it to switch to a historical VaR - D. It is in the amber zone and its backtesting multiplier goes from 1.50 to 1.83 *Answer D: It is in the amber zone and its backtesting multiplier goes from 1.50 to 1.83.* Green from 0 to 4 exceptions (1.50), amber from 5 to 9 (1.70, 1.76, 1.83, 1.88 and 1.92) and red with 10 or more (2.00). With a correct model the amber zone comes up 10.8% of the time: it does not prove that the model is wrong, but it makes it more expensive. 4. What does the Christoffersen test check that the Kupiec test does not? - A. That exceptions do not cluster: the probability of an exception must not depend on whether there was one yesterday - B. That the number of exceptions is the expected one - C. That the mean loss of the exceptions is correct - D. That VaR is larger than ES *Answer A: That exceptions do not cluster: the probability of an exception must not depend on whether there was one yesterday.* Kupiec only looks at frequency. A VaR can hit 1% on average and fail in streaks in every crisis. Christoffersen compares the probability of an exception after a day without one with the probability after a day with one. 5. With 250 days, the 5% critical value of the Acerbi and Székely Z2 comes out at −0.69 in the simulation; with 4,780 days, −0.16. What does it imply for FHS, which gives Z2 = −0.229 on the full sample? - A. That it passes: it is above −0.69 - B. That with the full sample its ES is rejected at 5% (−0.229 < −0.16), although in a one-year window that value would not stand out - C. That the FHS ES is too high - D. That Z2 does not work for FHS *Answer B: That with the full sample its ES is rejected at 5% (−0.229 < −0.16), although in a one-year window that value would not stand out.* The critical value depends on the number of days: with more data, smaller deviations are significant. FHS underestimates the tail overall and, with 19 years, this is detected; with one, it is not. ## Key points - VaR is a quantile of the loss and ES the mean of what lies beyond it. Basel uses the 99% VaR for backtesting (MAR32) and the 97.5% ES for capital (MAR33). - With the same book, the 250-session historical and normal VaRs have twice as many exceptions as they should, and the error of the historical one versus the true one is 105,800 USD on average: they react late and stay high for a year. - A GJR-GARCH-t estimated each January with the past stays 16,200 USD away from the true VaR on average, passes Kupiec and Christoffersen and spends 88% of the time in green, like the true model. - The square-root-of-10 rule fails when volatility reverts to the mean: the Monte Carlo of the true model gives 945,000 USD at 10 days in October 2008 versus 858,000 with the square root of 10. - The tests have little power with one year of data: a correct model falls into amber 10.8% of the time and one with twice the exceptions often stays in green. You have to look at long series, clustering and ES, not just the count. - The Acerbi-Székely Z2 tests ES with what is stored each day (VaR, ES and P&L): it rejects the historical, normal, EWMA and FHS models, and not the GARCH-t or the true one. ## Resources ### Documentation - [BCBS, MAR32: backtesting and profit and loss attribution test of the internal model (version in force since January 1, 2023)](https://www.bis.org/committees/bcbs/basel-framework/81326/chapter.pdf) · bis.org - [BCBS, MAR33: capital calculation with internal models (ES at 97.5%, MAR33.3)](https://www.bis.org/committees/bcbs/basel-framework/81327/chapter.pdf) · bis.org - [J.P. Morgan/Reuters (1996): RiskMetrics Technical Document, 4th ed.](https://www.msci.com/documents/10199/5915b101-4206-4ba0-aee2-3449d5c7e95a) · msci.com ### Books - [Acerbi, C. and Szekely, B. (2014): Backtesting Expected Shortfall. MSCI Research Insight](https://www.msci.com/documents/10199/22aa9922-f874-4060-b77a-0f0e267a489b) · msci.com - [Kupiec, P. H. (1995): Techniques for Verifying the Accuracy of Risk Measurement Models. The Journal of Derivatives 3(2), 73-84](https://doi.org/10.3905/jod.1995.407942) · doi.org - [Christoffersen, P. F. (1998): Evaluating Interval Forecasts. International Economic Review 39(4), 841-862](https://doi.org/10.2307/2527341) · doi.org ### Data - [Level 4 data: book P&L with its true VaR and ES (CSV)](https://pythonparatrading.com/curso/datos/nivel-4/riesgo4_pyg_verdad.csv) · Download · riesgo4_pyg_verdad.csv Previous lesson: [4.2.2 · Kelly criterion and volatility targeting: how much to risk](https://pythonparatrading.com/en/python-trading-course/level-4/kelly-criterion-and-volatility-targeting/) Next lesson: [4.2.4 · Stress testing and correlation, liquidity, gap and counterparty risk](https://pythonparatrading.com/en/python-trading-course/level-4/stress-testing-correlation-liquidity-counterparty-risk/)