Returns, volatility, rolling correlations and drawdown.

Two indices, seven years

In level 0 you computed returns, the volatility of a 21-session window and the drawdown of a fictional ETF (lessons 0.5.3 and 0.7.1). This lesson starts from there and moves on to what matters for trading: how volatility changes, with what lag each estimator detects it, how far you can trust a rolling correlation and what a drawdown looks like from the inside.

The data are two fictional level 1 indices, with the real NYSE sessions from 2019 to 2025: ISX, a total return index (etf_indice_diario.csv), and FIX, a futures-based price index (fix_indice_diario.csv). Both have episodes of high volatility in 2020, 2022 and the spring of 2025.

import numpy as np
import pandas as pd

isx = pd.read_csv("etf_indice_diario.csv", parse_dates=["fecha"],
                  index_col="fecha")["indice"]
fix = pd.read_csv("fix_indice_diario.csv", parse_dates=["fecha"],
                  index_col="fecha")["indice"]
px = pd.DataFrame({"ISX": isx, "FIX": fix})
r = np.log(px).diff().dropna()   # log returns
print(px.index[0].date(), "to", px.index[-1].date(), len(px), "sessions")
anual = pd.DataFrame({
    "media_%": 100 * 252 * r.mean(),
    "vol_%": 100 * np.sqrt(252) * r.std(),
    "peor_dia_%": 100 * r.min(),
    "mejor_dia_%": 100 * r.max()})
anual.round(2)

2019-01-02 to 2025-12-31 1760 sessions
     media_%  vol_%  peor_dia_%  mejor_dia_%
ISX    12.83  20.87       -7.95         7.84
FIX    14.55  20.07       -8.98         8.15

Both have an annual volatility of around 20%, worst days of −8% and −9%, best days of +8% and mean returns of 12.8% and 14.6% (in log terms, annualized by 252). A volatility over the whole sample says little: it is the average of very quiet years and very turbulent months. What is useful is the series.

Fixed-window volatility

The historical volatility of n sessions is the standard deviation of the last n returns, annualized by √252. The size of the window is a trade-off: short, noisy and fast; long, stable and slow.

x = r["ISX"]
vol = pd.DataFrame({
    f"v{n}": x.rolling(n).std() * np.sqrt(252)
    for n in (21, 63, 252)})
(100 * vol.loc["2020-02-14":"2020-05-29"].iloc[::10]).round(1)

             v21   v63  v252
fecha
2020-02-14  10.6  12.1  11.4
2020-03-02  17.5  13.2  11.7
2020-03-16  22.7  17.9  13.1
2020-03-30  27.7  23.4  15.1
2020-04-14  34.3  26.1  16.7
2020-04-28  38.2  29.8  18.4
2020-05-12  45.5  36.7  20.9
2020-05-27  37.2  38.6  21.7

In mid-February 2020 all three windows see a calm market, at around 11-12%. The 21-session window reaches 34.3% in mid-April and 45.5% in May; the 63-session one, 26.1% and 36.7%; the 252-session one, which carries a year of quiet days inside it, does not go above 21.7% over the whole period. A rectangular window weighs all its days equally: until the quiet days leave it, it underestimates the new regime.

EWMA: weight decays with age

The exponentially weighted moving average (EWMA) of squared returns gives each day a weight that decays geometrically. It is the one used by RiskMetrics, J.P. Morgan's risk methodology from the nineties, which set λ = 0.94 for daily data and λ = 0.97 for monthly data11Note 1 · J.P. Morgan/Reuters, RiskMetrics — Technical Document, 4th edition, 1996: correlations and volatilities with exponential averages and decay factors of 0.94 (daily data, “74 day cutoff 1%”) and 0.97 (monthly).:

Yesterday's return weighs 1 − λ = 6%; the day before's, 6% × 0.94; the one from k days ago, 0.06 × 0.94^(k−1). RiskMetrics assumes a zero mean, reasonable with daily data, and that is why it uses r² instead of (r − mean)².

lam = 0.94
k = np.arange(1, 253)
w = (1 - lam) * lam ** (k - 1)
print(f"Half-life: {np.log(0.5) / np.log(lam):.1f} sessions")
for n in (10, 21, 63, 74):
    print(f"Weight of the last {n}: {w[:n].sum():.1%}")

Half-life: 11.2 sessions
Weight of the last 10: 46.1%
Weight of the last 21: 72.7%
Weight of the last 63: 98.0%
Weight of the last 74: 99.0%

Half of the weight is in the last eleven sessions and 99% in the last 74 (the cutoff figure that the RiskMetrics document itself gives for λ = 0.94). A 63-session window gives 1/63 = 1.6% to each day and 0 to the 64th: the day that leaves the window causes a jump in the estimate even if nothing happens today.

The recursion can be written as a loop or with ewm. With adjust=False, pandas applies the recursion exactly; with alpha = 1 − λ:

def ewma_var(x, lam=0.94, var0=None):
    """EWMA variance for each day with data up to the previous day."""
    v = np.empty(len(x))
    v[0] = np.var(x[:21]) if var0 is None else var0
    for t in range(1, len(x)):
        v[t] = lam * v[t - 1] + (1 - lam) * x[t - 1] ** 2
    return v

xa = x.to_numpy()
a_mano = ewma_var(xa)
# ewm(adjust=False) includes day t: shift by one
con_ewm = (x.pow(2).ewm(alpha=0.06, adjust=False).mean()
           .shift(1).to_numpy())
dif = np.abs(np.sqrt(a_mano[300:]) - np.sqrt(con_ewm[300:]))
print(f"Maximum difference after 300 sessions: {dif.max():.2e}")
ewma = pd.Series(np.sqrt(252 * a_mano), index=x.index)
vol["ewma"] = ewma
(100 * vol.loc["2020-02-14":"2020-05-29"].iloc[::10]).round(1)

Maximum difference after 300 sessions: 2.32e-11
             v21   v63  v252  ewma
fecha
2020-02-14  10.6  12.1  11.4  10.6
2020-03-02  17.5  13.2  11.7  17.4
2020-03-16  22.7  17.9  13.1  26.5
2020-03-30  27.7  23.4  15.1  34.5
2020-04-14  34.3  26.1  16.7  33.7
2020-04-28  38.2  29.8  18.4  37.4
2020-05-12  45.5  36.7  20.9  40.3
2020-05-27  37.2  38.6  21.7  37.7

The two versions match except for the initial value, whose effect disappears within a few weeks (0.94³⁰⁰ is negligible). And the difference in timing is in the table: the EWMA goes from 10.6% to 34.5% in six weeks and settles between 34% and 40% while the 21-session window keeps rising to 45.5%.

The figure at the top draws it day by day against the 63-session window, both with data up to the previous day. When does each one cross 30%?

v63 = (x.rolling(63).std() * np.sqrt(252)).shift(1)
for nombre, serie in [("EWMA", ewma), ("Window 63", v63)]:
    s = serie.loc["2020"]
    print(f"{nombre:>10}: 30% on {s[s >= 0.30].index[0].date()},"
          f" maximum {s.max():.1%} on {s.idxmax().date()}")

      EWMA: 30% on 2020-03-24, maximum 51.1% on 2020-06-16
 Window 63: 30% on 2020-04-30, maximum 45.1% on 2020-06-22

The EWMA crosses 30% on March 24; the 63-session window, on April 30, five weeks later. A system that reduces exposure when volatility exceeds a threshold would have reacted more than a month apart depending on the estimator. On the way out, the same thing happens in reverse: the EWMA returns to low levels sooner and restores exposure sooner.

One detail of the cell matters more than the rest: the variance for day t uses returns up to t − 1. If volatility is used to size today's position (volatility targeting, lesson 0.7.3), the estimator cannot include today's return, which has not happened yet when the decision is made. rolling(...).std() and ewm(...).mean() include day t: in a backtest, always shift(1).

flowchart LR
  R[r up to t-1] --> S[σ estimated at t]
  S --> P[Position at t]
  P --> G[Outcome r at t]

Which estimator forecasts better

A volatility estimator is a forecast: it says how much the price will move tomorrow. It can be evaluated like any forecast, comparing the estimated variance with the next day's squared return. Two common measures: the mean squared error and the QLIKE loss, which penalizes falling short more than overshooting and is robust to the noise in r²:

Another, more intuitive way: divide each return by the forecast volatility. If the estimator is good, the standardized returns have a standard deviation close to 1, the lowest possible kurtosis and a share of days below −2.33 standard deviations close to what the normal distribution expects (1%).

prev = pd.DataFrame({
    "v21": x.rolling(21).std(), "v63": x.rolling(63).std(),
    "v252": x.rolling(252).std(),
    "ewma94": np.sqrt(x.pow(2).ewm(alpha=0.06, adjust=False).mean()),
    "ewma97": np.sqrt(x.pow(2).ewm(alpha=0.03, adjust=False).mean()),
}).shift(1)                      # forecast for the next day
d = prev.join(x.rename("r")).dropna()
filas = {}
for c in prev:
    s2, r2 = d[c] ** 2, d["r"] ** 2
    z = d["r"] / d[c]
    filas[c] = {"qlike": (r2 / s2 - np.log(r2 / s2) - 1).mean(),
                "z_desv": z.std(), "z_curtosis": z.kurt(),
                "bajo_-2,33_%": 100 * (z < -2.326).mean()}
pd.DataFrame(filas).T.round(3)

        qlike  z_desv  z_curtosis  bajo_-2,33_%
v21     1.818   1.195      10.664         2.190
v63     1.827   1.122       9.979         2.057
v252    2.180   1.193      11.176         2.588
ewma94  1.696   1.088       7.326         1.924
ewma97  1.751   1.061       9.304         1.924

The EWMA with λ = 0.94 has the lowest QLIKE (1.696) and the lowest kurtosis (7.3 versus 10-11 for the windows); the one with λ = 0.97 brings the standard deviation closest to 1 (1.06 versus 1.09), and the two tie on days below −2.33 (1.9%). The windows lag behind on everything. None gets down to 1% of days below −2.33: the tails are still fatter than the normal's even after standardizing (lesson 1.6.2). The 252-session window is the worst on QLIKE, kurtosis and days below −2.33: a “last year” volatility is a poor forecast of tomorrow's.

Rolling correlations: signal and noise

The correlation between two assets also changes, and a diversified portfolio depends on it. The 63-session rolling correlation between ISX and FIX:

rc = r["ISX"].rolling(63).corr(r["FIX"])
banda = 1.96 / np.sqrt(63)
print(f"Full-sample correlation: {r.corr().iloc[0, 1]:.3f}")
print(rc.describe().round(3).to_string())
print(f"95% band if the true correlation is 0: ±{banda:.3f}")
print(f"Outside the band: {(rc.abs() > banda).mean():.1%}")
print("Maximum:", rc.idxmax().date(), round(rc.max(), 3))

Full-sample correlation: 0.066
count    1697.000
mean        0.010
std         0.155
min        -0.438
25%        -0.080
50%        -0.006
75%         0.099
max         0.511
95% band if the true correlation is 0: ±0.247
Outside the band: 10.1%
Maximum: 2025-04-23 0.511

The full-sample correlation is almost zero and the rolling one oscillates from −0.44 to +0.51. Is the relationship between the two indices changing, or is it noise? With 63 data points and a true correlation of zero, 95% of the estimates fall within ±0.247 (standard error ≈ 1/√n). Here it falls outside 10.1% of the time: double the 5% that chance would give, but far from what would be needed to read each peak as a change. The maximum, 0.51, is on April 23, 2025, in the middle of the fall of both indices, and the March 2020 peak reaches almost 0.50: the two stress episodes are the two peaks.

To check how much a rolling correlation oscillates with no relationship at all, it is enough to simulate it:

rng = np.random.default_rng(7)
ind = pd.DataFrame(rng.standard_normal((len(r), 2)))
rc0 = ind[0].rolling(63).corr(ind[1])
print(f"Minimum {rc0.min():.3f}, maximum {rc0.max():.3f}")
print(f"Outside the band: {(rc0.abs() > banda).mean():.1%}")

Minimum -0.255, maximum 0.300
Outside the band: 1.2%

Two series with no relationship at all produce a rolling correlation that ranges from −0.26 to +0.30. In this simulation only 1.2% of the time falls outside the band: with overlapping windows, the values on consecutive days are not independent and the fraction for a single path varies a lot around 5%. Every rolling correlation chart has to be read with that reference next to it.

And the stress periods? In real markets it is often observed that correlations between risk assets rise during selloffs, exactly when diversification would be most needed. There is a trace of that in this data, but it comes from frequency, not from day-to-day moves:

for regla, nombre in [("W-FRI", "weekly"), ("ME", "monthly"),
                      ("QE", "quarterly")]:
    rr = np.log(px.resample(regla).last()).diff().dropna()
    print(f"{nombre:>10}: {rr.corr().iloc[0, 1]:.3f}"
          f"  ({len(rr)} data points)")
vol21 = r.rolling(21).std()
print(f"Correlation of the 21-session volatilities: "
      f"{vol21['ISX'].corr(vol21['FIX']):.3f}")

    weekly: 0.156  (365 data points)
   monthly: 0.453  (83 data points)
 quarterly: 0.819  (27 data points)
Correlation of the 21-session volatilities: 0.650

At daily frequency the correlation is 0.06; weekly, 0.16; monthly, 0.45, and quarterly, 0.82. The two indices share the big calendar trends (the falls of 2020, 2022 and 2025 and the subsequent rallies), but not the moves of each day, which in this data are independent. When aggregating, the daily noise cancels out and the common part remains. And their volatilities do move together (0.65): the high-volatility regimes are the same. For a portfolio, the consequence is clear: the diversification measured with daily data may not exist at the horizon that really matters, that of months.

Drawdown and time underwater

Each day's drawdown is the fall from the previous peak:

Its minimum is the maximum drawdown, the figure everybody quotes. But a drawdown has two dimensions: how far it falls and how long it lasts. An underwater episode starts on the day the price falls below its peak and ends on the day it exceeds it again.

stateDiagram-v2
  [*] --> AtPeak
  AtPeak --> Underwater: falls from the peak
  Underwater --> Underwater: stays below
  Underwater --> AtPeak: exceeds the peak

def episodios(precio):
    pico = precio.cummax()
    dd = precio / pico - 1
    bajo = dd < 0
    grupo = (~bajo).cumsum()         # one number per episode
    filas = []
    for _, e in dd[bajo].groupby(grupo[bajo]):
        fin = e.index[-1]
        sig = precio.index.searchsorted(fin, side="right")
        recuperado = sig < len(precio)
        filas.append({
            "inicio": e.index[0].date(),
            "minimo": e.idxmin().date(),
            "recupera": precio.index[sig].date()
                        if recuperado else None,
            "profundidad_%": 100 * e.min(),
            "sesiones_hasta_minimo": e.index.get_loc(e.idxmin()) + 1,
            "sesiones_bajo_agua": len(e)})
    return pd.DataFrame(filas), dd

ep, dd_isx = episodios(px["ISX"])
print(f"Sessions underwater: {(dd_isx < 0).mean():.1%}")
print(f"Episodes: {len(ep)}")
ep.sort_values("profundidad_%").head(5).round(1)

Sessions underwater: 92.9%
Episodes: 54
        inicio      minimo    recupera  profundidad_%  sesiones_hasta_minimo  sesiones_bajo_agua
22  2020-02-20  2020-04-21  2021-03-29          -43.2                     43                 278
31  2022-02-15  2022-05-25  2023-04-17          -28.9                     70                 292
41  2025-02-20  2025-04-04  2025-06-25          -28.0                     32                  86
35  2023-07-17  2024-01-02  2024-10-03          -18.6                    118                 307
36  2024-10-09  2024-11-26  2025-01-13           -7.2                     35                  64

92.9% of ISX sessions are below the previous peak: you are almost always “underwater”, even if only slightly. What sets an episode apart is its depth and its duration, and they do not go together:

ep.sort_values("sesiones_bajo_agua", ascending=False).head(4)[
    ["inicio", "recupera", "profundidad_%",
     "sesiones_hasta_minimo", "sesiones_bajo_agua"]].round(1)

        inicio    recupera  profundidad_%  sesiones_hasta_minimo  sesiones_bajo_agua
35  2023-07-17  2024-10-03          -18.6                    118                 307
31  2022-02-15  2023-04-17          -28.9                     70                 292
22  2020-02-20  2021-03-29          -43.2                     43                 278
41  2025-02-20  2025-06-25          -28.0                     32                  86

The 2020 crash is the deepest (−43.2%) but takes 278 sessions to recover, fewer than the episode that starts in July 2023, only −18.6% deep and with 307 sessions underwater: a moderate fall followed by a slow recovery. The 2022 fall lasts 292. For someone following a strategy, fourteen months without a new high weigh as much as a big fall: that is when systems get abandoned. That is why the level 2 risk metrics (lesson 2.6.1) include drawdown duration and the Ulcer Index, which combines depth and duration.

What this changes in a system

• Volatility with shift(1). Every estimate that feeds a decision on day t uses data up to t − 1. With pandas' rolling and ewm, day t is inside: shift it.
• Choose the estimator by its use. For sizing positions day by day, a fast one (EWMA, λ = 0.94, or a short window); for comparing strategies or setting risk limits, a slow and stable one.
• No correlation without its band. With n data points, the noise is on the order of ±2/√n. And measure the correlation at the frequency of your horizon, not just daily.
• Depth and duration. Report the maximum drawdown and the longest underwater episode. They are different risks.

Exercises

1. Repeat the estimator evaluation with FIX and with λ = 0.90, 0.94 and 0.97. Which λ gives the lowest QLIKE for each index?

def qlike(serie, lam):
    s2 = serie.pow(2).ewm(alpha=1 - lam, adjust=False).mean()
    dd = pd.DataFrame({"s2": s2.shift(1),
                       "r2": serie.pow(2)}).iloc[63:]
    dd = dd[dd["r2"] > 0]   # days with no change: ln(0)
    q = dd["r2"] / dd["s2"]
    return (q - np.log(q) - 1).mean()

tabla = pd.DataFrame({ind: {lam: qlike(r[ind], lam)
                            for lam in (0.90, 0.94, 0.97)}
                      for ind in ["ISX", "FIX"]})
tabla.round(4)

         ISX     FIX
0.90  1.6955  1.6794
0.94  1.6973  1.6746
0.97  1.7454  1.7173

2. Compute the EWMA correlation (λ = 0.94) between ISX and FIX and compare it with the 63-session window one: which oscillates more?

a = 0.06
cov = (r["ISX"] * r["FIX"]).ewm(alpha=a, adjust=False).mean()
var_i = r["ISX"].pow(2).ewm(alpha=a, adjust=False).mean()
var_f = r["FIX"].pow(2).ewm(alpha=a, adjust=False).mean()
corr_ewma = (cov / np.sqrt(var_i * var_f)).iloc[63:]
pd.DataFrame({"ventana_63": rc.iloc[63:].describe(),
              "ewma_094": corr_ewma.describe()}).round(3)

       ventana_63  ewma_094
count    1696.000  1696.000
mean        0.010     0.020
std         0.155     0.182
min        -0.438    -0.517
25%        -0.080    -0.083
50%        -0.006     0.006
75%         0.099     0.110
max         0.511     0.699

3. One strategy has a maximum drawdown of −15% and another of −25%. What else do you need to know before saying which one is less risky?