Lesson 2.6.1 · 65 min

CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index

An equity curve is summarized in a few figures, and each one hides a convention: what counts as a year, which risk-free rate is subtracted, which part of the distribution is looked at. This lesson implements them from scratch on three strategies with very different return shapes and checks that they match empyrical-reloaded.

Fig. — Three strategies on level 2 synthetic data, February 2005 to 2025: cumulative wealth on a log scale
Level 2 lessons
  1. 2.1 · Statistics and econometrics for trading

    1. 2.1.1 Hypothesis tests, p-values, bootstrap and regression with robust standard errors
    2. 2.1.2 Stationarity: ADF, KPSS, differencing and fractional differencing
    3. 2.1.3 ARMA and ARIMA models: autocorrelation (ACF) and partial autocorrelation (PACF)
    4. 2.1.4 Volatility models: ARCH, GARCH, EGARCH and GJR with the arch library
    5. 2.1.5 Cointegration: Engle-Granger, Johansen and the half-life of mean reversion
    6. 2.1.6 Kalman filter for dynamic hedge ratios
    7. 2.1.7 Regime detection: hidden Markov models and structural breaks
    8. 2.1.8 The multiple testing problem: Bonferroni, FDR and the “factor zoo”
  2. 2.2 · Anatomy of a trading system

    1. 2.2.1 Edge hypotheses: risk, behavior, structure and information
    2. 2.2.2 Universe, signal, filter, entry, exit, sizing and risk management
    3. 2.2.3 Frequencies: intraday, swing and position trading, and the capacity of a strategy
    4. 2.2.4 Research journal and experiment tracking against data snooping
  3. 2.3 · Strategy families

    1. 2.3.1 Trend following and momentum
    2. 2.3.2 Mean reversion: RSI and Bollinger, intraday reversal and gaps
    3. 2.3.3 Pairs trading and statistical arbitrage: distance, cointegration and Kalman
    4. 2.3.4 Carry in currencies, futures and bonds; seasonality and calendar effects
    5. 2.3.5 Factors (value, quality, low volatility, size) and event strategies
    6. 2.3.6 Introduction to market making: inventory, spread and adverse selection (Avellaneda-Stoikov)
    7. 2.3.7 Crypto-specific strategies: momentum, funding arbitrage and basis
  4. 2.4 · Backtesting: engines and paradigms

    1. 2.4.1 Vectorized backtest from scratch: matrices, lag, costs and limits
    2. 2.4.2 Event-driven backtest from scratch: loop, portfolio, orders and fills
    3. 2.4.3 vectorbt: massive parameter sweeps and portfolio simulation
    4. 2.4.4 backtesting.py and backtrader: two teaching engines and their status
    5. 2.4.5 NautilusTrader: architecture, data, strategies and the same code in backtest and live
    6. 2.4.6 LEAN/QuantConnect and Zipline Reloaded; bt for asset allocation
    7. 2.4.7 Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing
    8. 2.4.8 Fill modeling: limit orders, queue, partial fills, gaps and stops
  5. 2.5 · Biases and validation

    1. 2.5.1 Backtest biases: look-ahead, survivorship, selection and narrative
    2. 2.5.2 In-sample and out-of-sample: anchored and rolling walk-forward
    3. 2.5.3 Combinatorial purged cross-validation for strategies
    4. 2.5.4 Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)
    5. 2.5.5 Monte Carlo: trade resampling, block bootstrap and signal permutations
    6. 2.5.6 Parameter sensitivity: plateaus versus peaks, robustness across assets and periods
    7. 2.5.7 Synthetic data for stress testing: GBM, GARCH and bootstrapping
    8. 2.5.8 “Honest backtest” checklist before moving to paper trading
  6. 2.6 · Performance metrics

    1. 2.6.1 CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index
    2. 2.6.2 Win rate, payoff, expectancy, alpha, beta, information ratio, turnover and capacity
    3. 2.6.3 Reports with QuantStats or pyfolio-reloaded and custom tearsheets

Three ways of making money

A backtest returns a series of returns, and almost everything that is said about a strategy comes from summarizing it in a few figures. Each figure looks at a different aspect of the series, and to see what each one looks at it pays to measure them on strategies that make money in different ways. This lesson builds three on the level 2 data:

  • IDA index. Long the continuous future of a fictitious equity index, bought and held. It is the benchmark: it earns the equity premium and suffers all its crises.
  • TSMOM on futures. Time-series momentum on the 24 markets of futuros2_continuos.parquet: long if the market has risen over the last 252 sessions, short if it has fallen, with each position sized for 40% annual volatility divided by 24, as in Moskowitz, Ooi and Pedersen (2012)1Note 1 · Tobias J. Moskowitz, Yao Hua Ooi and Lasse Heje Pedersen, “Time series momentum”, Journal of Financial Economics 104(2), 228-250 (2012), https://doi.org/10.1016/j.jfineco.2011.11.003.. Its typical shape: many small losses when there is no trend and large gains when there is one.
  • Currency carry. At each month-end, long the three currencies with the highest interest rate and short the three with the lowest in fx2_carry.parquet. It collects the rate differential almost every day and gives it back all at once in the crashes.
IN [01]python
import numpy as np
import pandas as pd

# 1) Futures: long IDA and time-series momentum (TSMOM)
fut = pd.read_parquet("futuros2_continuos.parquet")
px = fut.pivot(index="fecha", columns="mercado",
               values="cierre_ajustado")
r_fut = px.pct_change()
vol = r_fut.ewm(span=60, min_periods=60).std() * np.sqrt(252)
senal = np.sign(px / px.shift(252) - 1)
peso = (senal * 0.40 / px.shape[1] / vol).shift(1)
tsmom = (peso * r_fut).sum(axis=1, min_count=1)

# 2) Currency carry: long the 3 with the highest rate, short 3
fx = pd.read_parquet("fx2_carry.parquet")
tabla = lambda c: fx.pivot(index="fecha", columns="divisa", values=c)
spot, dif = tabla("spot"), tabla("tipo_divisa") - tabla("tipo_usd")
dias = spot.index.to_series().diff().dt.days
interes = dif.shift(1) / 100 * dias.to_numpy()[:, None] / 360
r_fx = spot.pct_change() + interes    # spot + accrued differential
fin_mes = dif.index.to_series().groupby(dif.index.to_period("M")).max()
rango = dif.loc[fin_mes].rank(axis=1)
w_fx = ((rango >= 8).astype(float) - (rango <= 3).astype(float)) / 3
w_fx = w_fx.reindex(dif.index).ffill().shift(1)
carry = (w_fx * r_fx).sum(axis=1, min_count=1)

inicio = "2005-02-01"
rets = {"indice": r_fut["IDA"].loc[inicio:],
        "tsmom": tsmom.loc[inicio:],
        "carry": carry.loc[inicio:]}

def describe(r):
    anos = (r.index[-1] - r.index[0]).days / 365.25
    m = (1 + r).resample("ME").prod() - 1
    return {"sesiones": len(r), "por_ano": len(r) / anos,
            "asim_diaria": r.skew(), "asim_mensual": m.skew(),
            "curtosis_mensual": m.kurt()}

(pd.DataFrame({k: describe(r) for k, r in rets.items()}).T
 .astype({"sesiones": int}).round(2))
OUT [01]Three strategies on level 2 data
        sesiones  por_ano  asim_diaria  asim_mensual  curtosis_mensual
indice      5263   251.68         0.12         -2.15             15.90
tsmom       5263   251.68         0.02          0.16             -0.07
carry       5457   260.95        -0.71         -1.38              7.39

All three start on February 1, 2005, the first day with carry weights: comparing metrics from different periods is the first mistake of any strategy table, because each period brings its own crises. They are stored in a dictionary and not in a DataFrame because they do not share a calendar: the futures trade on NYSE sessions (251.7 a year) and the currencies on business days (261 a year, with no holidays). In a DataFrame, NYSE holidays would be gaps in the carry.

The table already shows the difference in shape. The carry's monthly skewness is −1.38 and its excess kurtosis, 7.39: a long left tail. The index's is even more extreme (−2.15 and 15.90), but because of a single month: March 2020, with −48.3% (the February-March crisis of the script, which the index shares with the level's stocks). Without that month, the index's stays at −0.46, the same as the carry's without its worst month (February 2009). TSMOM is the only one with positive monthly skewness (0.16) and thin tails (excess kurtosis −0.07). The header figure shows it in the cumulative capital: the index falls in 2008 and in 2020 and recovers in about three years each time; the carry rises smoothly until 2008, falls sharply and takes a decade to recover.

ReturnsGrowthDispersionPathCAGRSharpe, SortinoOmegaDD, Calmar, Ulcer
Mermaid source
flowchart TB
  R[Returns] --> A[Growth]
  R --> V[Dispersion]
  R --> T[Path]
  A --> CAGR[CAGR]
  V --> S[Sharpe, Sortino]
  V --> O[Omega]
  T --> D[DD, Calmar, Ulcer]

The metrics in this lesson fall into three families. Growth metrics say how much the capital has grown. Dispersion metrics compare the mean with the variability of returns and do not depend on their order: shuffling the days does not change a Sharpe. Path metrics depend on the order: the same set of days, ordered with the losses together, gives a much worse drawdown. To see the risk of a strategy you need all three.

Return: the CAGR and what a year is

The compound annual growth rate is the constant growth that takes the initial capital to the final one:

CAGR=(VTV0)365.25/D−1\text{CAGR} = \left(\frac{V_T}{V_0}\right)^{365.25 / D} - 1

with DD the calendar days between the first and the last date. Lesson 0.7.1 already computed it this way. Libraries do it differently: they count years as sessions divided by 252.

IN [02]python
def cagr_fechas(r):
    anos = (r.index[-1] - r.index[0]).days / 365.25
    return (1 + r).prod() ** (1 / anos) - 1

def cagr_sesiones(r, n=252):
    return (1 + r).prod() ** (n / len(r)) - 1

pd.DataFrame({k: {"por_fechas": cagr_fechas(r),
                  "por_sesiones_252": cagr_sesiones(r),
                  "media_x252": r.mean() * 252}
              for k, r in rets.items()}).T.round(4)
OUT [02]CAGR by dates and by sessions
        por_fechas  por_sesiones_252  media_x252
indice      0.0971            0.0972      0.1134
tsmom       0.1239            0.1241      0.1244
carry       0.0389            0.0375      0.0409

In the futures both calculations match, because the NYSE has 251.7 sessions a year. In the carry they do not: 5,457 business days divided by 252 are 21.7 “years”, when the calendar only has 20.9, and the same growth spread over more years gives a CAGR of 3.75% instead of 3.89%. Fourteen basis points a year, because of a convention. In crypto, which trades 365 days, the error would be much larger.

The last column is the daily mean times 252, the arithmetic “annual return”. It is always higher than the CAGR, and the difference grows with volatility: it is the σ2/2\sigma^2/2 term from lesson 0.7.1. In the index, with 20% volatility, the mean times 252 (11.34%) exceeds the CAGR (9.71%) by 1.6 points. When a report says “annual return”, you have to ask which of the two.

growthmean or volUS equitiescurrenciescryptoWhat is annualized?Years by datesMarket?252 sessions261 days365 days
Mermaid source
flowchart TB
  Q{What is annualized?} -- growth --> F[Years by dates]
  Q -- mean or vol --> N{Market?}
  N -- US equities --> B[252 sessions]
  N -- currencies --> D[261 days]
  N -- crypto --> C[365 days]

Volatility and the Sharpe ratio

Annual volatility is the daily standard deviation times the square root of the sessions in the year. The Sharpe ratio divides the mean return in excess of the risk-free rate by its standard deviation3Note 3 · William F. Sharpe, “The Sharpe Ratio”, The Journal of Portfolio Management 21(1), 49-58 (1994): the “ex post” version is the mean of the difference between the fund's return and the benchmark's (here, the risk-free rate) divided by its standard deviation.:

SR=r−rf‾σ(r−rf) nSR = \frac{\overline{r - r_f}}{\sigma(r - r_f)}\,\sqrt{n}

with nn the observations per year. Both are annualized with n\sqrt{n} assuming independent returns; further below we see what happens when they are not.

IN [03]python
def vol_anual(r, n=252):
    return r.std() * np.sqrt(n)

def sharpe(r, rf=0.0, n=252):
    exceso = r - rf
    return exceso.mean() / exceso.std() * np.sqrt(n)

def sesiones_por_ano(r):
    return len(r) / ((r.index[-1] - r.index[0]).days / 365.25)

pd.DataFrame({k: {"vol": vol_anual(r), "sharpe_252": sharpe(r),
                  "sharpe_real": sharpe(r, n=sesiones_por_ano(r))}
              for k, r in rets.items()}).T.round(3)
OUT [03]Volatility and Sharpe
          vol  sharpe_252  sharpe_real
indice  0.203       0.559        0.559
tsmom   0.121       1.026        1.025
carry   0.091       0.451        0.459

With 252 or with the actual sessions, the futures Sharpes barely change (from 1.026 to 1.025 in TSMOM). The carry's goes from 0.451 to 0.459: annualizing by 252 a series with 261 days a year understates it by 1.8% (261/252\sqrt{261/252}). It is small, but it is a systematic bias against currencies when they are compared with stocks in the same table.

The risk-free rate needs more care. A future does not require paying out the notional: a margin is deposited and the rest of the capital can sit in T-bills earning the risk-free rate. That is why the return of a future is already an excess over the risk-free rate, and the same happens with a currency position funded in dollars, which earns the rate differential. On those series, the Sharpe is computed without subtracting anything. A stock portfolio, on the other hand, gives a total return, and each day's risk-free rate has to be subtracted. The typical mistake is mixing the two:

IN [04]python
tl = pd.read_csv("tipo_libre2.csv", parse_dates=["fecha"],
                 index_col="fecha")["tipo_3m"]
rf = (tl / 100 / 252).reindex(rets["tsmom"].index).ffill()
total = rets["tsmom"] + rf      # the collateral earns the risk-free rate

print(f"Average risk-free rate: {rf.mean() * 252:.2%} annual")
print(f"Sharpe of total without subtracting rf: {sharpe(total):.3f}")
print(f"Sharpe of total subtracting rf:         {sharpe(total, rf):.3f}")
print(f"Sharpe of the excess (the series):      {sharpe(rets['tsmom']):.3f}")
OUT [04]Sharpe with the risk-free rate
Average risk-free rate: 1.79% annual
Sharpe of total without subtracting rf: 1.173
Sharpe of total subtracting rf:         1.026
Sharpe of the excess (the series):      1.026

If the interest on the collateral is added to TSMOM (the synthetic risk-free rate of tipo_libre2.csv, 1.79% annual on average) and the Sharpe is computed without subtracting it, you get 1.173 instead of 1.026. Subtracting each day's rate takes you back exactly to the excess series. The error grows with the level of rates: with T-bills at 5%, a strategy that does nothing has a huge “Sharpe”, because its volatility is almost zero.

How far can you trust a Sharpe

A measured Sharpe is an estimate, and it has a standard error. Lo (2002)4Note 4 · Andrew W. Lo, “The Statistics of Sharpe Ratios”, Financial Analysts Journal 58(4), 36-52 (2002). derives it for independent, normal returns:

SE(SR^)≈1+12 SR2TSE(\widehat{SR}) \approx \sqrt{\frac{1 + \tfrac{1}{2}\,SR^2}{T}}

with SRSR and TT at the same frequency (for example, monthly Sharpe and number of months). Since the monthly Sharpe is small, the numerator is almost 1 and the error of the annual Sharpe is around 1/A1/\sqrt{A}, with AA the years of data: with 21 years, 0.22. Lesson 0.1.2 already used that approximation to compute how many years are needed to tell a Sharpe apart from chance.

Two corrections matter here. The first, the shape of the distribution: with skewness γ3\gamma_3 and kurtosis γ4\gamma_4 (not excess), the error is5Note 5 · The version with skewness and kurtosis is due to Mertens (2002); it is reproduced in David H. Bailey and Marcos López de Prado, “The Sharpe ratio efficient frontier”, The Journal of Risk 15(2), 3-44 (2012), https://doi.org/10.21314/JOR.2012.255, the basis of the Deflated Sharpe of lesson 2.5.4.

SE(SR^)≈1−γ3 SR+γ4−14 SR2T−1SE(\widehat{SR}) \approx \sqrt{\frac{1 - \gamma_3\,SR + \tfrac{\gamma_4 - 1}{4}\,SR^2}{T - 1}}

and negative skewness increases it. The second, autocorrelation: multiplying a monthly Sharpe by 12\sqrt{12} assumes independent months. Lo (2002) gives the correct factor,

SRannual=η(q) SRmonthly,η(q)=qq+2∑k=1q−1(q−k) ρkSR_{\text{annual}} = \eta(q)\,SR_{\text{monthly}}, \qquad \eta(q) = \frac{q}{\sqrt{q + 2\sum_{k=1}^{q-1}(q-k)\,\rho_k}}

with q=12q = 12 and ρk\rho_k the autocorrelation of monthly returns at lag kk. With positive autocorrelation, η(12)\eta(12) is smaller than 12≈3.464\sqrt{12} \approx 3.464 and the annual Sharpe goes down.

IN [05]python
from scipy import stats

def error_sharpe(r):
    m = (1 + r).resample("ME").prod() - 1
    T, sr = len(m), m.mean() / m.std()
    g3, g4 = stats.skew(m), stats.kurtosis(m, fisher=False)
    iid = np.sqrt((1 + sr**2 / 2) / T)
    no_normal = np.sqrt((1 - g3 * sr + (g4 - 1) / 4 * sr**2)
                        / (T - 1))
    rho = [m.autocorr(k) for k in range(1, 12)]
    eta = 12 / np.sqrt(12 + 2 * sum((12 - k) * rho[k - 1]
                                    for k in range(1, 12)))
    return {"meses": T, "sr_mensual_x_raiz12": sr * np.sqrt(12),
            "se_iid": iid * np.sqrt(12),
            "se_no_normal": no_normal * np.sqrt(12),
            "eta": eta, "sr_anual_lo": sr * eta}

pd.DataFrame({k: error_sharpe(r) for k, r in rets.items()}).T.round(3)
OUT [05]Standard error of the Sharpe (Lo, 2002)
        meses  sr_mensual_x_raiz12  se_iid  se_no_normal    eta  sr_anual_lo
indice  251.0                0.549   0.220         0.264  4.242        0.672
tsmom   251.0                1.033   0.223         0.219  3.717        1.109
carry   251.0                0.409   0.219         0.239  2.935        0.347
Fig. — Annual Sharpe of the three strategies with a ± 2 standard error interval (skewness and kurtosis formula, 251 months); the thin mark is the Sharpe annualised with Lo's (2002) autocorrelation correction

With 251 months, the three standard errors are between 0.219 and 0.264. The shape of the distribution raises two of them: the index's, from 0.220 to 0.264, because of March 2020 (skewness −2.15, kurtosis 15.9), and the carry's, from 0.219 to 0.239. Autocorrelation sets the three apart. The carry's η\eta is 2.935, well below 3.464: the carry's bad months come in a row (the crashes of the script last one to four months), and its corrected annual Sharpe drops from 0.41 to 0.35. The index's is 4.242, above: its autocorrelations at lags 3 to 10 are negative, because each drop is followed by a rebound, and its corrected annual Sharpe rises from 0.55 to 0.67. TSMOM's is 3.717 (it rises from 1.03 to 1.11). The same correction lowers or raises a Sharpe depending on the series.

The figure shows the ± 2 standard error intervals. Only TSMOM's stays far from zero (from 0.60 to 1.47). The index's grazes it (from 0.02 to 1.08) and the carry's includes it (from −0.07 to 0.89). And that is with the carry having a planted effect in the data: 21 years of a Sharpe of 0.4 are not enough to tell it apart from chance with this rule. When, in addition, many variants have been tried before settling on one, the bar rises; that is the Deflated Sharpe of lesson 2.5.4.

Sortino, Calmar and Omega

The Sharpe penalizes rises and falls alike. Sortino and Price (1994)6Note 6 · Frank A. Sortino and Lee N. Price, “Performance Measurement in a Downside Risk Framework”, The Journal of Investing 3(3), 59-64 (1994). propose dividing only by the downside deviation with respect to a target τ\tau:

Sortino=r−τ‾1N∑tmin⁡(rt−τ, 0)2  n\text{Sortino} = \frac{\overline{r - \tau}}{\sqrt{\tfrac{1}{N}\sum_{t} \min(r_t - \tau,\ 0)^2}}\;\sqrt{n}

The sum runs over all periods (NN), not only over the negative ones: days above the target count as zero. It is the convention of empyrical and quantstats. If the distribution were symmetric, the downside deviation would be σ/2\sigma/\sqrt{2} and the Sortino, 2≈1.41\sqrt{2} \approx 1.41 times the Sharpe. The ratio between the two measures the shape:

IN [06]python
def sortino(r, n=252, objetivo=0.0):
    exceso = r - objetivo
    abajo = np.sqrt((exceso.clip(upper=0) ** 2).mean())
    return exceso.mean() / abajo * np.sqrt(n)

def mensual(r):
    return (1 + r).resample("ME").prod() - 1

pd.DataFrame({k: {"sharpe": sharpe(r), "sortino": sortino(r),
                  "cociente": sortino(r) / sharpe(r),
                  "cociente_mensual": sortino(mensual(r), 12)
                                      / sharpe(mensual(r), n=12)}
              for k, r in rets.items()}).T.round(3)
OUT [06]Sortino
        sharpe  sortino  cociente  cociente_mensual
indice   0.559    0.805     1.441             1.350
tsmom    1.026    1.527     1.489             1.901
carry    0.451    0.634     1.405             1.391

On daily data, the three ratios are between 1.41 and 1.49, close to 2\sqrt{2}: a day does not have time to show the shape of the strategy. On monthly data it appears: TSMOM reaches 1.90, because its good months are larger than its bad ones; the carry drops to 1.39 and the index to 1.35, because their worst months (the crashes and the crises) are larger than their best. The Sortino captures skewness, but only if it is measured at a frequency at which skewness exists.

The Calmar divides the CAGR by the maximum drawdown in absolute value. It is the figure an investor in a futures fund looks at the most: how much it earns per point of the worst drop. The Omega of Keating and Shadwick (2002)7Note 7 · Con Keating and William F. Shadwick, “A Universal Performance Measure”, The Journal of Performance Measurement 6(3) (2002). compares the whole mass of the distribution above a threshold LL with the mass below:

Ω(L)=∑t(rt−L)+∑t(L−rt)+\Omega(L) = \frac{\sum_t (r_t - L)^{+}}{\sum_t (L - r_t)^{+}}

An Omega greater than 1 means that, in total, more is gained above the threshold than is lost below it.

IN [07]python
def omega(r, umbral=0.0):
    d = r - umbral
    return d.clip(lower=0).sum() / -d.clip(upper=0).sum()

u5 = 1.05 ** (1 / 252) - 1      # 5% annual, per session
pd.DataFrame({k: {"diaria_0": omega(r), "diaria_5%": omega(r, u5),
                  "mensual_0": omega(mensual(r))}
              for k, r in rets.items()}).T.round(3)
OUT [07]Omega
        diaria_0  diaria_5%  mensual_0
indice     1.115      1.064      1.609
tsmom      1.183      1.108      2.153
carry      1.079      0.985      1.390

Omega depends a lot on the frequency and the threshold. On daily data, with a zero threshold, the three are between 1.08 and 1.18: each day the mean is tiny compared with the dispersion. With a threshold of 5% annual (converted to its per-session equivalent, as empyrical does), the carry drops below 1: it does not beat that target. On monthly data, TSMOM (2.15) pulls away from the index and the carry (1.61 and 1.39). An Omega without its threshold and its frequency cannot be compared with another.

Drawdown: depth, duration and Ulcer Index

Each day's drawdown is the drop in capital from its previous peak (lesson 1.6.1). There is a detail when computing it from returns: the capital starts at 1 before the first day. If the first return is negative, the maximum of cumprod() starts below 1 and that first day does not count as a drop; clip(lower=1.0) corrects it. quantstats adds a dummy date with the initial value for the same purpose.

IN [08]python
def drawdown(r):
    capital = (1 + r).cumprod()
    pico = capital.cummax().clip(lower=1.0)   # we start at 1
    return capital / pico - 1

def episodios(r):
    dd = drawdown(r)
    bajo = dd < 0
    grupo = (~bajo).cumsum()[bajo]
    filas = []
    for _, e in dd[bajo].groupby(grupo):
        sig = dd.index.searchsorted(e.index[-1], side="right")
        filas.append({
            "inicio": e.index[0].date(), "minimo": e.idxmin().date(),
            "recupera": dd.index[sig].date() if sig < len(dd) else None,
            "profundidad": e.min(), "sesiones": len(e),
            "anos": ((dd.index[min(sig, len(dd) - 1)] - e.index[0])
                     .days / 365.25)})
    return pd.DataFrame(filas)

ep_carry = episodios(rets["carry"])
ep_carry.sort_values("profundidad").head(3).round(3)
OUT [08]Drawdown with the starting point
         inicio      minimo    recupera  profundidad  sesiones   anos
49   2008-08-29  2011-03-31  2018-06-15       -0.479      2555  9.793
68   2019-07-29  2020-03-31  2022-04-13       -0.235       707  2.708
109  2025-03-17  2025-07-03        None       -0.108       208  0.791

The carry's worst episode starts on August 29, 2008, bottoms out on March 31, 2011 at −47.9% and does not recover its peak until June 15, 2018: 2,555 sessions, 9.8 years underwater. The 2008 crash leaves the drop at −24.7% in December; the 2010 and 2011 crashes double it. For an investor, depth is half the damage: the other half is a decade without new highs.

Peter Martin's Ulcer Index (1987)8Note 8 · Peter G. Martin, “Ulcer Index: An Alternative Approach to the Measurement of Investment Risk & Risk-Adjusted Performance”, https://www.tangotools.com/ui/ui.htm: the author developed it in 1987 and published it in 1989; he defines the UPI with the annualized total return minus that of the risk-free asset. combines both dimensions in one figure: the square root of the mean of the squared drawdowns,

UI=1N∑t=1NDDt2UI = \sqrt{\frac{1}{N}\sum_{t=1}^{N} DD_t^{2}}

A deep and long drawdown weighs much more than an equally deep but brief one, because it is added up over many days. The Martin ratio (Ulcer Performance Index) divides the annual excess return by the Ulcer Index, as the Sharpe divides by the standard deviation.

IN [09]python
def ulcer(r):
    return np.sqrt((drawdown(r) ** 2).mean())

def martin(r, rf_anual=0.0):
    return (cagr_fechas(r) - rf_anual) / ulcer(r)

def resumen_dd(r):
    e = episodios(r)
    return {"dd_max": e["profundidad"].min(),
            "sesiones_max": e["sesiones"].max(),
            "anos_max": e["anos"].max(),
            "bajo_el_agua": (drawdown(r) < 0).mean(),
            "ulcer": ulcer(r), "martin": martin(r),
            "calmar": cagr_fechas(r) / -drawdown(r).min()}

pd.DataFrame({k: resumen_dd(r) for k, r in rets.items()}).T.round(3)
OUT [09]Ulcer Index and Martin ratio
        dd_max  sesiones_max  anos_max  bajo_el_agua  ulcer  martin  calmar
indice  -0.622         959.0     3.817         0.921  0.178   0.545   0.156
tsmom   -0.194         675.0     2.683         0.914  0.064   1.931   0.639
carry   -0.479        2555.0     9.793         0.958  0.213   0.182   0.081
Fig. — Anatomy of a drawdown: FX carry from 2005 to 2025, with its maximum depth, the length of the worst episode and the Ulcer Index

The figure draws the anatomy of the carry's worst drawdown: the maximum depth in amber, the bracket of the 9.8 years underwater and the Ulcer Index, 21.3%, as a line: it is the “typical” drawdown level in root mean square. All three strategies spend more than 91% of the sessions below their previous peak; what sets them apart is how much and for how long.

The index falls more than the carry: its maximum drawdown is −62.2% (from November 20, 2019 to May 19, 2020), and the 2008 one, −56.0% (from October 2007 to April 2009). But it recovers from both in a little over three years: in September 2023 (3.8 years) and in November 2010 (3.1 years). The carry, with a smaller drop, spends 9.8 years underwater. Depth and duration are different risks and both have to be reported.

Fig. — Four metrics of the three strategies, February 2005 to 2025: FX carry comes close to the index on Sharpe, but its Calmar is half and its Ulcer higher

The comparison in the figure sums up the lesson. In Sharpe, the index and the carry are close (0.56 and 0.45). In Calmar, the carry has half (0.08 against 0.16), and in Ulcer, more (21.3% against 17.8%), with less than half the index's volatility (9.1% against 20.3%): its maximum drawdown is equivalent to 5.3 times its annual volatility, and the index's, to 3.1. It is the signature of a strategy with negative skewness and losses that chain together: the standard deviation makes it look safe until the day of the crash. TSMOM wins on all four metrics; in these synthetic data, because the momentum term is planted in the expected return of each future.

Against empyrical-reloaded

empyrical-reloaded is the metrics library used by Zipline Reloaded and pyfolio-reloaded (code in its repository, Apache 2.0). Comparing each of your own functions with its counterpart is the cheapest test that the formulas are correctly implemented:

IN [10]python
import empyrical as ep

def comparar(r):
    return pd.Series({
        "sharpe": sharpe(r) - ep.sharpe_ratio(r),
        "sortino": sortino(r) - ep.sortino_ratio(r),
        "omega": omega(r) - ep.omega_ratio(r),
        "dd_max": drawdown(r).min() - ep.max_drawdown(r),
        "vol": vol_anual(r) - ep.annual_volatility(r),
        "cagr_252": cagr_sesiones(r) - ep.cagr(r),
        "cagr_fechas": cagr_fechas(r) - ep.cagr(r),
    })

dif_max = pd.DataFrame({k: comparar(r) for k, r in rets.items()})
dif_max.abs().max(axis=1)
OUT [10]Against empyrical-reloaded
sharpe         1.665335e-15
sortino        2.886580e-15
omega          2.220446e-16
dd_max         5.551115e-17
vol            0.000000e+00
cagr_252       0.000000e+00
cagr_fechas    1.358127e-03
dtype: float64

Sharpe, Sortino, Omega, maximum drawdown and volatility match up to floating-point rounding error (10−1510^{-15}). So does our own CAGR with 252 sessions. The CAGR by dates differs by 0.0014, the carry difference from the CAGR section: empyrical is not wrong, it uses another convention (annualization=261 in its functions would change it). Ulcer, duration and the Martin ratio are not in empyrical; quantstats does have an ulcer_index, and lesson 2.6.3 compares it with this one.

Which metric for which return shape

  • For a symmetric strategy with thin tails, Sharpe and Sortino tell almost the same story and the Sharpe is the standard for comparing. A stock portfolio almost always is one, but not in its crises: the index in this lesson has a monthly skewness of −2.15 because of a single month.
  • For a strategy with negative skewness (carry, volatility selling, reversal with distant stops), the Sharpe is optimistic. Look at Calmar, Ulcer, drawdown duration and the monthly Sortino; and distrust any backtest that does not include a crash.
  • For a strategy with positive skewness (trend), the Sharpe is conservative and what weighs is the duration of the flat stretches: the 2.7 years of TSMOM's worst episode are the moment when a system gets abandoned.
  • Always: the same period for all, the frequency and the annualization convention in plain view, the risk-free rate only where it applies and the standard error next to each Sharpe.

Exercises

  1. Compute the carry's Calmar with data up to December 31, 2007 and with the whole sample. What would someone evaluating it at the beginning of 2008 have concluded?
Solution
IN [11]python
for fin in ("2007-12-31", "2025-12-31"):
    r = rets["carry"].loc[:fin]
    dd = drawdown(r).min()
    print(f"up to {fin}: CAGR {cagr_fechas(r):.2%}, "
          f"max DD {dd:.1%}, Calmar {cagr_fechas(r) / -dd:.2f}, "
          f"Sharpe {sharpe(r, n=261):.2f}")
OUT [11]Calmar before and after the crash
up to 2007-12-31: CAGR 11.09%, max DD -9.1%, Calmar 1.22, Sharpe 1.30
up to 2025-12-31: CAGR 3.89%, max DD -47.9%, Calmar 0.08, Sharpe 0.46

Up to 2007, the carry had a CAGR of 11.09%, a maximum drawdown of −9.1%, a Calmar of 1.22 and a Sharpe of 1.30: an excellent strategy in any table. With the whole sample, the Calmar drops to 0.08 and the Sharpe to 0.46. Nothing in the previous three years announced the crash: negative skewness is only seen when it happens. That is why a carry backtest has to include crisis periods, and why the Calmar of a short sample with no crash says nothing.

  1. Martin recommends computing the Ulcer Index with weekly data. Compare it with the daily one for the three strategies, taking the last capital value of each week.
Solution
IN [12]python
def ulcer_semanal(r):
    capital = (1 + r).cumprod().resample("W-FRI").last()
    dd = capital / capital.cummax().clip(lower=1.0) - 1
    return np.sqrt((dd ** 2).mean())

pd.DataFrame({k: {"diario": ulcer(r), "semanal": ulcer_semanal(r)}
              for k, r in rets.items()}).T.round(4)
OUT [12]Daily and weekly Ulcer
        diario  semanal
indice  0.1781   0.1719
tsmom   0.0642   0.0615
carry   0.2133   0.2117

With weekly data the Ulcer Index drops very little (from 17.81% to 17.19% in the index, from 6.42% to 6.15% in TSMOM, from 21.33% to 21.17% in the carry): as Martin says, it is almost independent of the frequency, unlike the standard deviation, which has to be annualized. It drops a little because the Friday close skips some midweek lows.

  1. A manager presents a Sharpe of 1.2 computed with 36 months of monthly returns multiplied by √12. Which four questions would you ask before believing it?
Solution

First, whether the risk-free rate was subtracted and which one: with rates at 4 or 5%, not subtracting it greatly inflates the Sharpe of a low-volatility strategy. Second, what its standard error is: with 3 years, 1/3≈0.581/\sqrt{3} \approx 0.58, so the ± 2 standard error interval goes from roughly 0.05 to 2.35 and the figure is compatible with a mediocre strategy. Third, whether the monthly returns are autocorrelated: a portfolio with illiquid assets or assets valued with a delay has positive autocorrelation, and Lo's η(12)\eta(12) factor lowers the annual Sharpe. Fourth, what shape the distribution has and what happened in the worst month: if the strategy sells volatility or collects carry, 36 months without a crash give a high Sharpe that negative skewness will refute; ask for the maximum drawdown, its duration and the monthly Sortino.

Footnotes

  1. Tobias J. Moskowitz, Yao Hua Ooi and Lasse Heje Pedersen, “Time series momentum”, Journal of Financial Economics 104(2), 228-250 (2012), https://doi.org/10.1016/j.jfineco.2011.11.003. ↩

  2. Markus K. Brunnermeier, Stefan Nagel and Lasse H. Pedersen, “Carry Trades and Currency Crashes”, NBER Macroeconomics Annual 23, 313-348 (2008), https://doi.org/10.1086/593088. ↩

  3. William F. Sharpe, “The Sharpe Ratio”, The Journal of Portfolio Management 21(1), 49-58 (1994): the “ex post” version is the mean of the difference between the fund's return and the benchmark's (here, the risk-free rate) divided by its standard deviation. ↩

  4. Andrew W. Lo, “The Statistics of Sharpe Ratios”, Financial Analysts Journal 58(4), 36-52 (2002). ↩

  5. The version with skewness and kurtosis is due to Mertens (2002); it is reproduced in David H. Bailey and Marcos López de Prado, “The Sharpe ratio efficient frontier”, The Journal of Risk 15(2), 3-44 (2012), https://doi.org/10.21314/JOR.2012.255, the basis of the Deflated Sharpe of lesson 2.5.4. ↩

  6. Frank A. Sortino and Lee N. Price, “Performance Measurement in a Downside Risk Framework”, The Journal of Investing 3(3), 59-64 (1994). ↩

  7. Con Keating and William F. Shadwick, “A Universal Performance Measure”, The Journal of Performance Measurement 6(3) (2002). ↩

  8. Peter G. Martin, “Ulcer Index: An Alternative Approach to the Measurement of Investment Risk & Risk-Adjusted Performance”, https://www.tangotools.com/ui/ui.htm: the author developed it in 1987 and published it in 1989; he defines the UPI with the annualized total return minus that of the risk-free asset. ↩

Each answer is folded under its question

Self-assessment

  1. The currency carry trades 261 days a year. Why is its CAGR computed with empyrical.cagr (3.75%) lower than the one computed with the dates (3.89%)?

    • ABecause empyrical subtracts the risk-free rate
    • BBecause empyrical counts years as sessions divided by 252, and 5,457 business days are 21.7 “years” of 252 when the calendar only has 20.9
    • CBecause empyrical uses log returns
    • DBecause the series has gaps on holidays
    Show answer

    Answer B. Because empyrical counts years as sessions divided by 252, and 5,457 business days are 21.7 “years” of 252 when the calendar only has 20.9

    The same total growth spread over more years gives a lower rate. For the CAGR, always count years with the dates; to annualize means and volatilities, use the market's actual sessions per year.

  2. TSMOM on futures has a Sharpe of 1.026. If you add the interest earned by the collateral and compute the Sharpe without subtracting the risk-free rate, you get 1.173. Which one is correct?

    • A1.026: the Sharpe measures the excess over the risk-free rate, and the futures series is already an excess return; on the total return the risk-free rate has to be subtracted, and you are back to 1.026
    • B1.173, because it is the return actually earned
    • CThe average of the two
    • DNeither: with futures the Sharpe makes no sense
    Show answer

    Answer A. 1.026: the Sharpe measures the excess over the risk-free rate, and the futures series is already an excess return; on the total return the risk-free rate has to be subtracted, and you are back to 1.026

    Without subtracting the risk-free rate, any strategy that leaves the money in T-bills seems to have an edge. With an average rate of 1.79%, the error adds 0.15 to TSMOM's Sharpe.

  3. With 251 months of data, the standard error of an annual Sharpe is about 0.22. What does that imply for the currency carry, with an annualized monthly Sharpe of 0.41?

    • AThat its edge is proven at 95%
    • BThat the standard error does not depend on the years of data
    • CThat its ± 2 standard error interval goes from −0.07 to 0.89: with 21 years of data, that Sharpe cannot be told apart from zero
    • DThat the true Sharpe is exactly 0.41
    Show answer

    Answer C. That its ± 2 standard error interval goes from −0.07 to 0.89: with 21 years of data, that Sharpe cannot be told apart from zero

    The standard error of the annual Sharpe is of the order of 1/√years. With its negative skewness and its kurtosis, the carry's rises to 0.239: the interval includes zero, even though the effect is planted in the data.

  4. The IDA index and the FX carry have a similar Sharpe (0.56 and 0.45). Why is the carry's Calmar half as large?

    • ABecause the carry has more volatility
    • BBecause the Calmar penalizes currency markets
    • CBecause of a calculation error
    • DBecause the carry earns 3.9% a year and, with a volatility of 9%, falls 47.9% in its worst episode, 5.3 times its volatility: its losses come in crashes that the standard deviation does not anticipate; the index falls more (62.2%), but earns 9.7% a year
    Show answer

    Answer D. Because the carry earns 3.9% a year and, with a volatility of 9%, falls 47.9% in its worst episode, 5.3 times its volatility: its losses come in crashes that the standard deviation does not anticipate; the index falls more (62.2%), but earns 9.7% a year

    The Sharpe only sees dispersion. Both have negative monthly skewness, but the index rebounds after each crisis and the carry chains bad months together (its Lo η is 2.94, below √12). The Calmar, the Ulcer (21.3% against 17.8%) and the drawdown duration (9.8 years against 3.8) do capture that path.

  5. What does the Ulcer Index measure?

    • AThe square root of the mean of the squared drawdowns: it grows with the depth and with the time spent underwater
    • BThe worst daily loss
    • CThe standard deviation of the negative returns
    • DThe number of drawdown episodes
    Show answer

    Answer A. The square root of the mean of the squared drawdowns: it grows with the depth and with the time spent underwater

    A deep but brief drawdown and a moderate but very long one can give the same Ulcer. It is the metric closest to what someone following the strategy feels; Martin proposes dividing the annual excess return by it (Martin ratio or UPI).

Key points

  • Each metric brings a convention: years by dates for the CAGR; actual sessions per year (252 in equities, 261 in currencies) for means and volatilities; and a risk-free rate that is subtracted only if the series is not already an excess return.
  • A Sharpe is an estimate with an error: about 1/√years units. With 21 years, ± 0.44 at two standard errors; negative skewness and autocorrelation make it worse.
  • Sharpe and Sortino look at dispersion; Calmar, duration and Ulcer look at the path. A carry with negative skewness comes close in Sharpe to an index (0.45 and 0.56), has half the Calmar and more Ulcer with less than half the volatility.
  • Drawdown has depth and duration: the index's is deeper (−62.2%) and recovers in 3.8 years; the carry's (−47.9%) lasts 9.8.
  • Implemented from scratch, Sharpe, Sortino, Omega, volatility and drawdown match empyrical-reloaded up to rounding; the CAGR only matches if years are counted the way it does.
  • The data are synthetic and the effects are planted: these figures teach how to measure, they do not prove that TSMOM or the carry work in real markets.

Resources

Data