Lesson 4.1.1 · 70 min
Markowitz: efficient frontier and real constraints
Mean-variance optimization is exact in the mathematics and fragile in the statistics: it gives the best portfolio for expected returns and a covariance that are never known. This lesson solves it with the closed-form solution and with cvxpy, adds the constraints of a real mandate and measures, with a synthetic universe whose truth is published, how far the estimated frontier is from the one actually obtained.
- Module
- 4.1 · Portfolio theory and optimisation
- Duration
- 70 min
- Prerequisites
- 2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward, 2.6.1 · CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index
- Author
- Jesús Cuesta
- Updated
Level 4 lessons
4.1 · Portfolio theory and optimisation
- 4.1.1 Markowitz: efficient frontier and real constraints
- 4.1.2 Covariance estimation: shrinkage, factor models and denoising
- 4.1.3 Black-Litterman: from market equilibrium to views
- 4.1.4 Risk parity, minimum variance, HRP and HERC
- 4.1.5 Optimization with CVaR, drawdown, turnover and costs
- 4.1.6 Factor models: CAPM, Fama-French, PCA and attribution
- 4.1.7 Rebalancing and strategy portfolios
- 4.1.8 Portfolio tools: Riskfolio-Lib, PyPortfolioOpt, skfolio and bt
4.2 · Risk management and position sizing
- 4.2.1 Risk per trade, stops and risk of ruin
- 4.2.2 Kelly criterion and volatility targeting: how much to risk
- 4.2.3 VaR, Expected Shortfall and VaR backtesting
- 4.2.4 Stress testing and correlation, liquidity, gap and counterparty risk
- 4.2.5 Drawdown control and kill switch
- 4.2.6 Derivatives risk: portfolio Greeks, margin and liquidations
4.3 · Microstructure and execution
4.4 · Brokers and APIs
- 4.4.1 Choosing a broker from Spain or the EU: regulation, assets, costs and API
- 4.4.2 Interactive Brokers: TWS and IB Gateway, ib_async, contracts, data and orders
- 4.4.3 Alpaca, Tradier, MetaTrader 5 with Python and OANDA v20
- 4.4.4 Crypto exchange APIs: ccxt, Binance, Bybit, Kraken and Coinbase
- 4.4.5 Secure credentials: environment variables, secret managers and least privilege
4.5 · Live trading system
- 4.5.1 Live system architecture: from data to reconciliation
- 4.5.2 Order management: states, partial fills, cancellations and reconnections
- 4.5.3 Paper trading, shadow trading and slippage monitoring
- 4.5.4 Task scheduling and alerts: cron, APScheduler, Prefect, Telegram, Slack and email
- 4.5.5 Fault tolerance: idempotency, persistent state and reconciliation at startup
- 4.5.6 NautilusTrader or LEAN live: from backtest to production with the same code
A portfolio is a quadratic problem
Harry Markowitz (1952) turned the choice of a portfolio into an optimization problem: among all portfolios with the same expected return, the best one is the one with the least variance. With assets, a weight vector , the expected returns in excess of the risk-free rate and the covariance matrix :
The mean-variance portfolio for a target solves
It is a quadratic program: the objective is a convex quadratic form (because is positive semidefinite) and the constraints are linear. Sweeping gives the efficient frontier, the curve of the portfolios that cannot improve their return without increasing their risk. Any point below it is an inefficient portfolio.
The problem has two inputs, and , and neither is known. They are estimated with past data, and the optimizer treats them as if they were exact. That is the idea that runs through the whole lesson:
Mermaid source
flowchart TB H[History] --> E["Estimate μ and Σ"] E --> O[Optimizer] O --> W[Weights] W --> R[Real result] V["True μ and Σ"] --> R
What the optimizer sees are the estimates; what the portfolio gets depends on the true values. In real markets the true values are never known. In the level 4 data they are: the universe comes from a published factor model, so the distance between what is promised and what is obtained can be measured.
The universe and its truth
The data is 38 synthetic total return indices in USD (equities by region, sector and style, bonds, credit, real estate, commodities and money market) from 2006 to 2025, with the real NYSE sessions. It is not market data: it imitates the profile of real indices (volatility, crises, correlations), with the crises of 2008, 2020 and 2022 written into a script. To keep the figures readable, the lesson works with ten of them, one or two per asset class. The excess return of each day is the simple return minus the risk-free rate of the day:
import numpy as np
import pandas as pd
precios = pd.read_parquet("universo4_precios.parquet")
precios = precios.set_index("fecha")
fac = pd.read_parquet("universo4_factores_diarios.parquet")
rf = fac.set_index("fecha")["rf"]
r = precios.pct_change()
r.iloc[0] = precios.iloc[0] / 100 - 1 # base 100 the day before
exceso = r.sub(rf, axis=0)
activos = ["RV_EEUU", "RV_EUR", "RV_JAP", "RV_EMG", "INM_EEUU",
"RF_T710", "RF_T20", "RF_CIG", "RF_CHY", "MP_ORO"]
R = exceso[activos]
print(R.shape, R.index[0].date(), R.index[-1].date())(5031, 10) 2006-01-03 2025-12-31Returns are simple, not logarithmic. The return of a portfolio is the weighted sum of the simple returns of its assets; with logarithms that sum is not exact and the portfolio's Sharpe comes out lower than it is.
The “truth” is in two files: universo4_verdad.csv holds the expected excess return of each asset over the period
(mu_verdadero) and universo4_cov_verdad.csv, the true annual covariance per regime and the overall one (total),
which is what the full sample estimates:
verdad = pd.read_csv("universo4_verdad.csv", index_col="activo")
cov_v = pd.read_csv("universo4_cov_verdad.csv")
total = cov_v[cov_v["regimen"] == "total"].set_index("activo")
mu_v = verdad.loc[activos, "mu_verdadero"].to_numpy()
S_v = total.loc[activos, activos].to_numpy()
mu_m = R.mean().to_numpy() * 252
S_m = R.cov().to_numpy() * 252
tabla = pd.DataFrame({
"mu_verdad": mu_v, "mu_muestra": mu_m,
"vol_verdad": np.sqrt(np.diag(S_v)),
"vol_muestra": np.sqrt(np.diag(S_m))}, index=activos)
print((100 * tabla).round(2))
ee = np.sqrt(np.diag(S_m) / 20)
print("standard error of the mean (20 years):", (100 * ee).round(1)) mu_verdad mu_muestra vol_verdad vol_muestra
RV_EEUU 9.62 8.79 20.37 20.59
RV_EUR 5.82 3.20 22.63 22.81
RV_JAP 5.45 3.84 21.86 22.00
RV_EMG 3.80 5.46 26.54 27.16
INM_EEUU 9.17 8.98 23.35 23.46
RF_T710 1.60 0.49 5.92 5.90
RF_T20 3.68 1.54 13.69 13.58
RF_CIG 2.76 2.13 5.56 5.59
RF_CHY 4.96 4.68 10.92 10.48
MP_ORO 9.98 10.03 14.59 14.17
standard error of the mean (20 years): [4.6 5.1 4.9 6.1 5.2 1.3 3. 1.3 2.3 3.2]Volatilities are estimated well: with 5,031 sessions, the sample one is within 0.7 points of the true one for every
asset. Means are not. RV_EUR returns 3.20% in the sample against an expected 5.82%; the long bond, 1.54% against
3.68%. It is not a flaw in the data: the standard error of an annual mean is the volatility divided by the square root
of the number of years, and with 20 years and a 20% volatility it comes to almost 5 points. It is the law that governs
the rest of the lesson: variance is estimated with days; the mean, with years.
The closed-form solution
With no constraints other than the target and the sum of weights, the problem is solved with Lagrange multipliers (Merton, 1972). With the constants
the frontier portfolio with return and its variance are
The variance is a parabola in , so the frontier in the (volatility, return) plane is a hyperbola. Its vertex is the minimum variance portfolio, , with volatility and return . And since is linear in , every efficient portfolio is a combination of any two efficient portfolios: this is the two-fund theorem.
iS = np.linalg.inv(S_v)
uno = np.ones(len(activos))
A = uno @ iS @ uno
B = uno @ iS @ mu_v
C = mu_v @ iS @ mu_v
D = A * C - B**2
w_mv = iS @ uno / A
def w_frontera(m):
return ((C - B * m) * iS @ uno + (A * m - B) * iS @ mu_v) / D
w5 = w_frontera(0.05)
cerrada = pd.DataFrame({"min_var": w_mv, "frontera_5": w5},
index=activos)
print((100 * cerrada).round(1))
print(f"min. variance: vol {np.sqrt(1 / A):.2%}, mu {B / A:.2%}")
print(f"5% frontier: vol {np.sqrt(w5 @ S_v @ w5):.2%}, "
f"formula {np.sqrt((A * .05**2 - 2 * B * .05 + C) / D):.2%}") min_var frontera_5
RV_EEUU 2.7 41.9
RV_EUR 0.2 -7.3
RV_JAP 0.4 -2.0
RV_EMG -0.6 -18.2
INM_EEUU -1.8 -3.9
RF_T710 152.9 67.6
RF_T20 -67.1 -35.0
RF_CIG 14.0 46.2
RF_CHY -1.7 -8.2
MP_ORO 1.0 19.0
min. variance: vol 1.81%, mu 0.48%
5% frontier: vol 4.33%, formula 4.33%The minimum variance portfolio has a volatility of 1.81%, less than the 7-10 year bond on its own (5.9%). It gets there with 153% in that bond and −67% in the 20+ year bond: both load on the same rate factor and, in the right proportion, they cancel out. It is correct on paper and it is a leveraged portfolio that depends on that correlation being exactly the estimated one. The 5% efficient portfolio also sells short: emerging markets (−18%), the long bond (−35%) and Europe.
The same with cvxpy
The closed form stops existing as soon as an inequality appears (long only, a bound). That is what convex optimization is for. cvxpy lets you write the problem almost as on the whiteboard and picks a suitable solver; here CLARABEL is fixed, an interior-point solver for conic problems that ships with cvxpy. Before adding anything, it is worth checking that it reproduces the formula:
import cvxpy as cp
w = cp.Variable(len(activos))
objetivo = cp.Parameter()
riesgo = cp.quad_form(w, cp.psd_wrap(S_v))
prob = cp.Problem(cp.Minimize(riesgo),
[cp.sum(w) == 1, mu_v @ w == objetivo])
objetivo.value = 0.05
prob.solve(solver="CLARABEL")
print(prob.status)
print("max. difference from the formula < 1e-6:",
np.abs(w.value - w5).max() < 1e-6)optimal
max. difference from the formula < 1e-6: Truecp.psd_wrap tells cvxpy that the matrix is positive semidefinite, instead of checking it with a decomposition that,
with nearly singular matrices, can fail through rounding. cp.Parameter makes it possible to change the target and
solve again without rebuilding the problem, which is what is needed to sweep the frontier.
Real constraints
A real mandate never lets you short 35% of the long bond. The usual constraints are written in one line each:
- Long only: . The rule for almost all mutual funds and pension plans.
- Per-asset bounds: , so that no position dominates.
- Groups: equities (stocks and real estate) between 20% and 60%, for example in a balanced profile.
- Gross leverage: , a 130/30 portfolio (130% long, 30% short). In cvxpy,
cp.norm1(w) <= 1.6.
rv = [0, 1, 2, 3, 4] # stocks and real estate
reglas = {
"unconstrained": [],
"130/30": [cp.norm1(w) <= 1.6],
"long only": [w >= 0],
"mandate": [w >= 0, w <= 0.30, cp.sum(w[rv]) >= 0.2,
cp.sum(w[rv]) <= 0.6],
}
problemas = {
k: cp.Problem(cp.Minimize(riesgo),
[cp.sum(w) == 1, mu_v @ w >= objetivo] + extra)
for k, extra in reglas.items()}
filas = {}
for k, pr in problemas.items():
pr.solve(solver="CLARABEL")
filas[k] = list(w.value) + [np.sqrt(pr.value)]
pesos = pd.DataFrame(filas, index=activos + ["vol"])
print((100 * pesos).round(1) + 0) # + 0 removes the -0.0 unconstrained 130/30 long only mandate
RV_EEUU 41.9 26.3 11.4 20.0
RV_EUR -7.3 0.0 0.0 0.0
RV_JAP -2.0 0.0 0.0 0.0
RV_EMG -18.2 -15.3 0.0 0.0
INM_EEUU -3.9 -1.1 0.0 0.0
RF_T710 67.6 63.0 25.9 30.0
RF_T20 -35.0 -13.5 0.0 3.0
RF_CIG 46.2 13.8 35.2 30.0
RF_CHY -8.2 8.3 4.6 0.8
MP_ORO 19.0 18.5 23.0 16.2
vol 4.3 4.7 5.8 6.1The target return becomes an inequality (): below the return of the minimum variance portfolio of each set of rules, the optimum is that portfolio, and the frontier is cut at its vertex. The 130/30 portfolio keeps part of the short positions (emerging markets, long bond) and its volatility stays at 4.7%. With long only they all disappear and the portfolio concentrates in five assets, four of them above 10%: the 7-10 year bond, investment grade credit, gold and US stocks. Volatility rises to 5.8%. The mandate forces a 20% equity allocation and caps the bond and credit at 30%: 6.1%. The full frontier shows where each rule bites:
objetivos = np.round(np.arange(0.03, 0.0951, 0.01), 3)
fr = {}
for k, pr in problemas.items():
vols = []
for m in objetivos:
objetivo.value = m
pr.solve(solver="CLARABEL")
ok = pr.status == "optimal"
vols.append(np.sqrt(pr.value) if ok else np.nan)
fr[k] = vols
fr = pd.DataFrame(fr, index=objetivos)
fr.index.name = "objetivo"
print((100 * fr).round(2)) unconstrained 130/30 long only mandate
objetivo
0.03 2.84 3.48 4.83 6.06
0.04 3.56 4.05 5.11 6.06
0.05 4.33 4.72 5.83 6.11
0.06 5.14 5.46 6.82 6.91
0.07 5.96 6.23 8.02 8.22
0.08 6.80 7.08 9.27 10.31
0.09 7.64 8.12 10.75 13.73Each constraint shifts the frontier to the right: for the same return, more volatility. And it cuts it short: with long only, no target can exceed the return of the best asset (gold, 9.98%); with the mandate, which caps each position at 30%, the maximum falls below that 9.98% and a 9% target requires a volatility of 13.7%. With the true parameters constraints can only worsen the result; the interesting question is what happens when the parameters are estimated.
Constraints fall into three families, according to the solver they need:
Mermaid source
flowchart LR P[Constraints] --> L[Linear] P --> C[Convex] P --> M[Integer] L --> L1["w ≥ 0, bounds"] L --> L2[Groups] C --> C1[Leverage] C --> C2[Tracking error] M --> M1[Cardinality] M --> M2[Minimum lot]
Linear and convex constraints (leverage is a 1-norm; tracking error against an index, , a quadratic constraint) are handled by any conic solver. Integer ones, such as “at most five assets” or “lots of 100 shares”, turn the problem into a mixed-integer one:
import warnings
warnings.filterwarnings("ignore", category=UserWarning,
message="Your problem is mixed-integer")
z = cp.Variable(len(activos), boolean=True)
cardinal = cp.Problem(cp.Minimize(riesgo), [
cp.sum(w) == 1, mu_v @ w >= 0.05, w >= 0, w <= z,
cp.sum(z) <= 3])
try:
cardinal.solve()
except cp.error.SolverError as e:
print(str(e).split(".")[0])You need a mixed-integer solver for this modelThe warnings filter silences, by its message, the UserWarning with which cvxpy suggests installing SCIP; without it,
the warning would appear in the output with the file path. Among the solvers of this environment (CLARABEL, OSQP, SCS,
HiGHS and SciPy) there is none that solves a mixed-integer quadratic problem: HiGHS solves integer linear programs and
continuous quadratic ones, but not both at once. The options are to install one that does (SCIP is free; Gurobi, MOSEK
and CPLEX, commercial) or a reasonable heuristic: solve the continuous problem, keep the assets with the largest weight
and optimize again with only those. Transaction costs and turnover, which also change the shape of the problem, are
the topic of lesson 4.1.5.
Maximum Sharpe
If you can invest in the risk-free asset, the best risky portfolio is the one that maximizes the Sharpe, the tangency portfolio. Without constraints its direction is ; the size is set by the risk you want to take. With constraints, the ratio is not concave, but it becomes a quadratic program with a change of variable: minimize with and the homogeneous constraints on (for example ), and then .
def sharpe(w, mu, S):
return w @ mu / np.sqrt(w @ S @ w)
def max_sharpe(mu, S):
y = cp.Variable(len(mu))
pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
[mu @ y == 1, y >= 0])
pr.solve(solver="CLARABEL")
return y.value / y.value.sum()
w_tan = iS @ mu_v
w_tan = w_tan * 0.10 / np.sqrt(w_tan @ S_v @ w_tan) # 10% vol.
w_lo = max_sharpe(mu_v, S_v)
tang = pd.DataFrame({"sin_limites_vol10": w_tan, "solo_largos": w_lo},
index=activos)
print((100 * tang).round(1))
print("Sharpe unconstrained:", sharpe(w_tan, mu_v, S_v).round(3))
print("Sharpe long only:", sharpe(w_lo, mu_v, S_v).round(3))
print("sum of unconstrained weights:", w_tan.sum().round(2)) sin_limites_vol10 solo_largos
RV_EEUU 100.3 14.6
RV_EUR -18.4 0.0
RV_JAP -5.4 0.0
RV_EMG -44.3 0.0
INM_EEUU -7.5 0.0
RF_T710 -20.4 0.0
RF_T20 -4.2 0.0
RF_CIG 97.0 52.9
RF_CHY -18.3 0.0
MP_ORO 45.9 32.5
Sharpe unconstrained: 1.178
Sharpe long only: 0.879
sum of unconstrained weights: 1.25With the true parameters, the unconstrained portfolio reaches a Sharpe of 1.18, and the long-only one, 0.88. The difference comes from long-short positions: buying investment grade credit (97%) and selling Treasury bonds, emerging markets and Europe. It exploits differences in expected return of one or two points between highly correlated assets. With the truth in front of you, those differences exist. With 20 years of data, the standard error of each mean is 1.3 to 6 points: the unconstrained portfolio bets precisely on what is measured worst.
The sum of the unconstrained weights for a 10% volatility is not 1: with a risk-free asset, the difference up to 1 goes to cash or is borrowed at the risk-free rate (here the portfolio sums to 1.25: 25% financed). Normalizing the tangency portfolio so that it sums to 1, , is dangerous with estimated means: if the denominator approaches zero, the weights blow up, and if it changes sign the portfolio flips.
Estimation error
Now the experiment that matters. A manager who optimizes on a date uses the data available, for example the last five years. For each five-year window, the long-only maximum Sharpe portfolio is computed with the window's mean and covariance, and its Sharpe is measured twice: the one it promises (with the window's estimates) and the one it really has (with the true and ):
filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
("2016", "2020"), ("2021", "2025")]:
Rv = R.loc[a0:a1]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
w_e = max_sharpe(mu_e, S_e)
filas.append({"ventana": f"{a0}-{a1}",
"sharpe_estimado": sharpe(w_e, mu_e, S_e),
"sharpe_real": sharpe(w_e, mu_v, S_v),
"activos": int((w_e > 0.01).sum()),
"mayor_peso": activos[int(w_e.argmax())]})
ventanas = pd.DataFrame(filas)
print(ventanas.round(2))
print("true maximum:", sharpe(w_lo, mu_v, S_v).round(2)) ventana sharpe_estimado sharpe_real activos mayor_peso
0 2006-2010 1.75 0.76 4 RF_CIG
1 2011-2015 0.49 0.53 3 RF_T20
2 2016-2020 1.16 0.71 5 MP_ORO
3 2021-2025 1.11 0.80 3 MP_ORO
true maximum: 0.88The 2006-2010 window promises a Sharpe of 1.75 and delivers 0.76. No window reaches the true maximum of 0.88: it is impossible by construction, because that maximum belongs to the optimal weights under the truth, and any other set of weights falls below it. What can be avoided is believing the promised number. Three of the four windows promise more than they deliver, and each one picks a different portfolio, of three to five assets, with the largest weight in credit, in the long bond or in gold depending on the window. The only one that promises less (2011-2015) does so because everything returned little in that window.
The figure at the top shows it over the whole frontier, with the 2021-2025 window: the estimated frontier (what the manager sees on screen) lies above the true one over almost its whole range (they only cross below 6% volatility), and the realized one (the same weights evaluated with the truth) lies below. The estimated frontier is optimistic for two reasons that add up. The first, the error in the mean: in that window gold returned 13.7% and the long bond −11.6% a year in excess, far from their expected 10.0% and 3.7%. The second, selection: the optimizer picks exactly the assets whose error was favorable. That is why Michaud (1989) called mean-variance optimization an “error maximizer”.
Rv = R.loc["2021":"2025"]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
est = cp.Problem(cp.Minimize(cp.quad_form(w, cp.psd_wrap(S_e))),
[cp.sum(w) == 1, mu_e @ w >= objetivo, w >= 0])
ver = problemas["long only"]
filas = []
print("gold and long bond in the window:",
(100 * mu_e[[9, 6]]).round(1))
for m in [0.04, 0.07, 0.09]:
objetivo.value = m
est.solve(solver="CLARABEL")
w_e = w.value.copy()
ver.solve(solver="CLARABEL")
filas.append({"objetivo": m,
"vol_estimada": np.sqrt(w_e @ S_e @ w_e),
"mu_real": w_e @ mu_v,
"vol_real": np.sqrt(w_e @ S_v @ w_e),
"vol_verdadera_al_objetivo": np.sqrt(ver.value)})
print((100 * pd.DataFrame(filas)).round(2))gold and long bond in the window: [ 13.7 -11.6]
objetivo vol_estimada mu_real vol_real vol_verdadera_al_objetivo
0 4.0 5.89 4.97 6.60 5.11
1 7.0 7.12 6.11 8.33 8.02
2 9.0 8.20 7.21 9.29 10.75For a 7% target, the window says a volatility of 7.1% is enough. Those weights really return 6.1% with 8.3% volatility; with the truth, 7% is achieved with 8.0%. The manager loses on both sides: less return and more risk than expected.
Monte Carlo from the truth
The real windows mix two things: estimation error and the regime changes of the script (the 2008 crisis falls entirely in the first window). To isolate estimation error, samples of independent normal returns are simulated with exactly the true and , each sample is optimized and the real Sharpe of each portfolio is measured. Two hundred samples of five and of twenty years:
rng = np.random.default_rng(41)
L = np.linalg.cholesky(S_v / 252)
for anos in (5, 20):
res = []
for _ in range(200):
X = mu_v / 252 + rng.standard_normal((252 * anos, 10)) @ L.T
m_e = X.mean(axis=0) * 252
if (m_e <= 0).all():
continue
S_e = np.cov(X, rowvar=False) * 252
w_e = max_sharpe(m_e, S_e)
res.append((sharpe(w_e, m_e, S_e), sharpe(w_e, mu_v, S_v)))
res = np.array(res)
p5, p95 = np.percentile(res[:, 1], [5, 95])
print(f"{anos} years: promised {np.median(res[:, 0]):.2f}, "
f"real {np.median(res[:, 1]):.2f} ({p5:.2f} to {p95:.2f}), "
f"overpromises {np.mean(res[:, 0] > res[:, 1]):.0%}")5 years: promised 1.17, real 0.73 (0.49 to 0.85), overpromises 85%
20 years: promised 0.98, real 0.82 (0.73 to 0.87), overpromises 78%With five years, the median promised Sharpe is 1.17 and the median real one, 0.73; in 85% of the samples more is promised than obtained, and one in twenty portfolios falls below 0.49. With twenty years, the gap narrows (0.98 against 0.82) but does not disappear. And that is with ideal data: normal, independent and without regime changes. In real markets, with fat tails and correlations that change in crises, the error is larger.
What to do about the error
Mean-variance is not abandoned; it is protected. The usual defenses, and where they appear in the course:
| Defense | What it tackles | Lesson |
|---|---|---|
| Constraints (long only, bounds, groups) | Long-short bets on small differences | This one |
| Better estimators of Σ (shrinkage, factors, denoising) | Covariance error | 4.1.2 |
| Equilibrium and views (Black-Litterman) | Mean error | 4.1.3 |
| Portfolios that do not use μ (minimum variance, risk parity, HRP) | They ignore the most fragile part | 4.1.4 |
| Penalizing turnover and costs | Rebalances that chase noise | 4.1.5 |
Jagannathan and Ma (2003) showed that imposing long only on a minimum variance portfolio is equivalent to shrinking the covariances of the assets the optimizer would want to sell: a “wrong” constraint from the point of view of theory that acts as a better estimator. That is why a mandate with reasonable bounds is not only a commercial requirement; it is also a form of regularization.
Typical mistakes when optimizing:
- Using log returns for portfolio means and covariances, or annualizing the log mean as if it were simple.
- Mixing total and excess returns: the Sharpe and the tangency portfolio are computed in excess of the risk-free rate.
- Normalizing the tangency portfolio with when that number is close to zero.
- Believing the estimated frontier: it is the frontier of the estimates, not that of the market.
- Not fixing the solver or not checking
prob.status: an infeasible problem returnsNonein the weights andinfeasiblein the status, not an exception.
Exercises
- Repeat the unconstrained minimum variance portfolio with the 38 assets of the universe and the overall true
covariance. What weight does the money market (
MONET) get, and why?
Solution
todos = list(total.index)
S38 = total.loc[todos, todos].to_numpy()
iS38 = np.linalg.inv(S38)
u38 = np.ones(len(todos))
w38 = pd.Series(iS38 @ u38 / (u38 @ iS38 @ u38), index=todos)
print(f"vol: {np.sqrt(1 / (u38 @ iS38 @ u38)):.3%}")
print((100 * w38).round(1).sort_values().iloc[[0, 1, -2, -1]])vol: 0.152%
RF_T710 -2.2
RF_T20 -1.2
RF_T13 8.8
MONET 92.5
dtype: float64The money market has a volatility of 0.2% and almost no correlation with anything else: the minimum variance portfolio takes almost all of it, and the volatility falls below its own. It is the right answer to the wrong question. If there is an almost riskless asset, minimum variance over risky assets reduces to it; that is why portfolios are optimized in excess of the risk-free rate and without the money market inside the risky universe.
- Repeat the window experiment with a 30% bound per asset in addition to long only. Does the gap between the promised and the real Sharpe narrow?
Solution
def max_sharpe_cota(mu, S, cota=0.30):
y = cp.Variable(len(mu))
k = cp.Variable(nonneg=True) # y = k·w, so y <= cota·k
pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
[mu @ y == 1, y >= 0, y <= cota * k,
cp.sum(y) == k])
pr.solve(solver="CLARABEL")
return y.value / y.value.sum()
filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
("2016", "2020"), ("2021", "2025")]:
Rv = R.loc[a0:a1]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
w_c = max_sharpe_cota(mu_e, S_e)
filas.append({"ventana": f"{a0}-{a1}",
"estimado": sharpe(w_c, mu_e, S_e),
"real": sharpe(w_c, mu_v, S_v)})
print(pd.DataFrame(filas).round(2)) ventana estimado real
0 2006-2010 1.72 0.75
1 2011-2015 0.48 0.54
2 2016-2020 1.16 0.70
3 2021-2025 1.02 0.67The bound is a non-homogeneous constraint (), so in the change of variable it has to be multiplied by the scale factor . The bound barely changes the first three windows (the portfolios were already spread out) and in 2021-2025 it lowers the real Sharpe from 0.80 to 0.67: it limits gold, which under the truth is precisely the most attractive asset. A constraint does not recover the truth: it reduces how much is bet on each window's error, and that sometimes has a cost. Its value lies in the worst case, not in the average of four windows.
- With 20 years of data, the standard error of the mean of
RV_EEUUis about 4.6 points. How many years would it take to bring it down to 1 point? And why does using daily data instead of annual data not shorten that period?
Solution
With years of data, the standard error of the annual mean is : with , getting down to 1 point requires years. Using daily data does not help: the annual mean estimated with days is , with standard error , which only depends on the number of years the sample covers, not on how many observations it contains. Frequency does improve the estimation of the variance, which depends on the number of observations. That is why portfolios that do not need (lesson 4.1.4) and methods that take it from equilibrium (lesson 4.1.3) exist.
Each answer is folded under its question
Self-assessment
The maximum Sharpe portfolio estimated with 2006-2010 data promises a Sharpe of 1.75. What Sharpe does it have with the true parameters?
- A1.75: the optimization is exact
- B0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance
- CMore than 1.75, because the sample is pessimistic
- DIt cannot be known
Show answer
Answer B. 0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance
The optimizer maximizes over estimates with error and picks precisely the assets whose error was favorable. The Sharpe estimated on the optimization's own sample is biased upward; the real Sharpe of those weights falls below the true maximum. In Monte Carlo with five years, the estimate exceeds the real one in 85% of cases.
Why does the unconstrained minimum variance portfolio of the ten assets hold 153% in the 7-10 year bond and −67% in the 20+ year bond?
- ABecause of an optimizer error
- BBecause the long bond has a negative expected return
- CBecause the closed form forces the use of every asset
- DBecause both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio
Show answer
Answer D. Because both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio
Minimum variance only looks at the covariance. Two highly correlated assets with different volatilities make it possible to build an almost riskless position by buying one and selling the other in the right proportion. It is correct on paper and depends on that correlation holding.
What does a long-only constraint do to estimation error?
- ANothing: it only worsens the frontier
- BIt makes it worse, because it leaves fewer degrees of freedom
- CIt bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance
- DIt removes it completely
Show answer
Answer C. It bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance
With the true parameters, the constraint can only worsen the frontier (maximum Sharpe from 1.18 to 0.88). With estimated parameters, it avoids the long-short bets most sensitive to error and usually improves the out-of-sample result.
cvxpy rejects a problem with variance as the objective and a five-asset limit with binary variables if only CLARABEL, OSQP, SCS, HIGHS and SCIPY are installed. Why?
- ABecause it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once
- BBecause cardinality cannot be written in cvxpy
- CBecause variance is not convex
- DBecause data is missing
Show answer
Answer A. Because it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once
The problem is valid but not convex in the strict sense because of the integer variables. It needs an MIQP solver (SCIP, Gurobi, MOSEK, CPLEX) or a heuristic: solve the continuous version, keep the largest weights and optimize again.
For a target of 5% annual excess return, the unconstrained portfolio has a volatility of 4.3% and the long-only one, 5.8%. Which one should you choose?
- AThe unconstrained one, which dominates on paper
- BIt depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely
- CThe long-only one, because it always has less risk
- DNeither: use the equal-weighted one
Show answer
Answer B. It depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely
The unconstrained frontier is the best one if μ and Σ are exact. In practice you work with estimates, and the standard error of the mean of an equity index over 20 years is around 5 points: much more than the differences the long-short portfolio exploits.
Key points
- Mean-variance minimizes w′Σw with a target return and weights summing to 1. Without further constraints it has a closed-form solution: the frontier is a hyperbola and every efficient portfolio combines the minimum variance and the tangency portfolios.
- cvxpy solves the same problem (weight difference below one millionth) and accepts linear and convex constraints: long only, bounds, groups, leverage, tracking error. Cardinality needs a mixed-integer solver.
- Each constraint shifts the frontier to the right with the true parameters: maximum Sharpe 1.18 unconstrained, 0.88 long only.
- With estimated parameters, the sample frontier is optimistic: the maximum Sharpe estimated in 2006-2010 promises 1.75 and delivers 0.76; in Monte Carlo with five years, a median of 1.17 promised against 0.73 real.
- Expected returns are the most fragile part: with 20 years, the standard error of the mean of an equity index is around 5 points. Lessons 4.1.2 and 4.1.3 tackle the covariance and the mean.
Resources
Documentation
Books
- Markowitz, H. (1952): Portfolio Selection. The Journal of Finance 7(1), 77-91
- Merton, R. C. (1972): An Analytic Derivation of the Efficient Portfolio Frontier. Journal of Financial and Quantitative Analysis 7(4), 1851-1872
- Michaud, R. O. (1989): The Markowitz Optimization Enigma: Is 'Optimized' Optimal? Financial Analysts Journal 45(1), 31-42
- Jagannathan, R. and Ma, T. (2003): Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps. The Journal of Finance 58(4), 1651-1683