---
title: "4.1.1 Markowitz: efficient frontier and constraints"
description: "Markowitz mean-variance with the closed-form solution and with cvxpy, real mandate constraints and estimation error measured against the truth."
url: "https://pythonparatrading.com/en/python-trading-course/level-4/markowitz-efficient-frontier-and-constraints/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-4/markowitz-efficient-frontier-and-constraints.png"
---
# 4.1.1 · Markowitz: efficient frontier and real constraints
> Mean-variance optimization is exact in the mathematics and fragile in the statistics: it gives the best portfolio for expected returns and a covariance that are never known. This lesson solves it with the closed-form solution and with cvxpy, adds the constraints of a real mandate and measures, with a synthetic universe whose truth is published, how far the estimated frontier is from the one actually obtained.
- Course: [Level 4: Portfolios, risk, execution and live trading](https://pythonparatrading.com/en/python-trading-course/level-4/)
- Module: 4.1 · Portfolio theory and optimisation
- Duration: 70 min
- Prerequisites: [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/), [2.6.1 · CAGR, Sharpe, Sortino, Calmar, Omega, drawdown and Ulcer Index](https://pythonparatrading.com/en/python-trading-course/level-2/return-and-risk-metrics/)
## By the end of this lesson
- Set up the mean-variance portfolio as a quadratic problem and derive the efficient frontier and the minimum variance portfolio with the closed-form solution.
- Solve the same problem with cvxpy, check that it matches the closed form and add real constraints: long only, per-asset bounds, groups and leverage.
- Compute the maximum Sharpe portfolio with and without short selling, and explain why the unconstrained version bets on return differences that cannot be estimated.
- Measure estimation error: estimated frontier against realized frontier and against the true one, with real five-year windows and with Monte Carlo from the truth.
## A portfolio is a quadratic problem
Harry Markowitz (1952) turned the choice of a portfolio into an optimization problem: among all portfolios with the
same expected return, the best one is the one with the least variance. With $n$ assets, a weight vector $w$, the
expected returns in excess of the risk-free rate $\mu$ and the covariance matrix $\Sigma$:
$$
\mu_p = w^\top \mu, \qquad \sigma_p^2 = w^\top \Sigma\, w .
$$
The mean-variance portfolio for a target $m$ solves
$$
\min_w \; w^\top \Sigma\, w \quad \text{subject to} \quad w^\top \mu = m, \quad w^\top \mathbf{1} = 1 .
$$
It is a **quadratic program**: the objective is a convex quadratic form (because $\Sigma$ is positive semidefinite)
and the constraints are linear. Sweeping $m$ gives the **efficient frontier**, the curve of the portfolios that cannot
improve their return without increasing their risk. Any point below it is an inefficient portfolio.
The problem has two inputs, $\mu$ and $\Sigma$, and neither is known. They are estimated with past data, and the
optimizer treats them as if they were exact. That is the idea that runs through the whole lesson:
```mermaid
flowchart TB
H[History] --> E["Estimate μ and Σ"]
E --> O[Optimizer]
O --> W[Weights]
W --> R[Real result]
V["True μ and Σ"] --> R
```
What the optimizer sees are the estimates; what the portfolio gets depends on the true values. In real markets the
true values are never known. In the level 4 data they are: the universe comes from a published factor model, so the
distance between what is promised and what is obtained can be measured.
## The universe and its truth
The data is 38 synthetic total return indices in USD (equities by region, sector and style, bonds, credit, real
estate, commodities and money market) from 2006 to 2025, with the real NYSE sessions. It is not market data: it
imitates the profile of real indices (volatility, crises, correlations), with the crises of 2008, 2020 and 2022
written into a script. To keep the figures readable, the lesson works with ten of them, one or two per asset class.
The excess return of each day is the simple return minus the risk-free rate of the day:
```python
import numpy as np
import pandas as pd
precios = pd.read_parquet("universo4_precios.parquet")
precios = precios.set_index("fecha")
fac = pd.read_parquet("universo4_factores_diarios.parquet")
rf = fac.set_index("fecha")["rf"]
r = precios.pct_change()
r.iloc[0] = precios.iloc[0] / 100 - 1 # base 100 the day before
exceso = r.sub(rf, axis=0)
activos = ["RV_EEUU", "RV_EUR", "RV_JAP", "RV_EMG", "INM_EEUU",
"RF_T710", "RF_T20", "RF_CIG", "RF_CHY", "MP_ORO"]
R = exceso[activos]
print(R.shape, R.index[0].date(), R.index[-1].date())
```
```output
(5031, 10) 2006-01-03 2025-12-31
```
Returns are **simple**, not logarithmic. The return of a portfolio is the weighted sum of the simple returns of its
assets; with logarithms that sum is not exact and the portfolio's Sharpe comes out lower than it is.
The “truth” is in two files: `universo4_verdad.csv` holds the expected excess return of each asset over the period
(`mu_verdadero`) and `universo4_cov_verdad.csv`, the true annual covariance per regime and the overall one (`total`),
which is what the full sample estimates:
```python
verdad = pd.read_csv("universo4_verdad.csv", index_col="activo")
cov_v = pd.read_csv("universo4_cov_verdad.csv")
total = cov_v[cov_v["regimen"] == "total"].set_index("activo")
mu_v = verdad.loc[activos, "mu_verdadero"].to_numpy()
S_v = total.loc[activos, activos].to_numpy()
mu_m = R.mean().to_numpy() * 252
S_m = R.cov().to_numpy() * 252
tabla = pd.DataFrame({
"mu_verdad": mu_v, "mu_muestra": mu_m,
"vol_verdad": np.sqrt(np.diag(S_v)),
"vol_muestra": np.sqrt(np.diag(S_m))}, index=activos)
print((100 * tabla).round(2))
ee = np.sqrt(np.diag(S_m) / 20)
print("standard error of the mean (20 years):", (100 * ee).round(1))
```
```output
mu_verdad mu_muestra vol_verdad vol_muestra
RV_EEUU 9.62 8.79 20.37 20.59
RV_EUR 5.82 3.20 22.63 22.81
RV_JAP 5.45 3.84 21.86 22.00
RV_EMG 3.80 5.46 26.54 27.16
INM_EEUU 9.17 8.98 23.35 23.46
RF_T710 1.60 0.49 5.92 5.90
RF_T20 3.68 1.54 13.69 13.58
RF_CIG 2.76 2.13 5.56 5.59
RF_CHY 4.96 4.68 10.92 10.48
MP_ORO 9.98 10.03 14.59 14.17
standard error of the mean (20 years): [4.6 5.1 4.9 6.1 5.2 1.3 3. 1.3 2.3 3.2]
```
Volatilities are estimated well: with 5,031 sessions, the sample one is within 0.7 points of the true one for every
asset. Means are not. `RV_EUR` returns 3.20% in the sample against an expected 5.82%; the long bond, 1.54% against
3.68%. It is not a flaw in the data: the standard error of an annual mean is the volatility divided by the square root
of the number of years, and with 20 years and a 20% volatility it comes to almost 5 points. It is the law that governs
the rest of the lesson: **variance is estimated with days; the mean, with years**.
## The closed-form solution
With no constraints other than the target and the sum of weights, the problem is solved with Lagrange multipliers
(Merton, 1972). With the constants
$$
A = \mathbf{1}^\top \Sigma^{-1} \mathbf{1}, \quad B = \mathbf{1}^\top \Sigma^{-1} \mu, \quad
C = \mu^\top \Sigma^{-1} \mu, \quad D = AC - B^2,
$$
the frontier portfolio with return $m$ and its variance are
$$
w(m) = \frac{C - Bm}{D}\, \Sigma^{-1}\mathbf{1} + \frac{Am - B}{D}\, \Sigma^{-1}\mu, \qquad
\sigma^2(m) = \frac{A m^2 - 2Bm + C}{D}.
$$
The variance is a parabola in $m$, so the frontier in the (volatility, return) plane is a hyperbola. Its vertex is the
**minimum variance portfolio**, $w_{mv} = \Sigma^{-1}\mathbf{1}/A$, with volatility $\sqrt{1/A}$ and return $B/A$.
And since $w(m)$ is linear in $m$, every efficient portfolio is a combination of any two efficient portfolios: this is
the two-fund theorem.
```python
iS = np.linalg.inv(S_v)
uno = np.ones(len(activos))
A = uno @ iS @ uno
B = uno @ iS @ mu_v
C = mu_v @ iS @ mu_v
D = A * C - B**2
w_mv = iS @ uno / A
def w_frontera(m):
return ((C - B * m) * iS @ uno + (A * m - B) * iS @ mu_v) / D
w5 = w_frontera(0.05)
cerrada = pd.DataFrame({"min_var": w_mv, "frontera_5": w5},
index=activos)
print((100 * cerrada).round(1))
print(f"min. variance: vol {np.sqrt(1 / A):.2%}, mu {B / A:.2%}")
print(f"5% frontier: vol {np.sqrt(w5 @ S_v @ w5):.2%}, "
f"formula {np.sqrt((A * .05**2 - 2 * B * .05 + C) / D):.2%}")
```
```output
min_var frontera_5
RV_EEUU 2.7 41.9
RV_EUR 0.2 -7.3
RV_JAP 0.4 -2.0
RV_EMG -0.6 -18.2
INM_EEUU -1.8 -3.9
RF_T710 152.9 67.6
RF_T20 -67.1 -35.0
RF_CIG 14.0 46.2
RF_CHY -1.7 -8.2
MP_ORO 1.0 19.0
min. variance: vol 1.81%, mu 0.48%
5% frontier: vol 4.33%, formula 4.33%
```
The minimum variance portfolio has a volatility of 1.81%, less than the 7-10 year bond on its own (5.9%). It gets there
with 153% in that bond and −67% in the 20+ year bond: both load on the same rate factor and, in the right proportion,
they cancel out. It is correct on paper and it is a leveraged portfolio that depends on that correlation being exactly
the estimated one. The 5% efficient portfolio also sells short: emerging markets (−18%), the long bond (−35%) and
Europe.
## The same with cvxpy
The closed form stops existing as soon as an inequality appears (long only, a bound). That is what convex optimization
is for. **cvxpy** lets you write the problem almost as on the whiteboard and picks a suitable solver; here CLARABEL is
fixed, an interior-point solver for conic problems that ships with cvxpy. Before adding anything, it is worth checking
that it reproduces the formula:
```python
import cvxpy as cp
w = cp.Variable(len(activos))
objetivo = cp.Parameter()
riesgo = cp.quad_form(w, cp.psd_wrap(S_v))
prob = cp.Problem(cp.Minimize(riesgo),
[cp.sum(w) == 1, mu_v @ w == objetivo])
objetivo.value = 0.05
prob.solve(solver="CLARABEL")
print(prob.status)
print("max. difference from the formula < 1e-6:",
np.abs(w.value - w5).max() < 1e-6)
```
```output
optimal
max. difference from the formula < 1e-6: True
```
`cp.psd_wrap` tells cvxpy that the matrix is positive semidefinite, instead of checking it with a decomposition that,
with nearly singular matrices, can fail through rounding. `cp.Parameter` makes it possible to change the target and
solve again without rebuilding the problem, which is what is needed to sweep the frontier.
## Real constraints
A real mandate never lets you short 35% of the long bond. The usual constraints are written in one line each:
- **Long only**: $w \ge 0$. The rule for almost all mutual funds and pension plans.
- **Per-asset bounds**: $w_i \le 30\%$, so that no position dominates.
- **Groups**: equities (stocks and real estate) between 20% and 60%, for example in a balanced profile.
- **Gross leverage**: $\sum_i |w_i| \le 1.6$, a 130/30 portfolio (130% long, 30% short). In cvxpy,
`cp.norm1(w) <= 1.6`.
```python
rv = [0, 1, 2, 3, 4] # stocks and real estate
reglas = {
"unconstrained": [],
"130/30": [cp.norm1(w) <= 1.6],
"long only": [w >= 0],
"mandate": [w >= 0, w <= 0.30, cp.sum(w[rv]) >= 0.2,
cp.sum(w[rv]) <= 0.6],
}
problemas = {
k: cp.Problem(cp.Minimize(riesgo),
[cp.sum(w) == 1, mu_v @ w >= objetivo] + extra)
for k, extra in reglas.items()}
filas = {}
for k, pr in problemas.items():
pr.solve(solver="CLARABEL")
filas[k] = list(w.value) + [np.sqrt(pr.value)]
pesos = pd.DataFrame(filas, index=activos + ["vol"])
print((100 * pesos).round(1) + 0) # + 0 removes the -0.0
```
```output
unconstrained 130/30 long only mandate
RV_EEUU 41.9 26.3 11.4 20.0
RV_EUR -7.3 0.0 0.0 0.0
RV_JAP -2.0 0.0 0.0 0.0
RV_EMG -18.2 -15.3 0.0 0.0
INM_EEUU -3.9 -1.1 0.0 0.0
RF_T710 67.6 63.0 25.9 30.0
RF_T20 -35.0 -13.5 0.0 3.0
RF_CIG 46.2 13.8 35.2 30.0
RF_CHY -8.2 8.3 4.6 0.8
MP_ORO 19.0 18.5 23.0 16.2
vol 4.3 4.7 5.8 6.1
```
The target return becomes an inequality ($\mu^\top w \ge m$): below the return of the minimum variance portfolio of
each set of rules, the optimum is that portfolio, and the frontier is cut at its vertex. The 130/30 portfolio keeps
part of the short positions (emerging markets, long bond) and its volatility stays at 4.7%. With long only they all
disappear and the portfolio concentrates in five assets, four of them above 10%: the 7-10 year bond, investment grade
credit, gold and US stocks. Volatility rises to 5.8%. The mandate forces a 20% equity allocation and caps the bond and
credit at 30%: 6.1%. The full frontier shows where each rule bites:
```python
objetivos = np.round(np.arange(0.03, 0.0951, 0.01), 3)
fr = {}
for k, pr in problemas.items():
vols = []
for m in objetivos:
objetivo.value = m
pr.solve(solver="CLARABEL")
ok = pr.status == "optimal"
vols.append(np.sqrt(pr.value) if ok else np.nan)
fr[k] = vols
fr = pd.DataFrame(fr, index=objetivos)
fr.index.name = "objetivo"
print((100 * fr).round(2))
```
```output
unconstrained 130/30 long only mandate
objetivo
0.03 2.84 3.48 4.83 6.06
0.04 3.56 4.05 5.11 6.06
0.05 4.33 4.72 5.83 6.11
0.06 5.14 5.46 6.82 6.91
0.07 5.96 6.23 8.02 8.22
0.08 6.80 7.08 9.27 10.31
0.09 7.64 8.12 10.75 13.73
```
*Fig. — True efficient frontier of ten assets under four rule sets: each constraint shifts it to the right and truncates it.* Four curves of volatility against expected return with the true parameters. Unconstrained, the frontier starts at 1.8% volatility; the 130/30 one lies slightly to the right; the long-only one starts at 4.8% and ends at gold (9.98% return); the mandate (highlighted: long only, at most 30% per asset and equities between 20% and 60%) starts at 6.1% and needs 13.7% volatility for 9%.
Each constraint shifts the frontier to the right: for the same return, more volatility. And it cuts it short: with
long only, no target can exceed the return of the best asset (gold, 9.98%); with the mandate, which caps each position
at 30%, the maximum falls below that 9.98% and a 9% target requires a volatility of 13.7%. With the true parameters
constraints can only worsen the result; the interesting question is what happens when the parameters are estimated.
Constraints fall into three families, according to the solver they need:
```mermaid
flowchart LR
P[Constraints] --> L[Linear]
P --> C[Convex]
P --> M[Integer]
L --> L1["w ≥ 0, bounds"]
L --> L2[Groups]
C --> C1[Leverage]
C --> C2[Tracking error]
M --> M1[Cardinality]
M --> M2[Minimum lot]
```
Linear and convex constraints (leverage is a 1-norm; tracking error against an index,
$(w - w_b)^\top \Sigma (w - w_b) \le TE^2$, a quadratic constraint) are handled by any conic solver. Integer ones, such
as “at most five assets” or “lots of 100 shares”, turn the problem into a mixed-integer one:
```python
import warnings
warnings.filterwarnings("ignore", category=UserWarning,
message="Your problem is mixed-integer")
z = cp.Variable(len(activos), boolean=True)
cardinal = cp.Problem(cp.Minimize(riesgo), [
cp.sum(w) == 1, mu_v @ w >= 0.05, w >= 0, w <= z,
cp.sum(z) <= 3])
try:
cardinal.solve()
except cp.error.SolverError as e:
print(str(e).split(".")[0])
```
```output
You need a mixed-integer solver for this model
```
The warnings filter silences, by its message, the `UserWarning` with which cvxpy suggests installing SCIP; without it,
the warning would appear in the output with the file path. Among the solvers of this environment (CLARABEL, OSQP, SCS,
HiGHS and SciPy) there is none that solves a mixed-integer quadratic problem: HiGHS solves integer linear programs and
continuous quadratic ones, but not both at once. The options are to install one that does (SCIP is free; Gurobi, MOSEK
and CPLEX, commercial) or a reasonable heuristic: solve the continuous problem, keep the assets with the largest weight
and optimize again with only those. Transaction costs and turnover, which also change the shape of the problem, are
the topic of lesson 4.1.5.
## Maximum Sharpe
If you can invest in the risk-free asset, the best risky portfolio is the one that maximizes the Sharpe, the
**tangency portfolio**. Without constraints its direction is $\Sigma^{-1}\mu$; the size is set by the risk you want to
take. With constraints, the ratio $\mu^\top w / \sqrt{w^\top \Sigma w}$ is not concave, but it becomes a quadratic
program with a change of variable: minimize $y^\top \Sigma y$ with $\mu^\top y = 1$ and the homogeneous constraints on
$y$ (for example $y \ge 0$), and then $w = y / \mathbf{1}^\top y$.
```python
def sharpe(w, mu, S):
return w @ mu / np.sqrt(w @ S @ w)
def max_sharpe(mu, S):
y = cp.Variable(len(mu))
pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
[mu @ y == 1, y >= 0])
pr.solve(solver="CLARABEL")
return y.value / y.value.sum()
w_tan = iS @ mu_v
w_tan = w_tan * 0.10 / np.sqrt(w_tan @ S_v @ w_tan) # 10% vol.
w_lo = max_sharpe(mu_v, S_v)
tang = pd.DataFrame({"sin_limites_vol10": w_tan, "solo_largos": w_lo},
index=activos)
print((100 * tang).round(1))
print("Sharpe unconstrained:", sharpe(w_tan, mu_v, S_v).round(3))
print("Sharpe long only:", sharpe(w_lo, mu_v, S_v).round(3))
print("sum of unconstrained weights:", w_tan.sum().round(2))
```
```output
sin_limites_vol10 solo_largos
RV_EEUU 100.3 14.6
RV_EUR -18.4 0.0
RV_JAP -5.4 0.0
RV_EMG -44.3 0.0
INM_EEUU -7.5 0.0
RF_T710 -20.4 0.0
RF_T20 -4.2 0.0
RF_CIG 97.0 52.9
RF_CHY -18.3 0.0
MP_ORO 45.9 32.5
Sharpe unconstrained: 1.178
Sharpe long only: 0.879
sum of unconstrained weights: 1.25
```
With the true parameters, the unconstrained portfolio reaches a Sharpe of 1.18, and the long-only one, 0.88. The
difference comes from long-short positions: buying investment grade credit (97%) and selling Treasury bonds, emerging
markets and Europe. It exploits differences in expected return of one or two points between highly correlated assets.
With the truth in front of you, those differences exist. With 20 years of data, the standard error of each mean is 1.3
to 6 points: the unconstrained portfolio bets precisely on what is measured worst.
The sum of the unconstrained weights for a 10% volatility is not 1: with a risk-free asset, the difference up to 1 goes
to cash or is borrowed at the risk-free rate (here the portfolio sums to 1.25: 25% financed). Normalizing the tangency
portfolio so that it sums to 1, $w = \Sigma^{-1}\mu / (\mathbf{1}^\top \Sigma^{-1}\mu)$, is dangerous with estimated
means: if the denominator approaches zero, the weights blow up, and if it changes sign the portfolio flips.
## Estimation error
Now the experiment that matters. A manager who optimizes on a date uses the data available, for example the last five
years. For each five-year window, the long-only maximum Sharpe portfolio is computed with the window's mean and
covariance, and its Sharpe is measured twice: the one it promises (with the window's estimates) and the one it really
has (with the true $\mu$ and $\Sigma$):
```python
filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
("2016", "2020"), ("2021", "2025")]:
Rv = R.loc[a0:a1]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
w_e = max_sharpe(mu_e, S_e)
filas.append({"ventana": f"{a0}-{a1}",
"sharpe_estimado": sharpe(w_e, mu_e, S_e),
"sharpe_real": sharpe(w_e, mu_v, S_v),
"activos": int((w_e > 0.01).sum()),
"mayor_peso": activos[int(w_e.argmax())]})
ventanas = pd.DataFrame(filas)
print(ventanas.round(2))
print("true maximum:", sharpe(w_lo, mu_v, S_v).round(2))
```
```output
ventana sharpe_estimado sharpe_real activos mayor_peso
0 2006-2010 1.75 0.76 4 RF_CIG
1 2011-2015 0.49 0.53 3 RF_T20
2 2016-2020 1.16 0.71 5 MP_ORO
3 2021-2025 1.11 0.80 3 MP_ORO
true maximum: 0.88
```
*Fig. — Long-only maximum-Sharpe portfolio estimated in each five-year window: promised Sharpe (grey) and actual Sharpe with the true parameters (amber).* Four rows, one per window. 2006-2010: promises 1.75 and delivers 0.76. 2011-2015: promises 0.49 and delivers 0.53. 2016-2020: promises 1.16 and delivers 0.71. 2021-2025: promises 1.11 and delivers 0.80. A vertical line marks the true maximum, 0.88, which no window reaches.
The 2006-2010 window promises a Sharpe of 1.75 and delivers 0.76. No window reaches the true maximum of 0.88: it is
impossible by construction, because that maximum belongs to the optimal weights under the truth, and any other set of
weights falls below it. What can be avoided is believing the promised number. Three of the four windows promise more
than they deliver, and each one picks a different portfolio, of three to five assets, with the largest weight in
credit, in the long bond or in gold depending on the window. The only one that promises less (2011-2015) does so
because everything returned little in that window.
The figure at the top shows it over the whole frontier, with the 2021-2025 window: the estimated frontier (what the
manager sees on screen) lies above the true one over almost its whole range (they only cross below 6% volatility), and
the realized one (the same weights evaluated with the truth) lies below. The estimated frontier is optimistic for two
reasons that add up. The first, the error in the mean: in that window gold returned 13.7% and the long bond −11.6% a
year in excess, far from their expected 10.0% and 3.7%. The second, selection: the optimizer picks exactly the assets
whose error was favorable. That is why Michaud (1989) called mean-variance optimization an “error maximizer”.
```python
Rv = R.loc["2021":"2025"]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
est = cp.Problem(cp.Minimize(cp.quad_form(w, cp.psd_wrap(S_e))),
[cp.sum(w) == 1, mu_e @ w >= objetivo, w >= 0])
ver = problemas["long only"]
filas = []
print("gold and long bond in the window:",
(100 * mu_e[[9, 6]]).round(1))
for m in [0.04, 0.07, 0.09]:
objetivo.value = m
est.solve(solver="CLARABEL")
w_e = w.value.copy()
ver.solve(solver="CLARABEL")
filas.append({"objetivo": m,
"vol_estimada": np.sqrt(w_e @ S_e @ w_e),
"mu_real": w_e @ mu_v,
"vol_real": np.sqrt(w_e @ S_v @ w_e),
"vol_verdadera_al_objetivo": np.sqrt(ver.value)})
print((100 * pd.DataFrame(filas)).round(2))
```
```output
gold and long bond in the window: [ 13.7 -11.6]
objetivo vol_estimada mu_real vol_real vol_verdadera_al_objetivo
0 4.0 5.89 4.97 6.60 5.11
1 7.0 7.12 6.11 8.33 8.02
2 9.0 8.20 7.21 9.29 10.75
```
For a 7% target, the window says a volatility of 7.1% is enough. Those weights really return 6.1% with 8.3%
volatility; with the truth, 7% is achieved with 8.0%. The manager loses on both sides: less return and more risk than
expected.
## Monte Carlo from the truth
The real windows mix two things: estimation error and the regime changes of the script (the 2008 crisis falls entirely
in the first window). To isolate estimation error, samples of independent normal returns are simulated with exactly
the true $\mu$ and $\Sigma$, each sample is optimized and the real Sharpe of each portfolio is measured. Two hundred
samples of five and of twenty years:
```python
rng = np.random.default_rng(41)
L = np.linalg.cholesky(S_v / 252)
for anos in (5, 20):
res = []
for _ in range(200):
X = mu_v / 252 + rng.standard_normal((252 * anos, 10)) @ L.T
m_e = X.mean(axis=0) * 252
if (m_e <= 0).all():
continue
S_e = np.cov(X, rowvar=False) * 252
w_e = max_sharpe(m_e, S_e)
res.append((sharpe(w_e, m_e, S_e), sharpe(w_e, mu_v, S_v)))
res = np.array(res)
p5, p95 = np.percentile(res[:, 1], [5, 95])
print(f"{anos} years: promised {np.median(res[:, 0]):.2f}, "
f"real {np.median(res[:, 1]):.2f} ({p5:.2f} to {p95:.2f}), "
f"overpromises {np.mean(res[:, 0] > res[:, 1]):.0%}")
```
```output
5 years: promised 1.17, real 0.73 (0.49 to 0.85), overpromises 85%
20 years: promised 0.98, real 0.82 (0.73 to 0.87), overpromises 78%
```
With five years, the median promised Sharpe is 1.17 and the median real one, 0.73; in 85% of the samples more is
promised than obtained, and one in twenty portfolios falls below 0.49. With twenty years, the gap narrows (0.98 against
0.82) but does not disappear. And that is with ideal data: normal, independent and without regime changes. In real
markets, with fat tails and correlations that change in crises, the error is larger.
## What to do about the error
Mean-variance is not abandoned; it is protected. The usual defenses, and where they appear in the course:
| Defense | What it tackles | Lesson |
|---|---|---|
| Constraints (long only, bounds, groups) | Long-short bets on small differences | This one |
| Better estimators of Σ (shrinkage, factors, denoising) | Covariance error | 4.1.2 |
| Equilibrium and views (Black-Litterman) | Mean error | 4.1.3 |
| Portfolios that do not use μ (minimum variance, risk parity, HRP) | They ignore the most fragile part | 4.1.4 |
| Penalizing turnover and costs | Rebalances that chase noise | 4.1.5 |
Jagannathan and Ma (2003) showed that imposing long only on a minimum variance portfolio is equivalent to shrinking the
covariances of the assets the optimizer would want to sell: a “wrong” constraint from the point of view of theory that
acts as a better estimator. That is why a mandate with reasonable bounds is not only a commercial requirement; it is
also a form of regularization.
Typical mistakes when optimizing:
- **Using log returns** for portfolio means and covariances, or annualizing the log mean as if it were simple.
- **Mixing total and excess returns**: the Sharpe and the tangency portfolio are computed in excess of the risk-free
rate.
- **Normalizing the tangency portfolio** with $\mathbf{1}^\top\Sigma^{-1}\mu$ when that number is close to zero.
- **Believing the estimated frontier**: it is the frontier of the estimates, not that of the market.
- **Not fixing the solver** or not checking `prob.status`: an infeasible problem returns `None` in the weights and
`infeasible` in the status, not an exception.
## Exercises
1. Repeat the unconstrained minimum variance portfolio with the 38 assets of the universe and the overall true
covariance. What weight does the money market (`MONET`) get, and why?
**Solution.**
```python
todos = list(total.index)
S38 = total.loc[todos, todos].to_numpy()
iS38 = np.linalg.inv(S38)
u38 = np.ones(len(todos))
w38 = pd.Series(iS38 @ u38 / (u38 @ iS38 @ u38), index=todos)
print(f"vol: {np.sqrt(1 / (u38 @ iS38 @ u38)):.3%}")
print((100 * w38).round(1).sort_values().iloc[[0, 1, -2, -1]])
```
```output
vol: 0.152%
RF_T710 -2.2
RF_T20 -1.2
RF_T13 8.8
MONET 92.5
dtype: float64
```
The money market has a volatility of 0.2% and almost no correlation with anything else: the minimum variance portfolio
takes almost all of it, and the volatility falls below its own. It is the right answer to the wrong question. If there
is an almost riskless asset, minimum variance over risky assets reduces to it; that is why portfolios are optimized in
excess of the risk-free rate and without the money market inside the risky universe.
2. Repeat the window experiment with a 30% bound per asset in addition to long only. Does the gap between the promised
and the real Sharpe narrow?
**Solution.**
```python
def max_sharpe_cota(mu, S, cota=0.30):
y = cp.Variable(len(mu))
k = cp.Variable(nonneg=True) # y = k·w, so y <= cota·k
pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
[mu @ y == 1, y >= 0, y <= cota * k,
cp.sum(y) == k])
pr.solve(solver="CLARABEL")
return y.value / y.value.sum()
filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
("2016", "2020"), ("2021", "2025")]:
Rv = R.loc[a0:a1]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
w_c = max_sharpe_cota(mu_e, S_e)
filas.append({"ventana": f"{a0}-{a1}",
"estimado": sharpe(w_c, mu_e, S_e),
"real": sharpe(w_c, mu_v, S_v)})
print(pd.DataFrame(filas).round(2))
```
```output
ventana estimado real
0 2006-2010 1.72 0.75
1 2011-2015 0.48 0.54
2 2016-2020 1.16 0.70
3 2021-2025 1.02 0.67
```
The bound is a non-homogeneous constraint ($w_i \le 0.3$), so in the change of variable it has to be multiplied by the
scale factor $k = \mathbf{1}^\top y$. The bound barely changes the first three windows (the portfolios were already
spread out) and in 2021-2025 it lowers the real Sharpe from 0.80 to 0.67: it limits gold, which under the truth is
precisely the most attractive asset. A constraint does not recover the truth: it reduces how much is bet on each
window's error, and that sometimes has a cost. Its value lies in the worst case, not in the average of four windows.
3. With 20 years of data, the standard error of the mean of `RV_EEUU` is about 4.6 points. How many years would it take
to bring it down to 1 point? And why does using daily data instead of annual data not shorten that period?
**Solution.**
With $A$ years of data, the standard error of the annual mean is $\sigma/\sqrt{A}$: with $\sigma \approx 20.6\%$,
getting down to 1 point requires $(20.6)^2 \approx 424$ years. Using daily data does not help: the annual mean
estimated with $N$ days is $252 \cdot \overline{r}$, with standard error $252 \cdot \sigma_d/\sqrt{N} = \sigma/\sqrt{N/252}$,
which only depends on the number of years the sample covers, not on how many observations it contains. Frequency does
improve the estimation of the variance, which depends on the number of observations. That is why portfolios that do
not need $\mu$ (lesson 4.1.4) and methods that take it from equilibrium (lesson 4.1.3) exist.
## Self-assessment
1. The maximum Sharpe portfolio estimated with 2006-2010 data promises a Sharpe of 1.75. What Sharpe does it have with the true parameters?
- A. 1.75: the optimization is exact
- B. 0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance
- C. More than 1.75, because the sample is pessimistic
- D. It cannot be known
*Answer B: 0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance.* The optimizer maximizes over estimates with error and picks precisely the assets whose error was favorable. The Sharpe estimated on the optimization's own sample is biased upward; the real Sharpe of those weights falls below the true maximum. In Monte Carlo with five years, the estimate exceeds the real one in 85% of cases.
2. Why does the unconstrained minimum variance portfolio of the ten assets hold 153% in the 7-10 year bond and −67% in the 20+ year bond?
- A. Because of an optimizer error
- B. Because the long bond has a negative expected return
- C. Because the closed form forces the use of every asset
- D. Because both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio
*Answer D: Because both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio.* Minimum variance only looks at the covariance. Two highly correlated assets with different volatilities make it possible to build an almost riskless position by buying one and selling the other in the right proportion. It is correct on paper and depends on that correlation holding.
3. What does a long-only constraint do to estimation error?
- A. Nothing: it only worsens the frontier
- B. It makes it worse, because it leaves fewer degrees of freedom
- C. It bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance
- D. It removes it completely
*Answer C: It bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance.* With the true parameters, the constraint can only worsen the frontier (maximum Sharpe from 1.18 to 0.88). With estimated parameters, it avoids the long-short bets most sensitive to error and usually improves the out-of-sample result.
4. cvxpy rejects a problem with variance as the objective and a five-asset limit with binary variables if only CLARABEL, OSQP, SCS, HIGHS and SCIPY are installed. Why?
- A. Because it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once
- B. Because cardinality cannot be written in cvxpy
- C. Because variance is not convex
- D. Because data is missing
*Answer A: Because it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once.* The problem is valid but not convex in the strict sense because of the integer variables. It needs an MIQP solver (SCIP, Gurobi, MOSEK, CPLEX) or a heuristic: solve the continuous version, keep the largest weights and optimize again.
5. For a target of 5% annual excess return, the unconstrained portfolio has a volatility of 4.3% and the long-only one, 5.8%. Which one should you choose?
- A. The unconstrained one, which dominates on paper
- B. It depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely
- C. The long-only one, because it always has less risk
- D. Neither: use the equal-weighted one
*Answer B: It depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely.* The unconstrained frontier is the best one if μ and Σ are exact. In practice you work with estimates, and the standard error of the mean of an equity index over 20 years is around 5 points: much more than the differences the long-short portfolio exploits.
## Key points
- Mean-variance minimizes w′Σw with a target return and weights summing to 1. Without further constraints it has a closed-form solution: the frontier is a hyperbola and every efficient portfolio combines the minimum variance and the tangency portfolios.
- cvxpy solves the same problem (weight difference below one millionth) and accepts linear and convex constraints: long only, bounds, groups, leverage, tracking error. Cardinality needs a mixed-integer solver.
- Each constraint shifts the frontier to the right with the true parameters: maximum Sharpe 1.18 unconstrained, 0.88 long only.
- With estimated parameters, the sample frontier is optimistic: the maximum Sharpe estimated in 2006-2010 promises 1.75 and delivers 0.76; in Monte Carlo with five years, a median of 1.17 promised against 0.73 real.
- Expected returns are the most fragile part: with 20 years, the standard error of the mean of an equity index is around 5 points. Lessons 4.1.2 and 4.1.3 tackle the covariance and the mean.
## Resources
### Research
- [Price Discovery & Arbitrage Price Theory](https://pythonparatrading.com/en/arbitrage-pricing-theory-and-price-discovery-2/) · Research-0006 · 6 min
### Documentation
- [cvxpy: mixed-integer programs and available solvers](https://www.cvxpy.org/tutorial/advanced/index.html) · cvxpy.org
### Books
- [Markowitz, H. (1952): Portfolio Selection. The Journal of Finance 7(1), 77-91](https://doi.org/10.1111/j.1540-6261.1952.tb01525.x) · doi.org
- [Merton, R. C. (1972): An Analytic Derivation of the Efficient Portfolio Frontier. Journal of Financial and Quantitative Analysis 7(4), 1851-1872](https://doi.org/10.2307/2329621) · doi.org
- [Michaud, R. O. (1989): The Markowitz Optimization Enigma: Is 'Optimized' Optimal? Financial Analysts Journal 45(1), 31-42](https://doi.org/10.2469/faj.v45.n1.31) · doi.org
- [Jagannathan, R. and Ma, T. (2003): Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps. The Journal of Finance 58(4), 1651-1683](https://doi.org/10.1111/1540-6261.00580) · doi.org
### Data
- [Level 4 data: universe of 38 synthetic indices, 2006-2025 (Parquet)](https://pythonparatrading.com/curso/datos/nivel-4/universo4_precios.parquet) · Download · universo4_precios.parquet
Next lesson: [4.1.2 · Covariance estimation: shrinkage, factor models and denoising](https://pythonparatrading.com/en/python-trading-course/level-4/covariance-estimation-shrinkage-and-denoising/)