Lesson 4.1.1 · 70 min

Markowitz: efficient frontier and real constraints

Mean-variance optimization is exact in the mathematics and fragile in the statistics: it gives the best portfolio for expected returns and a covariance that are never known. This lesson solves it with the closed-form solution and with cvxpy, adds the constraints of a real mandate and measures, with a synthetic universe whose truth is published, how far the estimated frontier is from the one actually obtained.

Fig. — Long-only efficient frontier of ten assets: estimated with 2021-2025, realised by those same weights, and true
01/08
Module
4.1 · Portfolio theory and optimisation
Duration
70 min
Updated
Level 4 lessons
  1. 4.1 · Portfolio theory and optimisation

    1. 4.1.1 Markowitz: efficient frontier and real constraints
    2. 4.1.2 Covariance estimation: shrinkage, factor models and denoising
    3. 4.1.3 Black-Litterman: from market equilibrium to views
    4. 4.1.4 Risk parity, minimum variance, HRP and HERC
    5. 4.1.5 Optimization with CVaR, drawdown, turnover and costs
    6. 4.1.6 Factor models: CAPM, Fama-French, PCA and attribution
    7. 4.1.7 Rebalancing and strategy portfolios
    8. 4.1.8 Portfolio tools: Riskfolio-Lib, PyPortfolioOpt, skfolio and bt
  2. 4.2 · Risk management and position sizing

    1. 4.2.1 Risk per trade, stops and risk of ruin
    2. 4.2.2 Kelly criterion and volatility targeting: how much to risk
    3. 4.2.3 VaR, Expected Shortfall and VaR backtesting
    4. 4.2.4 Stress testing and correlation, liquidity, gap and counterparty risk
    5. 4.2.5 Drawdown control and kill switch
    6. 4.2.6 Derivatives risk: portfolio Greeks, margin and liquidations
  3. 4.3 · Microstructure and execution

    1. 4.3.1 Implementation shortfall, VWAP, TWAP and Almgren-Chriss
    2. 4.3.2 Order flow imbalance, microstructure signals and latency
  4. 4.4 · Brokers and APIs

    1. 4.4.1 Choosing a broker from Spain or the EU: regulation, assets, costs and API
    2. 4.4.2 Interactive Brokers: TWS and IB Gateway, ib_async, contracts, data and orders
    3. 4.4.3 Alpaca, Tradier, MetaTrader 5 with Python and OANDA v20
    4. 4.4.4 Crypto exchange APIs: ccxt, Binance, Bybit, Kraken and Coinbase
    5. 4.4.5 Secure credentials: environment variables, secret managers and least privilege
  5. 4.5 · Live trading system

    1. 4.5.1 Live system architecture: from data to reconciliation
    2. 4.5.2 Order management: states, partial fills, cancellations and reconnections
    3. 4.5.3 Paper trading, shadow trading and slippage monitoring
    4. 4.5.4 Task scheduling and alerts: cron, APScheduler, Prefect, Telegram, Slack and email
    5. 4.5.5 Fault tolerance: idempotency, persistent state and reconciliation at startup
    6. 4.5.6 NautilusTrader or LEAN live: from backtest to production with the same code

A portfolio is a quadratic problem

Harry Markowitz (1952) turned the choice of a portfolio into an optimization problem: among all portfolios with the same expected return, the best one is the one with the least variance. With nn assets, a weight vector ww, the expected returns in excess of the risk-free rate μ\mu and the covariance matrix Σ\Sigma:

μp=w⊤μ,σp2=w⊤Σ w.\mu_p = w^\top \mu, \qquad \sigma_p^2 = w^\top \Sigma\, w .

The mean-variance portfolio for a target mm solves

min⁡w  w⊤Σ wsubject tow⊤μ=m,w⊤1=1.\min_w \; w^\top \Sigma\, w \quad \text{subject to} \quad w^\top \mu = m, \quad w^\top \mathbf{1} = 1 .

It is a quadratic program: the objective is a convex quadratic form (because Σ\Sigma is positive semidefinite) and the constraints are linear. Sweeping mm gives the efficient frontier, the curve of the portfolios that cannot improve their return without increasing their risk. Any point below it is an inefficient portfolio.

The problem has two inputs, μ\mu and Σ\Sigma, and neither is known. They are estimated with past data, and the optimizer treats them as if they were exact. That is the idea that runs through the whole lesson:

HistoryEstimate μ and ΣOptimizerWeightsReal resultTrue μ and Σ
Mermaid source
flowchart TB
  H[History] --> E["Estimate μ and Σ"]
  E --> O[Optimizer]
  O --> W[Weights]
  W --> R[Real result]
  V["True μ and Σ"] --> R

What the optimizer sees are the estimates; what the portfolio gets depends on the true values. In real markets the true values are never known. In the level 4 data they are: the universe comes from a published factor model, so the distance between what is promised and what is obtained can be measured.

The universe and its truth

The data is 38 synthetic total return indices in USD (equities by region, sector and style, bonds, credit, real estate, commodities and money market) from 2006 to 2025, with the real NYSE sessions. It is not market data: it imitates the profile of real indices (volatility, crises, correlations), with the crises of 2008, 2020 and 2022 written into a script. To keep the figures readable, the lesson works with ten of them, one or two per asset class. The excess return of each day is the simple return minus the risk-free rate of the day:

IN [01]python
import numpy as np
import pandas as pd

precios = pd.read_parquet("universo4_precios.parquet")
precios = precios.set_index("fecha")
fac = pd.read_parquet("universo4_factores_diarios.parquet")
rf = fac.set_index("fecha")["rf"]

r = precios.pct_change()
r.iloc[0] = precios.iloc[0] / 100 - 1   # base 100 the day before
exceso = r.sub(rf, axis=0)

activos = ["RV_EEUU", "RV_EUR", "RV_JAP", "RV_EMG", "INM_EEUU",
           "RF_T710", "RF_T20", "RF_CIG", "RF_CHY", "MP_ORO"]
R = exceso[activos]
print(R.shape, R.index[0].date(), R.index[-1].date())
OUT [01]Daily excess returns
(5031, 10) 2006-01-03 2025-12-31

Returns are simple, not logarithmic. The return of a portfolio is the weighted sum of the simple returns of its assets; with logarithms that sum is not exact and the portfolio's Sharpe comes out lower than it is.

The “truth” is in two files: universo4_verdad.csv holds the expected excess return of each asset over the period (mu_verdadero) and universo4_cov_verdad.csv, the true annual covariance per regime and the overall one (total), which is what the full sample estimates:

IN [02]python
verdad = pd.read_csv("universo4_verdad.csv", index_col="activo")
cov_v = pd.read_csv("universo4_cov_verdad.csv")
total = cov_v[cov_v["regimen"] == "total"].set_index("activo")
mu_v = verdad.loc[activos, "mu_verdadero"].to_numpy()
S_v = total.loc[activos, activos].to_numpy()

mu_m = R.mean().to_numpy() * 252
S_m = R.cov().to_numpy() * 252
tabla = pd.DataFrame({
    "mu_verdad": mu_v, "mu_muestra": mu_m,
    "vol_verdad": np.sqrt(np.diag(S_v)),
    "vol_muestra": np.sqrt(np.diag(S_m))}, index=activos)
print((100 * tabla).round(2))
ee = np.sqrt(np.diag(S_m) / 20)
print("standard error of the mean (20 years):", (100 * ee).round(1))
OUT [02]Published truth against the 20-year sample
          mu_verdad  mu_muestra  vol_verdad  vol_muestra
RV_EEUU        9.62        8.79       20.37        20.59
RV_EUR         5.82        3.20       22.63        22.81
RV_JAP         5.45        3.84       21.86        22.00
RV_EMG         3.80        5.46       26.54        27.16
INM_EEUU       9.17        8.98       23.35        23.46
RF_T710        1.60        0.49        5.92         5.90
RF_T20         3.68        1.54       13.69        13.58
RF_CIG         2.76        2.13        5.56         5.59
RF_CHY         4.96        4.68       10.92        10.48
MP_ORO         9.98       10.03       14.59        14.17
standard error of the mean (20 years): [4.6 5.1 4.9 6.1 5.2 1.3 3.  1.3 2.3 3.2]

Volatilities are estimated well: with 5,031 sessions, the sample one is within 0.7 points of the true one for every asset. Means are not. RV_EUR returns 3.20% in the sample against an expected 5.82%; the long bond, 1.54% against 3.68%. It is not a flaw in the data: the standard error of an annual mean is the volatility divided by the square root of the number of years, and with 20 years and a 20% volatility it comes to almost 5 points. It is the law that governs the rest of the lesson: variance is estimated with days; the mean, with years.

The closed-form solution

With no constraints other than the target and the sum of weights, the problem is solved with Lagrange multipliers (Merton, 1972). With the constants

A=1⊤Σ−11,B=1⊤Σ−1μ,C=μ⊤Σ−1μ,D=AC−B2,A = \mathbf{1}^\top \Sigma^{-1} \mathbf{1}, \quad B = \mathbf{1}^\top \Sigma^{-1} \mu, \quad C = \mu^\top \Sigma^{-1} \mu, \quad D = AC - B^2,

the frontier portfolio with return mm and its variance are

w(m)=C−BmD Σ−11+Am−BD Σ−1μ,σ2(m)=Am2−2Bm+CD.w(m) = \frac{C - Bm}{D}\, \Sigma^{-1}\mathbf{1} + \frac{Am - B}{D}\, \Sigma^{-1}\mu, \qquad \sigma^2(m) = \frac{A m^2 - 2Bm + C}{D}.

The variance is a parabola in mm, so the frontier in the (volatility, return) plane is a hyperbola. Its vertex is the minimum variance portfolio, wmv=Σ−11/Aw_{mv} = \Sigma^{-1}\mathbf{1}/A, with volatility 1/A\sqrt{1/A} and return B/AB/A. And since w(m)w(m) is linear in mm, every efficient portfolio is a combination of any two efficient portfolios: this is the two-fund theorem.

IN [03]python
iS = np.linalg.inv(S_v)
uno = np.ones(len(activos))
A = uno @ iS @ uno
B = uno @ iS @ mu_v
C = mu_v @ iS @ mu_v
D = A * C - B**2

w_mv = iS @ uno / A
def w_frontera(m):
    return ((C - B * m) * iS @ uno + (A * m - B) * iS @ mu_v) / D

w5 = w_frontera(0.05)
cerrada = pd.DataFrame({"min_var": w_mv, "frontera_5": w5},
                       index=activos)
print((100 * cerrada).round(1))
print(f"min. variance: vol {np.sqrt(1 / A):.2%}, mu {B / A:.2%}")
print(f"5% frontier: vol {np.sqrt(w5 @ S_v @ w5):.2%}, "
      f"formula {np.sqrt((A * .05**2 - 2 * B * .05 + C) / D):.2%}")
OUT [03]Minimum variance and frontier with the closed form
          min_var  frontera_5
RV_EEUU       2.7        41.9
RV_EUR        0.2        -7.3
RV_JAP        0.4        -2.0
RV_EMG       -0.6       -18.2
INM_EEUU     -1.8        -3.9
RF_T710     152.9        67.6
RF_T20      -67.1       -35.0
RF_CIG       14.0        46.2
RF_CHY       -1.7        -8.2
MP_ORO        1.0        19.0
min. variance: vol 1.81%, mu 0.48%
5% frontier: vol 4.33%, formula 4.33%

The minimum variance portfolio has a volatility of 1.81%, less than the 7-10 year bond on its own (5.9%). It gets there with 153% in that bond and −67% in the 20+ year bond: both load on the same rate factor and, in the right proportion, they cancel out. It is correct on paper and it is a leveraged portfolio that depends on that correlation being exactly the estimated one. The 5% efficient portfolio also sells short: emerging markets (−18%), the long bond (−35%) and Europe.

The same with cvxpy

The closed form stops existing as soon as an inequality appears (long only, a bound). That is what convex optimization is for. cvxpy lets you write the problem almost as on the whiteboard and picks a suitable solver; here CLARABEL is fixed, an interior-point solver for conic problems that ships with cvxpy. Before adding anything, it is worth checking that it reproduces the formula:

IN [04]python
import cvxpy as cp

w = cp.Variable(len(activos))
objetivo = cp.Parameter()
riesgo = cp.quad_form(w, cp.psd_wrap(S_v))
prob = cp.Problem(cp.Minimize(riesgo),
                  [cp.sum(w) == 1, mu_v @ w == objetivo])
objetivo.value = 0.05
prob.solve(solver="CLARABEL")
print(prob.status)
print("max. difference from the formula < 1e-6:",
      np.abs(w.value - w5).max() < 1e-6)
OUT [04]The same problem in cvxpy
optimal
max. difference from the formula < 1e-6: True

cp.psd_wrap tells cvxpy that the matrix is positive semidefinite, instead of checking it with a decomposition that, with nearly singular matrices, can fail through rounding. cp.Parameter makes it possible to change the target and solve again without rebuilding the problem, which is what is needed to sweep the frontier.

Real constraints

A real mandate never lets you short 35% of the long bond. The usual constraints are written in one line each:

  • Long only: w≥0w \ge 0. The rule for almost all mutual funds and pension plans.
  • Per-asset bounds: wi≤30%w_i \le 30\%, so that no position dominates.
  • Groups: equities (stocks and real estate) between 20% and 60%, for example in a balanced profile.
  • Gross leverage: ∑i∣wi∣≤1.6\sum_i |w_i| \le 1.6, a 130/30 portfolio (130% long, 30% short). In cvxpy, cp.norm1(w) <= 1.6.
IN [05]python
rv = [0, 1, 2, 3, 4]   # stocks and real estate
reglas = {
    "unconstrained": [],
    "130/30": [cp.norm1(w) <= 1.6],
    "long only": [w >= 0],
    "mandate": [w >= 0, w <= 0.30, cp.sum(w[rv]) >= 0.2,
                cp.sum(w[rv]) <= 0.6],
}
problemas = {
    k: cp.Problem(cp.Minimize(riesgo),
                  [cp.sum(w) == 1, mu_v @ w >= objetivo] + extra)
    for k, extra in reglas.items()}
filas = {}
for k, pr in problemas.items():
    pr.solve(solver="CLARABEL")
    filas[k] = list(w.value) + [np.sqrt(pr.value)]
pesos = pd.DataFrame(filas, index=activos + ["vol"])
print((100 * pesos).round(1) + 0)   # + 0 removes the -0.0
OUT [05]Weights for a 5% excess return with each set of rules
          unconstrained  130/30  long only  mandate
RV_EEUU            41.9    26.3       11.4     20.0
RV_EUR             -7.3     0.0        0.0      0.0
RV_JAP             -2.0     0.0        0.0      0.0
RV_EMG            -18.2   -15.3        0.0      0.0
INM_EEUU           -3.9    -1.1        0.0      0.0
RF_T710            67.6    63.0       25.9     30.0
RF_T20            -35.0   -13.5        0.0      3.0
RF_CIG             46.2    13.8       35.2     30.0
RF_CHY             -8.2     8.3        4.6      0.8
MP_ORO             19.0    18.5       23.0     16.2
vol                 4.3     4.7        5.8      6.1

The target return becomes an inequality (μ⊤w≥m\mu^\top w \ge m): below the return of the minimum variance portfolio of each set of rules, the optimum is that portfolio, and the frontier is cut at its vertex. The 130/30 portfolio keeps part of the short positions (emerging markets, long bond) and its volatility stays at 4.7%. With long only they all disappear and the portfolio concentrates in five assets, four of them above 10%: the 7-10 year bond, investment grade credit, gold and US stocks. Volatility rises to 5.8%. The mandate forces a 20% equity allocation and caps the bond and credit at 30%: 6.1%. The full frontier shows where each rule bites:

IN [06]python
objetivos = np.round(np.arange(0.03, 0.0951, 0.01), 3)
fr = {}
for k, pr in problemas.items():
    vols = []
    for m in objetivos:
        objetivo.value = m
        pr.solve(solver="CLARABEL")
        ok = pr.status == "optimal"
        vols.append(np.sqrt(pr.value) if ok else np.nan)
    fr[k] = vols
fr = pd.DataFrame(fr, index=objetivos)
fr.index.name = "objetivo"
print((100 * fr).round(2))
OUT [06]The frontier with each set of rules
          unconstrained  130/30  long only  mandate
objetivo
0.03               2.84    3.48       4.83     6.06
0.04               3.56    4.05       5.11     6.06
0.05               4.33    4.72       5.83     6.11
0.06               5.14    5.46       6.82     6.91
0.07               5.96    6.23       8.02     8.22
0.08               6.80    7.08       9.27    10.31
0.09               7.64    8.12      10.75    13.73
Fig. — True efficient frontier of ten assets under four rule sets: each constraint shifts it to the right and truncates it

Each constraint shifts the frontier to the right: for the same return, more volatility. And it cuts it short: with long only, no target can exceed the return of the best asset (gold, 9.98%); with the mandate, which caps each position at 30%, the maximum falls below that 9.98% and a 9% target requires a volatility of 13.7%. With the true parameters constraints can only worsen the result; the interesting question is what happens when the parameters are estimated.

Constraints fall into three families, according to the solver they need:

ConstraintsLinearConvexIntegerw ≥ 0, boundsGroupsLeverageTracking errorCardinalityMinimum lot
Mermaid source
flowchart LR
  P[Constraints] --> L[Linear]
  P --> C[Convex]
  P --> M[Integer]
  L --> L1["w ≥ 0, bounds"]
  L --> L2[Groups]
  C --> C1[Leverage]
  C --> C2[Tracking error]
  M --> M1[Cardinality]
  M --> M2[Minimum lot]

Linear and convex constraints (leverage is a 1-norm; tracking error against an index, (w−wb)⊤Σ(w−wb)≤TE2(w - w_b)^\top \Sigma (w - w_b) \le TE^2, a quadratic constraint) are handled by any conic solver. Integer ones, such as “at most five assets” or “lots of 100 shares”, turn the problem into a mixed-integer one:

IN [07]python
import warnings
warnings.filterwarnings("ignore", category=UserWarning,
                        message="Your problem is mixed-integer")
z = cp.Variable(len(activos), boolean=True)
cardinal = cp.Problem(cp.Minimize(riesgo), [
    cp.sum(w) == 1, mu_v @ w >= 0.05, w >= 0, w <= z,
    cp.sum(z) <= 3])
try:
    cardinal.solve()
except cp.error.SolverError as e:
    print(str(e).split(".")[0])
OUT [07]Cardinality: a mixed-integer solver is needed
You need a mixed-integer solver for this model

The warnings filter silences, by its message, the UserWarning with which cvxpy suggests installing SCIP; without it, the warning would appear in the output with the file path. Among the solvers of this environment (CLARABEL, OSQP, SCS, HiGHS and SciPy) there is none that solves a mixed-integer quadratic problem: HiGHS solves integer linear programs and continuous quadratic ones, but not both at once. The options are to install one that does (SCIP is free; Gurobi, MOSEK and CPLEX, commercial) or a reasonable heuristic: solve the continuous problem, keep the assets with the largest weight and optimize again with only those. Transaction costs and turnover, which also change the shape of the problem, are the topic of lesson 4.1.5.

Maximum Sharpe

If you can invest in the risk-free asset, the best risky portfolio is the one that maximizes the Sharpe, the tangency portfolio. Without constraints its direction is Σ−1μ\Sigma^{-1}\mu; the size is set by the risk you want to take. With constraints, the ratio μ⊤w/w⊤Σw\mu^\top w / \sqrt{w^\top \Sigma w} is not concave, but it becomes a quadratic program with a change of variable: minimize y⊤Σyy^\top \Sigma y with μ⊤y=1\mu^\top y = 1 and the homogeneous constraints on yy (for example y≥0y \ge 0), and then w=y/1⊤yw = y / \mathbf{1}^\top y.

IN [08]python
def sharpe(w, mu, S):
    return w @ mu / np.sqrt(w @ S @ w)

def max_sharpe(mu, S):
    y = cp.Variable(len(mu))
    pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
                    [mu @ y == 1, y >= 0])
    pr.solve(solver="CLARABEL")
    return y.value / y.value.sum()

w_tan = iS @ mu_v
w_tan = w_tan * 0.10 / np.sqrt(w_tan @ S_v @ w_tan)  # 10% vol.
w_lo = max_sharpe(mu_v, S_v)
tang = pd.DataFrame({"sin_limites_vol10": w_tan, "solo_largos": w_lo},
                    index=activos)
print((100 * tang).round(1))
print("Sharpe unconstrained:", sharpe(w_tan, mu_v, S_v).round(3))
print("Sharpe long only:", sharpe(w_lo, mu_v, S_v).round(3))
print("sum of unconstrained weights:", w_tan.sum().round(2))
OUT [08]Tangency portfolio unconstrained and long only
          sin_limites_vol10  solo_largos
RV_EEUU               100.3         14.6
RV_EUR                -18.4          0.0
RV_JAP                 -5.4          0.0
RV_EMG                -44.3          0.0
INM_EEUU               -7.5          0.0
RF_T710               -20.4          0.0
RF_T20                 -4.2          0.0
RF_CIG                 97.0         52.9
RF_CHY                -18.3          0.0
MP_ORO                 45.9         32.5
Sharpe unconstrained: 1.178
Sharpe long only: 0.879
sum of unconstrained weights: 1.25

With the true parameters, the unconstrained portfolio reaches a Sharpe of 1.18, and the long-only one, 0.88. The difference comes from long-short positions: buying investment grade credit (97%) and selling Treasury bonds, emerging markets and Europe. It exploits differences in expected return of one or two points between highly correlated assets. With the truth in front of you, those differences exist. With 20 years of data, the standard error of each mean is 1.3 to 6 points: the unconstrained portfolio bets precisely on what is measured worst.

The sum of the unconstrained weights for a 10% volatility is not 1: with a risk-free asset, the difference up to 1 goes to cash or is borrowed at the risk-free rate (here the portfolio sums to 1.25: 25% financed). Normalizing the tangency portfolio so that it sums to 1, w=Σ−1μ/(1⊤Σ−1μ)w = \Sigma^{-1}\mu / (\mathbf{1}^\top \Sigma^{-1}\mu), is dangerous with estimated means: if the denominator approaches zero, the weights blow up, and if it changes sign the portfolio flips.

Estimation error

Now the experiment that matters. A manager who optimizes on a date uses the data available, for example the last five years. For each five-year window, the long-only maximum Sharpe portfolio is computed with the window's mean and covariance, and its Sharpe is measured twice: the one it promises (with the window's estimates) and the one it really has (with the true μ\mu and Σ\Sigma):

IN [09]python
filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
               ("2016", "2020"), ("2021", "2025")]:
    Rv = R.loc[a0:a1]
    mu_e = Rv.mean().to_numpy() * 252
    S_e = Rv.cov().to_numpy() * 252
    w_e = max_sharpe(mu_e, S_e)
    filas.append({"ventana": f"{a0}-{a1}",
                  "sharpe_estimado": sharpe(w_e, mu_e, S_e),
                  "sharpe_real": sharpe(w_e, mu_v, S_v),
                  "activos": int((w_e > 0.01).sum()),
                  "mayor_peso": activos[int(w_e.argmax())]})
ventanas = pd.DataFrame(filas)
print(ventanas.round(2))
print("true maximum:", sharpe(w_lo, mu_v, S_v).round(2))
OUT [09]What the optimization promises and what it delivers, by window
     ventana  sharpe_estimado  sharpe_real  activos mayor_peso
0  2006-2010             1.75         0.76        4     RF_CIG
1  2011-2015             0.49         0.53        3     RF_T20
2  2016-2020             1.16         0.71        5     MP_ORO
3  2021-2025             1.11         0.80        3     MP_ORO
true maximum: 0.88
Fig. — Long-only maximum-Sharpe portfolio estimated in each five-year window: promised Sharpe (grey) and actual Sharpe with the true parameters (amber)

The 2006-2010 window promises a Sharpe of 1.75 and delivers 0.76. No window reaches the true maximum of 0.88: it is impossible by construction, because that maximum belongs to the optimal weights under the truth, and any other set of weights falls below it. What can be avoided is believing the promised number. Three of the four windows promise more than they deliver, and each one picks a different portfolio, of three to five assets, with the largest weight in credit, in the long bond or in gold depending on the window. The only one that promises less (2011-2015) does so because everything returned little in that window.

The figure at the top shows it over the whole frontier, with the 2021-2025 window: the estimated frontier (what the manager sees on screen) lies above the true one over almost its whole range (they only cross below 6% volatility), and the realized one (the same weights evaluated with the truth) lies below. The estimated frontier is optimistic for two reasons that add up. The first, the error in the mean: in that window gold returned 13.7% and the long bond −11.6% a year in excess, far from their expected 10.0% and 3.7%. The second, selection: the optimizer picks exactly the assets whose error was favorable. That is why Michaud (1989) called mean-variance optimization an “error maximizer”.

IN [10]python
Rv = R.loc["2021":"2025"]
mu_e = Rv.mean().to_numpy() * 252
S_e = Rv.cov().to_numpy() * 252
est = cp.Problem(cp.Minimize(cp.quad_form(w, cp.psd_wrap(S_e))),
                 [cp.sum(w) == 1, mu_e @ w >= objetivo, w >= 0])
ver = problemas["long only"]
filas = []
print("gold and long bond in the window:",
      (100 * mu_e[[9, 6]]).round(1))
for m in [0.04, 0.07, 0.09]:
    objetivo.value = m
    est.solve(solver="CLARABEL")
    w_e = w.value.copy()
    ver.solve(solver="CLARABEL")
    filas.append({"objetivo": m,
                  "vol_estimada": np.sqrt(w_e @ S_e @ w_e),
                  "mu_real": w_e @ mu_v,
                  "vol_real": np.sqrt(w_e @ S_v @ w_e),
                  "vol_verdadera_al_objetivo": np.sqrt(ver.value)})
print((100 * pd.DataFrame(filas)).round(2))
OUT [10]The 2021-2025 frontier: estimated, realized and true
gold and long bond in the window: [ 13.7 -11.6]
   objetivo  vol_estimada  mu_real  vol_real  vol_verdadera_al_objetivo
0       4.0          5.89     4.97      6.60                       5.11
1       7.0          7.12     6.11      8.33                       8.02
2       9.0          8.20     7.21      9.29                      10.75

For a 7% target, the window says a volatility of 7.1% is enough. Those weights really return 6.1% with 8.3% volatility; with the truth, 7% is achieved with 8.0%. The manager loses on both sides: less return and more risk than expected.

Monte Carlo from the truth

The real windows mix two things: estimation error and the regime changes of the script (the 2008 crisis falls entirely in the first window). To isolate estimation error, samples of independent normal returns are simulated with exactly the true μ\mu and Σ\Sigma, each sample is optimized and the real Sharpe of each portfolio is measured. Two hundred samples of five and of twenty years:

IN [11]python
rng = np.random.default_rng(41)
L = np.linalg.cholesky(S_v / 252)
for anos in (5, 20):
    res = []
    for _ in range(200):
        X = mu_v / 252 + rng.standard_normal((252 * anos, 10)) @ L.T
        m_e = X.mean(axis=0) * 252
        if (m_e <= 0).all():
            continue
        S_e = np.cov(X, rowvar=False) * 252
        w_e = max_sharpe(m_e, S_e)
        res.append((sharpe(w_e, m_e, S_e), sharpe(w_e, mu_v, S_v)))
    res = np.array(res)
    p5, p95 = np.percentile(res[:, 1], [5, 95])
    print(f"{anos} years: promised {np.median(res[:, 0]):.2f}, "
          f"real {np.median(res[:, 1]):.2f} ({p5:.2f} to {p95:.2f}), "
          f"overpromises {np.mean(res[:, 0] > res[:, 1]):.0%}")
OUT [11]Promised and real Sharpe with simulated samples
5 years: promised 1.17, real 0.73 (0.49 to 0.85), overpromises 85%
20 years: promised 0.98, real 0.82 (0.73 to 0.87), overpromises 78%

With five years, the median promised Sharpe is 1.17 and the median real one, 0.73; in 85% of the samples more is promised than obtained, and one in twenty portfolios falls below 0.49. With twenty years, the gap narrows (0.98 against 0.82) but does not disappear. And that is with ideal data: normal, independent and without regime changes. In real markets, with fat tails and correlations that change in crises, the error is larger.

What to do about the error

Mean-variance is not abandoned; it is protected. The usual defenses, and where they appear in the course:

DefenseWhat it tacklesLesson
Constraints (long only, bounds, groups)Long-short bets on small differencesThis one
Better estimators of Σ (shrinkage, factors, denoising)Covariance error4.1.2
Equilibrium and views (Black-Litterman)Mean error4.1.3
Portfolios that do not use μ (minimum variance, risk parity, HRP)They ignore the most fragile part4.1.4
Penalizing turnover and costsRebalances that chase noise4.1.5

Jagannathan and Ma (2003) showed that imposing long only on a minimum variance portfolio is equivalent to shrinking the covariances of the assets the optimizer would want to sell: a “wrong” constraint from the point of view of theory that acts as a better estimator. That is why a mandate with reasonable bounds is not only a commercial requirement; it is also a form of regularization.

Typical mistakes when optimizing:

  • Using log returns for portfolio means and covariances, or annualizing the log mean as if it were simple.
  • Mixing total and excess returns: the Sharpe and the tangency portfolio are computed in excess of the risk-free rate.
  • Normalizing the tangency portfolio with 1⊤Σ−1μ\mathbf{1}^\top\Sigma^{-1}\mu when that number is close to zero.
  • Believing the estimated frontier: it is the frontier of the estimates, not that of the market.
  • Not fixing the solver or not checking prob.status: an infeasible problem returns None in the weights and infeasible in the status, not an exception.

Exercises

  1. Repeat the unconstrained minimum variance portfolio with the 38 assets of the universe and the overall true covariance. What weight does the money market (MONET) get, and why?
Solution
IN [12]python
todos = list(total.index)
S38 = total.loc[todos, todos].to_numpy()
iS38 = np.linalg.inv(S38)
u38 = np.ones(len(todos))
w38 = pd.Series(iS38 @ u38 / (u38 @ iS38 @ u38), index=todos)
print(f"vol: {np.sqrt(1 / (u38 @ iS38 @ u38)):.3%}")
print((100 * w38).round(1).sort_values().iloc[[0, 1, -2, -1]])
OUT [12]Minimum variance with the 38 assets
vol: 0.152%
RF_T710    -2.2
RF_T20     -1.2
RF_T13      8.8
MONET      92.5
dtype: float64

The money market has a volatility of 0.2% and almost no correlation with anything else: the minimum variance portfolio takes almost all of it, and the volatility falls below its own. It is the right answer to the wrong question. If there is an almost riskless asset, minimum variance over risky assets reduces to it; that is why portfolios are optimized in excess of the risk-free rate and without the money market inside the risky universe.

  1. Repeat the window experiment with a 30% bound per asset in addition to long only. Does the gap between the promised and the real Sharpe narrow?
Solution
IN [13]python
def max_sharpe_cota(mu, S, cota=0.30):
    y = cp.Variable(len(mu))
    k = cp.Variable(nonneg=True)   # y = k·w, so y <= cota·k
    pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
                    [mu @ y == 1, y >= 0, y <= cota * k,
                     cp.sum(y) == k])
    pr.solve(solver="CLARABEL")
    return y.value / y.value.sum()

filas = []
for a0, a1 in [("2006", "2010"), ("2011", "2015"),
               ("2016", "2020"), ("2021", "2025")]:
    Rv = R.loc[a0:a1]
    mu_e = Rv.mean().to_numpy() * 252
    S_e = Rv.cov().to_numpy() * 252
    w_c = max_sharpe_cota(mu_e, S_e)
    filas.append({"ventana": f"{a0}-{a1}",
                  "estimado": sharpe(w_c, mu_e, S_e),
                  "real": sharpe(w_c, mu_v, S_v)})
print(pd.DataFrame(filas).round(2))
OUT [13]Windows with a 30% bound
     ventana  estimado  real
0  2006-2010      1.72  0.75
1  2011-2015      0.48  0.54
2  2016-2020      1.16  0.70
3  2021-2025      1.02  0.67

The bound is a non-homogeneous constraint (wi≤0.3w_i \le 0.3), so in the change of variable it has to be multiplied by the scale factor k=1⊤yk = \mathbf{1}^\top y. The bound barely changes the first three windows (the portfolios were already spread out) and in 2021-2025 it lowers the real Sharpe from 0.80 to 0.67: it limits gold, which under the truth is precisely the most attractive asset. A constraint does not recover the truth: it reduces how much is bet on each window's error, and that sometimes has a cost. Its value lies in the worst case, not in the average of four windows.

  1. With 20 years of data, the standard error of the mean of RV_EEUU is about 4.6 points. How many years would it take to bring it down to 1 point? And why does using daily data instead of annual data not shorten that period?
Solution

With AA years of data, the standard error of the annual mean is σ/A\sigma/\sqrt{A}: with σ≈20.6%\sigma \approx 20.6\%, getting down to 1 point requires (20.6)2≈424(20.6)^2 \approx 424 years. Using daily data does not help: the annual mean estimated with NN days is 252⋅r‾252 \cdot \overline{r}, with standard error 252⋅σd/N=σ/N/252252 \cdot \sigma_d/\sqrt{N} = \sigma/\sqrt{N/252}, which only depends on the number of years the sample covers, not on how many observations it contains. Frequency does improve the estimation of the variance, which depends on the number of observations. That is why portfolios that do not need μ\mu (lesson 4.1.4) and methods that take it from equilibrium (lesson 4.1.3) exist.

Each answer is folded under its question

Self-assessment

  1. The maximum Sharpe portfolio estimated with 2006-2010 data promises a Sharpe of 1.75. What Sharpe does it have with the true parameters?

    • A1.75: the optimization is exact
    • B0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance
    • CMore than 1.75, because the sample is pessimistic
    • DIt cannot be known
    Show answer

    Answer B. 0.76, below even the true maximum (0.88): the optimizer picked the assets that did best in that window, partly by chance

    The optimizer maximizes over estimates with error and picks precisely the assets whose error was favorable. The Sharpe estimated on the optimization's own sample is biased upward; the real Sharpe of those weights falls below the true maximum. In Monte Carlo with five years, the estimate exceeds the real one in 85% of cases.

  2. Why does the unconstrained minimum variance portfolio of the ten assets hold 153% in the 7-10 year bond and −67% in the 20+ year bond?

    • ABecause of an optimizer error
    • BBecause the long bond has a negative expected return
    • CBecause the closed form forces the use of every asset
    • DBecause both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio
    Show answer

    Answer D. Because both bonds load on the same rate factor: the long-short combination cancels much of that risk and leaves a very low volatility, leveraged portfolio

    Minimum variance only looks at the covariance. Two highly correlated assets with different volatilities make it possible to build an almost riskless position by buying one and selling the other in the right proportion. It is correct on paper and depends on that correlation holding.

  3. What does a long-only constraint do to estimation error?

    • ANothing: it only worsens the frontier
    • BIt makes it worse, because it leaves fewer degrees of freedom
    • CIt bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance
    • DIt removes it completely
    Show answer

    Answer C. It bounds it: it stops the optimizer from building large long-short positions on noisily estimated differences; Jagannathan and Ma (2003) showed that it is equivalent to shrinking the covariance

    With the true parameters, the constraint can only worsen the frontier (maximum Sharpe from 1.18 to 0.88). With estimated parameters, it avoids the long-short bets most sensitive to error and usually improves the out-of-sample result.

  4. cvxpy rejects a problem with variance as the objective and a five-asset limit with binary variables if only CLARABEL, OSQP, SCS, HIGHS and SCIPY are installed. Why?

    • ABecause it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once
    • BBecause cardinality cannot be written in cvxpy
    • CBecause variance is not convex
    • DBecause data is missing
    Show answer

    Answer A. Because it is a mixed-integer quadratic problem (MIQP) and none of those solvers handles it: HiGHS does integer linear programming and continuous quadratic programming, but not both at once

    The problem is valid but not convex in the strict sense because of the integer variables. It needs an MIQP solver (SCIP, Gurobi, MOSEK, CPLEX) or a heuristic: solve the continuous version, keep the largest weights and optimize again.

  5. For a target of 5% annual excess return, the unconstrained portfolio has a volatility of 4.3% and the long-only one, 5.8%. Which one should you choose?

    • AThe unconstrained one, which dominates on paper
    • BIt depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely
    • CThe long-only one, because it always has less risk
    • DNeither: use the equal-weighted one
    Show answer

    Answer B. It depends on how much you trust the parameters: with the true ones the unconstrained one dominates; with estimated ones, its advantage is built with short positions that depend on return differences that cannot be measured precisely

    The unconstrained frontier is the best one if μ and Σ are exact. In practice you work with estimates, and the standard error of the mean of an equity index over 20 years is around 5 points: much more than the differences the long-short portfolio exploits.

Key points

  • Mean-variance minimizes w′Σw with a target return and weights summing to 1. Without further constraints it has a closed-form solution: the frontier is a hyperbola and every efficient portfolio combines the minimum variance and the tangency portfolios.
  • cvxpy solves the same problem (weight difference below one millionth) and accepts linear and convex constraints: long only, bounds, groups, leverage, tracking error. Cardinality needs a mixed-integer solver.
  • Each constraint shifts the frontier to the right with the true parameters: maximum Sharpe 1.18 unconstrained, 0.88 long only.
  • With estimated parameters, the sample frontier is optimistic: the maximum Sharpe estimated in 2006-2010 promises 1.75 and delivers 0.76; in Monte Carlo with five years, a median of 1.17 promised against 0.73 real.
  • Expected returns are the most fragile part: with 20 years, the standard error of the mean of an equity index is around 5 points. Lessons 4.1.2 and 4.1.3 tackle the covariance and the mean.

Resources