Black-Litterman: from market equilibrium to views.

The problem it solves

Lesson 4.1.1 left two results. The first, that the mean is the most fragile input of Markowitz: with 20 years, its standard error is around 5 points for an equity index. The second, that the optimizer amplifies that error: it looks for the estimated return differences and bets on them. It is enough to see what it does with the 20-year sample mean of the 38 assets of the universe, without constraints:

import numpy as np
import pandas as pd

precios = pd.read_parquet("universo4_precios.parquet")
precios = precios.set_index("fecha")
fac = pd.read_parquet("universo4_factores_diarios.parquet")
r = precios.pct_change()
r.iloc[0] = precios.iloc[0] / 100 - 1
exceso = r.sub(fac.set_index("fecha")["rf"], axis=0)
activos = list(exceso.columns)

info = pd.read_csv("universo4_activos.csv", index_col="activo")
verdad = pd.read_csv("universo4_verdad.csv", index_col="activo")
cov_v = pd.read_csv("universo4_cov_verdad.csv")
S = cov_v[cov_v["regimen"] == "total"].set_index("activo")
S = S.loc[activos, activos].to_numpy()
w_mkt = info.loc[activos, "peso_mercado"].to_numpy()
mu_v = verdad.loc[activos, "mu_verdadero"].to_numpy()
mu_m = exceso.mean().to_numpy() * 252

delta = 2.5
w_m = np.linalg.solve(delta * S, mu_m)
print(f"gross leverage: {np.abs(w_m).sum():.0f} times")
print((100 * pd.Series(w_m, index=activos)).round(0)
      .sort_values().iloc[[0, 1, -2, -1]])

gross leverage: 138 times
MONET     -2077.0
RF_T710   -1991.0
RF_CIG     1214.0
RF_T37     2395.0
dtype: float64

To isolate the problem of the mean, the lesson uses the overall true covariance of the universe (lesson 4.1.2 dealt with the covariance; exercise 1 repeats the calculation with the sample one). Even with the exact covariance, the optimal portfolio for a risk aversion of 2.5 has gross leverage of 138 times: it sells 2,077% of the money market and 1,991% of the 7-10 year bond to buy 2,395% of the 3-7 year bond, almost identical assets whose sample means differ by tenths. No manager would do it. Black and Litterman (1992), at Goldman Sachs, proposed another starting point.

Equilibrium: reverse optimization

The first-order condition of a mean-variance investor with risk aversion $\delta$ is $w = (\delta \Sigma)^{-1} \mu$. If the weights are known and the mean is not, you solve it the other way round:

$\pi$ are the expected excess returns that would make the market portfolio optimal. If every investor were mean-variance and had the same information, the market portfolio would be the tangency portfolio (the CAPM) and $\pi$ would be the truth. The risk aversion $\delta$ can be calibrated with the market Sharpe: $\delta = (\mu_m / \sigma_m) / \sigma_m$.

flowchart TB
  W[Market weights] --> PI["π = δΣw"]
  S[Covariance Σ] --> PI
  V["Views P, Q"] --> BL[Black-Litterman]
  O["Confidence Ω"] --> BL
  PI --> BL
  BL --> M[Posterior mean]
  M --> OP[Optimizer]

The universe publishes the weights of the market portfolio (peso_mercado in universo4_activos.csv: the 18 non-overlapping blocks, by capitalization as of 2025) and its own $\pi$ with $\delta = 2.5$:

pi = delta * S @ w_mkt
pub = verdad.loc[activos, "pi_equilibrio"].to_numpy()
print("same as the published one:", np.allclose(pi, pub, atol=1e-4))

mu_mkt, vol_mkt = w_mkt @ mu_v, np.sqrt(w_mkt @ S @ w_mkt)
print(f"market: excess {mu_mkt:.2%}, vol {vol_mkt:.2%}, "
      f"Sharpe {mu_mkt / vol_mkt:.2f}")
print(f"implied δ with the truth: {mu_mkt / vol_mkt**2:.2f}")

r_mkt = exceso.to_numpy() @ w_mkt
print(f"implied δ with the sample: "
      f"{r_mkt.mean() / r_mkt.var():.2f}")
w_tan = np.linalg.solve(S, mu_v)
print("Sharpe of the true tangency portfolio:",
      (w_tan @ mu_v / np.sqrt(w_tan @ S @ w_tan)).round(2))

same as the published one: True
market: excess 5.77%, vol 12.95%, Sharpe 0.45
implied δ with the truth: 3.44
implied δ with the sample: 2.94
Sharpe of the true tangency portfolio: 1.35

The $\delta$ implied by the data (2.94 with the sample, 3.44 with the truth) is somewhat higher than the 2.5 with which $\pi$ is published in the data. Changing $\delta$ scales the whole of $\pi$ equally; what matters is its shape. And the shape has two features of this universe that are worth seeing before going on:

ventana = exceso.loc["2021":"2025"].mean().to_numpy() * 252
def ecm(x):
    return np.sqrt(np.mean((x - mu_v) ** 2))

print(f"root mean squared error against the truth: "
      f"equilibrium {ecm(pi):.2%}, 20-year sample {ecm(mu_m):.2%}, "
      f"2021-2025 sample {ecm(ventana):.2%}")
sel = ["RV_EEUU", "RV_EMG", "RF_T20", "RF_CIG", "MP_ORO", "MP_ENE"]
tabla = pd.DataFrame({"equilibrio": pi, "verdad": mu_v,
                      "muestra_20": mu_m, "muestra_5": ventana},
                     index=activos).loc[sel]
print((100 * tabla).round(2))

root mean squared error against the truth: equilibrium 3.45%, 20-year sample 1.71%, 2021-2025 sample 6.32%
         equilibrio  verdad  muestra_20  muestra_5
RV_EEUU        6.46    9.62        8.79      10.51
RV_EMG         7.96    3.80        5.46       7.22
RF_T20        -0.52    3.68        1.54     -11.60
RF_CIG         0.41    2.76        2.13      -2.06
MP_ORO         0.77    9.98       10.03      13.72
MP_ENE         3.32   -4.62       -5.46      10.38

The equilibrium gives almost zero or negative premiums to Treasury bonds (−0.52% to the 20+ year bond, against a true 3.68%) because in the calm, the dominant regime, bonds are negatively correlated with stocks: in a market portfolio dominated by stocks, they hedge, and what hedges “deserves” a low premium. And it gives positive premiums to commodities (3.32% to energy), which lose in the script of these 20 years (−4.62%). Gold is the extreme case: equilibrium 0.77%, truth 9.98%. In this universe the market portfolio is not efficient: its true Sharpe is 0.45, and that of the true unconstrained tangency portfolio, 1.35.

In root mean squared error, the equilibrium (3.45 points) does worse than the 20-year sample mean (1.71 points) and better than the five-year one (6.32). The 20-year sample is unusually good: the generator seeds were chosen so that the realized means would end up close to the true ones. The normal case, and what a manager looking at five years has, looks more like the last column. The advantage of the equilibrium is not being more accurate than a good sample, but not producing absurd portfolios: a $\pi$ consistent with the market portfolio leads to the market portfolio.

Views: P, Q and Ω

A view is a statement about the expected return of a portfolio. Absolute views are about one asset (“emerging markets return 5% a year in excess”); relative ones, about a difference (“momentum beats value by 3%”). With $k$ views they are written as

with $P$ a $k \times n$ matrix (one row per view, with the weights of its portfolio), $Q$ the returns they claim and $\Omega$ their uncertainty, usually diagonal. The file universo4_visiones.csv contains six, with the answer known: verdad is $P \mu$ with the true mean and correcta flags the ones that are within one point of it.

vis = pd.read_csv("universo4_visiones.csv")
P = np.zeros((len(vis), len(activos)))
for i, fila in enumerate(vis["activos"]):
    for parte in fila.split(";"):
        activo, peso = parte.split(":")
        P[i, activos.index(activo)] = float(peso)
Q = vis["q"].to_numpy()
conf = vis["confianza"].to_numpy()
print(vis[["vision", "q", "confianza", "pi_equilibrio", "verdad",
           "correcta"]].round(4).to_string(index=False))
print("P·π and P·μ match the file:",
      np.allclose(P @ pi, vis["pi_equilibrio"], atol=1e-4),
      np.allclose(P @ mu_v, vis["verdad"], atol=1e-4))

vision     q  confianza  pi_equilibrio  verdad  correcta
    V1  0.05       0.50         0.0796  0.0380     False
    V2  0.02       0.40        -0.0055 -0.0330     False
    V3  0.03       0.60         0.0027  0.0328      True
    V4  0.02       0.30         0.0206  0.0220      True
    V5 -0.01       0.50        -0.0200 -0.0173      True
    V6  0.08       0.25         0.0077  0.0998     False
P·π and P·μ match the file: True True

Views V3, V4 and V5 are correct; V1, V2 and V6 are not. V6 comes “from an ML model” and claims that gold returns 8% a year in excess: it is incorrect by the one-point criterion (the truth is 9.98%), but it is much closer to the truth than the equilibrium. That difference is going to matter.

The Black-Litterman posterior

Black-Litterman treats $\pi$ as a prior on the true mean, $\mu \sim N(\pi, \tau\Sigma)$, with a small scalar $\tau$ (the uncertainty of the mean is much smaller than that of returns), and combines it with the views as in a Bayesian regression. The posterior mean is

The last factor is the “surprise” of each view against the equilibrium, $Q - P\pi$. It is spread across all assets according to their covariances with the view portfolios, so a view on emerging markets also moves the mean of the other stock markets and of emerging debt.

$\Omega$ is still missing. Idzorek proposed setting it with a more intuitive confidence $c$ between 0 and 1: with 100% confidence the view holds exactly; with 0%, it is ignored; with $c$, the portfolio weights travel a fraction $c$ of the way between the market portfolio and the full-confidence one. His original method finds each $\omega_k$ numerically; Walters, in his review of the model, showed that without constraints the solution is closed-form:

tau = 0.05
def omega_idzorek(P, conf):
    return np.diag([tau * (1 - c) / c * p @ S @ p
                    for p, c in zip(P, conf)])

def black_litterman(P, Q, Om):
    tS = tau * S
    sorpresa = np.linalg.solve(P @ tS @ P.T + Om, Q - P @ pi)
    return pi + tS @ P.T @ sorpresa

Om = omega_idzorek(P, conf)
mu_bl = black_litterman(P, Q, Om)

from pypfopt.black_litterman import BlackLittermanModel
blm = BlackLittermanModel(
    pd.DataFrame(S, index=activos, columns=activos),
    pi=pd.Series(pi, index=activos), P=P, Q=Q, tau=tau,
    omega="idzorek", view_confidences=conf)
print("same as PyPortfolioOpt:",
      np.allclose(blm.bl_returns().to_numpy(), mu_bl, atol=1e-12))
post = pd.DataFrame({"q": Q, "equilibrio": P @ pi,
                     "posterior": P @ mu_bl, "verdad": P @ mu_v},
                    index=vis["vision"])
print((100 * post).round(2))

same as PyPortfolioOpt: True
          q  equilibrio  posterior  verdad
vision
V1      5.0        7.96       6.21    3.80
V2      2.0       -0.55       0.43   -3.30
V3      3.0        0.27       1.76    3.28
V4      2.0        2.06       1.46    2.20
V5     -1.0       -2.00      -1.36   -1.72
V6      8.0        0.77       2.46    9.98

The figure at the top draws this table. The posterior of each view lies between the equilibrium and what the view claims, closer to the latter the higher the confidence and the lower the variance of its portfolio. V6, with 25% confidence, moves gold from 0.77% to 2.46%: less than the model says and very far from the truth. V4 moves from 2.06% to 1.46%, in the opposite direction to its own $Q$: views interact, because high yield credit is correlated with stocks and other views (V1, V2) move stocks.

That the closed form reproduces Idzorek's original method is checked by finding each $\omega_k$ the way he did: the one that leaves the unconstrained weights a fraction $c_k$ of the way.

from scipy.optimize import minimize_scalar

def pesos(m):
    return np.linalg.solve(delta * S, m)

for k in range(len(Q)):
    p, q = P[[k]], Q[[k]]
    w_100 = pesos(black_litterman(p, q, np.array([[1e-12]])))
    objetivo = w_mkt + conf[k] * (w_100 - w_mkt)
    def distancia(log_om):
        m = black_litterman(p, q, np.array([[np.exp(log_om)]]))
        return np.sum((pesos(m) - objetivo) ** 2)
    res = minimize_scalar(distancia, bounds=(-20, 5),
                          method="bounded",
                          options={"xatol": 1e-10})
    print(vis["vision"][k], f"{np.exp(res.x) / Om[k, k]:.6f}")

V1 1.000000
V2 1.000000
V3 1.000000
V4 1.000000
V5 1.000000
V6 1.000000

The ratio between the numerical $\omega$ and the closed-form one is 1 for all six views. With constraints (long only, for example) there is no longer a closed form and the numerical method is the only correct one.

Does each view help?

The practical question is not whether a view is correct, but whether the portfolio improves with it. For each view on its own, the posterior mean is computed, then the long-only maximum Sharpe portfolio (the same function as in lesson 4.1.1) and its Sharpe with the true mean and covariance:

import cvxpy as cp

def max_sharpe(m):
    y = cp.Variable(len(m))
    pr = cp.Problem(cp.Minimize(cp.quad_form(y, cp.psd_wrap(S))),
                    [m @ y == 1, y >= 0])
    pr.solve(solver="CLARABEL")
    return y.value / y.value.sum()

def sharpe(w):
    return w @ mu_v / np.sqrt(w @ S @ w)

filas = []
for k in range(len(Q)):
    m = black_litterman(P[[k]], Q[[k]], Om[[k]][:, [k]])
    filas.append({
        "vision": vis["vision"][k],
        "correcta": vis["correcta"][k],
        "error_equilibrio": abs(P[k] @ pi - P[k] @ mu_v),
        "error_q": abs(Q[k] - P[k] @ mu_v),
        "error_posterior": abs(P[k] @ m - P[k] @ mu_v),
        "sharpe": sharpe(max_sharpe(m))})
cada = pd.DataFrame(filas).set_index("vision")
print(cada.round(3))
print("Sharpe of the market portfolio:", sharpe(w_mkt).round(3))

        correcta  error_equilibrio  error_q  error_posterior  sharpe
vision
V1         False             0.042    0.012            0.027   0.492
V2         False             0.028    0.053            0.038   0.357
V3          True             0.030    0.003            0.014   0.583
V4          True             0.001    0.002            0.002   0.446
V5          True             0.003    0.007            0.002   0.473
V6         False             0.092    0.020            0.074   0.627
Sharpe of the market portfolio: 0.446

The table separates two ideas that are often confused:

• V1 and V6, “wrong”, improve the portfolio. Their $Q$ miss by more than a point, but the equilibrium was much further away: 4.2 points for emerging markets and 9.2 for gold. The gold view, with only 25% confidence, raises the Sharpe from 0.45 to 0.63.
• V4 and V5, “correct”, change almost nothing. The equilibrium was already 0.1 and 0.3 points from the truth; the view adds no information and the Sharpe stays at 0.45 and 0.47.
• V2 is the only harmful one. It claims that hedged Europe beats the US by 2%, when the truth is −3.30% and the equilibrium said −0.55%: it pulls the mean away from the truth and the Sharpe falls to 0.36, below the market.
• V3, correct and far from the equilibrium, is the textbook case: 3.0 points of error in the equilibrium, 0.3 in the view, and the Sharpe rises to 0.58.

A view is worth as much as it is closer to the truth than the equilibrium. That is why it is not enough to count the hits of an analyst or a model: you have to measure whether they are right more often than the market.

The Black-Litterman portfolio

He and Litterman (1999) showed a property that makes the model easy to explain to a committee: without constraints, the optimal portfolio with the posterior mean is the market portfolio plus a position in the portfolio of each view. Assets that do not appear in any view keep their market weight, even though their mean has changed:

w_bl = pesos(mu_bl)
dif = pd.Series(w_bl - w_mkt, index=activos)
en_vistas = np.abs(P).sum(axis=0) > 0
print("assets with change > 0.01%:", list(dif[dif.abs() > 1e-4].index))
cambian = set(dif[dif.abs() > 1e-4].index)
print("all in some view:",
      cambian <= set(np.array(activos)[en_vistas]))
print((100 * dif[dif.abs() > 1e-4]).round(1))

assets with change > 0.01%: ['RV_EEUU', 'RV_EUR_C', 'RV_EMG', 'ES_VAL', 'ES_MOM', 'ES_MIN', 'RF_CIG', 'RF_CHY', 'MP_ORO']
all in some view: True
RV_EEUU    -96.4
RV_EUR_C    71.8
RV_EMG      -6.9
ES_VAL     -55.7
ES_MOM      55.7
ES_MIN      24.7
RF_CIG     -10.3
RF_CHY      10.3
MP_ORO      34.7
dtype: float64

The unconstrained portfolio holds short positions in the US (its weight drops 96 points, because of V2 and V5), in value (V3) and in investment grade (V4): these are the portfolios of the relative views, which are long-short by definition. A long-only fund applies the optimizer with the posterior mean and its constraints. The final comparison, with the Sharpe evaluated with the truth:

grupos = {"market": None, "correct only": [2, 3, 4],
          "incorrect only": [0, 1, 5], "all six": list(range(6))}
res = {}
for nombre, idx in grupos.items():
    if idx is None:
        res[nombre] = sharpe(w_mkt)
        continue
    m = black_litterman(P[idx], Q[idx], Om[np.ix_(idx, idx)])
    res[nombre] = sharpe(max_sharpe(m))
res["2021-2025 mean"] = sharpe(max_sharpe(ventana))
res["20-year mean"] = sharpe(max_sharpe(mu_m))
print(pd.Series(res).round(3))

market            0.446
correct only      0.595
incorrect only    0.597
all six           0.700
2021-2025 mean    0.668
20-year mean      0.901
dtype: float64

With the six views, the true Sharpe goes from 0.45 (market) to 0.70. The three correct ones alone give 0.60; the three incorrect ones, another 0.60, because of V1 and V6. The mean of the last five years gives 0.67 with a portfolio chosen by the chance of the window (lesson 4.1.1), and the 20-year one, 0.90: with data as good as this generator's, the sample mean with long-only constraints wins. Black-Litterman does not promise the best portfolio; it promises to start from a reasonable one and to depart from it only where something is known.

Views from machine learning models

An ML model (those of level 5 of the course) produces predictions, and a prediction can be turned into a view: $Q$ is the prediction and $\Omega$, the uncertainty of that prediction. The typical mistake is to take the confidence from the fit inside the training sample. The right thing is to measure it out of sample and against the prior: if the model does not predict better than the equilibrium, its view should not move anything.

flowchart TB
  D[Data up to t] --> M[Model]
  M --> Q["Prediction Q"]
  M --> E[Out-of-sample errors]
  E --> O["Ω"]
  Q --> BL[Black-Litterman]
  O --> BL

A simple example with gold: a “model” that predicts next year's excess return with the mean of the previous three, evaluated year by year from 2009 to 2025 (each prediction uses only past data). Its squared error is compared with that of the equilibrium, and what the model gains over the noise of the annual return itself is what justifies $\Omega$:

oro = exceso["MP_ORO"]
anual = (1 + oro).groupby(oro.index.year).prod() - 1
pred = {a: oro.loc[str(a - 3):str(a - 1)].mean() * 252
        for a in range(2009, 2026)}
pred = pd.Series(pred)
real = anual.loc[pred.index]
i_oro = activos.index("MP_ORO")
ecm_mod = np.mean((pred - real) ** 2)
ecm_eq = np.mean((pi[i_oro] - real) ** 2)
print(f"squared error: model {np.sqrt(ecm_mod):.1%}, "
      f"equilibrium {np.sqrt(ecm_eq):.1%}, "
      f"out-of-sample R² {1 - ecm_mod / ecm_eq:.3f}")

ruido = S[i_oro, i_oro]               # variance of one year of gold
omega_ml = max(ecm_mod - ruido, 1e-6)  # model error on the mean
p6 = P[5]
c_ml = 1 / (1 + omega_ml / (tau * p6 @ S @ p6))
print(f"model ω {omega_ml:.4f} against the published {Om[5, 5]:.4f}")
print(f"implied confidence: {c_ml:.1%} (published {conf[5]:.0%})")

squared error: model 20.8%, equilibrium 21.5%, out-of-sample R² 0.068
model ω 0.0218 against the published 0.0032
implied confidence: 4.7% (published 25%)

The model beats the equilibrium by very little (an out-of-sample R² of 0.07, with 17 years, indistinguishable from zero). Its squared error is double the variance of one year of gold: half of the error is noise of the annual return itself, which no model can remove, and the other half, $\omega = 0.0218$, is model error on the mean, almost seven times the published $\omega$. The confidence that justifies is about 5%, not 25%. In this universe, view V6 is close to the truth, but a manager could not know that with this track record: its confidence has to come from the data and not from the expected result.

Four precautions with ML views:

• Truly out of sample: predictions made with data prior to each date, with the purging of lesson 2.5.3 if the labels overlap.
• Against the prior, not against zero: a model that predicts “stocks go up” is right more often than zero, but not more often than the equilibrium.
• Do not count twice: if the model uses the same data that estimates $\Sigma$ or the mean, the view repeats information the prior already has.
• Consistent horizon: $Q$ and $\Omega$ in annual excess terms, like $\pi$; a one-month prediction is annualized, and so is its error.

Typical mistakes

• Mixing total and excess returns between $\pi$ (in excess) and $Q$.
• Badly written relative views: the row of $P$ for “A beats B” is $+1$ in A and $-1$ in B; with several assets per side, the weights of each side sum to 1 and −1.
• Confusing τ with confidence: with the closed-form Idzorek, τ does not change the posterior mean (exercise 3); it does change the posterior covariance.
• Market weights that are not market weights: $\pi$ inherits whatever is put in $w$. With a universe that includes sectors and styles that overlap with the index, only the non-overlapping blocks should be used, as the file does.
• Believing the posterior is the truth: it is a compromise between two sources with error, and in this universe the equilibrium has large errors in bonds and commodities.

Exercises

1. Repeat the equilibrium and the posterior with the 20-year sample covariance instead of the true one. How much do $\pi$ and the posterior of each view change?

S_m = exceso.cov().to_numpy() * 252
pi_m = delta * S_m @ w_mkt
print(f"maximum difference in π: {np.abs(pi_m - pi).max():.2%}")
tS_m = tau * S_m
Om_m = np.diag([tau * (1 - c) / c * p @ S_m @ p
                for p, c in zip(P, conf)])
mu_bl_m = pi_m + tS_m @ P.T @ np.linalg.solve(
    P @ tS_m @ P.T + Om_m, Q - P @ pi_m)
print((100 * pd.DataFrame({"verdadera": P @ mu_bl,
                           "muestral": P @ mu_bl_m},
                          index=vis["vision"])).round(2))

maximum difference in π: 0.33%
        verdadera  muestral
vision
V1           6.21      6.44
V2           0.43      0.43
V3           1.76      1.83
V4           1.46      1.43
V5          -1.36     -1.37
V6           2.46      2.45

2. Remove view V2 from the six. How much does the true Sharpe of the long-only portfolio improve? Could a manager have known in advance that it was the view to remove?

idx = [0, 2, 3, 4, 5]
m = black_litterman(P[idx], Q[idx], Om[np.ix_(idx, idx)])
print("without V2:", sharpe(max_sharpe(m)).round(3),
      "| all six:", res["all six"].round(3))

without V2: 0.798 | all six: 0.7

3. Check that, with the closed-form Idzorek Ω, the posterior mean does not change if τ goes from 0.05 to 0.5. Explain why.

def bl_tau(t):
    Om_t = np.diag([t * (1 - c) / c * p @ S @ p
                    for p, c in zip(P, conf)])
    tS = t * S
    return pi + tS @ P.T @ np.linalg.solve(P @ tS @ P.T + Om_t,
                                            Q - P @ pi)

print("same mean:", np.allclose(bl_tau(0.05), bl_tau(0.5)))

same mean: True