Meta-labeling: one model for the side and another for the size.

Two decisions, two models

A trading system makes two decisions on every signal: which side to take and how much to bet (zero included). Most ML models mix them: a classifier predicts “up” or “down” and trades with a fixed size. Meta-labeling (López de Prado, 2018, section 3.6) separates them:

• The primary model decides the side. It can be a technical rule, an econometric model, a fundamental analyst or an ML model. It is designed to have high recall: to propose almost all the opportunities, even if many of them fail.
• The secondary model, or meta-model, receives each proposal from the primary and estimates the probability that it is right. It does not decide the side: it decides whether to trade and with what size. Its job is to raise the precision.

flowchart TB
  D[Data up to t] --> P[Primary: side]
  P --> TB[Triple barrier]
  TB --> Y[Meta-label]
  D --> F[Features at t]
  F --> S[Secondary: p]
  Y --> S
  S --> Z[Trade and size]

The separation has three advantages. The primary can be simple and interpretable, and ML only acts as a filter, which reduces how much it can overfit. The size comes from a probability, not from a separate rule. And the method works with any primary, including a discretionary one: the secondary learns under what conditions that analyst is right.

The case in this lesson has a known answer. In FNX (level 5 synthetic data), the primary, a momentum triggered by a CUSUM filter (lesson 5.3.5), is right 60.4% of the time in the trend regime and 44.6% in the reversion regime. Overall, 50.7%: it does not win. A good meta-model should get close to what someone who knew the regime would do.

The primary model and the meta-label

The data, the volatility and the two functions come from lessons 5.3.2 and 5.3.5, in compact form:

import numpy as np
import pandas as pd

barras = pd.read_parquet("hf5_barras_1min.parquet")
verdad = pd.read_parquet("hf5_verdad.parquet")
n_ses = len(barras) // 390


def matriz(col):
    """Bar column as a session x minute matrix."""
    return barras[col].to_numpy().reshape(n_ses, 390)


C, V = matriz("cierre"), matriz("volumen")
VC = matriz("volumen_comprador")
tend = (verdad["regimen"] == "tendencia").to_numpy()
tend = tend.reshape(n_ses, 390)        # the truth: only for measuring
lr = np.diff(np.log(C), axis=1)
sig = (pd.Series(lr.ravel()).ewm(span=100).std()
       .to_numpy().reshape(n_ses, 389))
print(C.shape, f"median σ {1e4 * np.nanmedian(sig):.2f} bp")

(247, 390) median σ 4.22 bp

def eventos_cusum(lr, sig, k=4.0):
    """Symmetric CUSUM filter per session (lesson 5.3.5)."""
    filas = []
    for d in range(lr.shape[0]):
        s_pos = s_neg = 0.0
        for m in range(lr.shape[1]):
            h = k * sig[d, m]
            if not np.isfinite(h):
                continue
            s_pos = max(0.0, s_pos + lr[d, m])
            s_neg = min(0.0, s_neg + lr[d, m])
            if s_pos > h or s_neg < -h:
                filas.append((d, m + 1, 1 if s_pos > h else -1,
                              sig[d, m]))
                s_pos = s_neg = 0.0
    return pd.DataFrame(
        filas, columns=["sesion", "minuto", "lado", "sigma"])


def triple_barrera(C, ev, k=1.0, h=30):
    """Barriers ±k·σ·√h and vertical at h minutes (lesson 5.3.2)."""
    fin, ret = [], []
    for d, m, s in ev[["sesion", "minuto", "sigma"]].itertuples(
            index=False):
        ult = min(m + h, C.shape[1] - 1)
        tramo = np.log(C[d, m + 1:ult + 1] / C[d, m])
        golpes = np.flatnonzero(np.abs(tramo) >= k * s * np.sqrt(h))
        j = golpes[0] if len(golpes) else len(tramo) - 1
        fin.append(m + 1 + j)
        ret.append(float(tramo[j]) if len(tramo) else 0.0)
    return ev.assign(fin=fin, ret=ret)


ev = triple_barrera(C, eventos_cusum(lr, sig))
ev["y"] = (ev.lado * ev.ret > 0).astype(int)   # meta-label
ev["tendencia"] = tend[ev.sesion, ev.minuto]
print(len(ev), "events; primary hit rate:", round(ev.y.mean(), 3))
ev.groupby("tendencia")["y"].agg(["size", "mean"]).round(3)

6550 events; primary hit rate: 0.507
           size   mean
tendencia
False      4010  0.446
True       2540  0.604

The meta-label y is 1 if the return up to the first touch of the triple barrier ($\pm\sigma\sqrt{30}$, vertical at 30 minutes) has the sign of the primary's side. These are the README figures: 6,550 events and a hit rate of 50.7%, 60.4% in the 2,540 events in trend and 44.6% in the 4,010 in reversion. The tendencia column comes from hf5_verdad.parquet and will not be used as a feature: it serves to check what the meta-model learns.

Features known at the event

The meta-model can only use what is known at the close of the event minute. The README describes four reference features, all over the 30-minute window that ends at that close:

• imb_lado: the flow imbalance (aggressive buys minus sells, over volume) with the sign of the side. Positive when the flow pushes in favor of the trade. In FNX, in trend, the buying fraction has a persistent bias toward the direction of the episode; in both regimes, in addition, it follows the price of each minute.
• imb_abs: its absolute value, the intensity of the flow without direction.
• vol_rel: the logarithm of the window's volume relative to that of the same time slot in the previous 20 sessions. Comparing it with the same time slot removes the intraday U of volume.
• rv_rel: the logarithm of the window's realized volatility relative to the one the EWMA predicted.

d, m = ev.sesion.to_numpy(), ev.minuto.to_numpy()
cv = np.cumsum(V, axis=1)               # cumulative volume
ci = np.cumsum(2 * VC - V, axis=1)      # buys minus sells


def ultimos_30(acum, d, m):
    """Sum of the 30 minutes ending at minute m (inclusive)."""
    antes = np.where(m >= 30, acum[d, np.maximum(m - 30, 0)], 0)
    return acum[d, m] - antes


vol30 = ultimos_30(cv, d, m)
imb30 = ultimos_30(ci, d, m) / vol30
# volume of the same time slot in the previous 20 sessions
hist = np.array([
    ultimos_30(cv, np.arange(max(0, s - 20), s),
               np.full(min(s, 20), mm)).mean() if s > 0 else np.nan
    for s, mm in zip(d, m)])
lc = np.log(C)
rv30 = np.array([np.sqrt((np.diff(lc[s, max(mm - 30, 0):mm + 1])**2)
                         .sum()) for s, mm in zip(d, m)])
X = pd.DataFrame({
    "imb_lado": ev.lado * imb30,          # in favor of the side: > 0
    "imb_abs": np.abs(imb30),
    "vol_rel": np.log(vol30 / hist),
    "rv_rel": np.log(rv30 / (ev.sigma * np.sqrt(np.minimum(m, 30)))),
})
X.describe().loc[["count", "mean", "std"]].round(3)

       imb_lado   imb_abs   vol_rel    rv_rel
count  6550.000  6550.000  6528.000  6550.000
mean      0.033     0.055    -0.012     0.015
std       0.062     0.045     0.265     0.239

np.cumsum per session allows any window to be summed with one subtraction. The historical volume is computed with the 20 previous sessions, so the events of the first session have no vol_rel (they are the 22 that are missing). The imbalance in favor of the side is positive on average (0.033): it makes sense, because the CUSUM fires after a move and the flow follows the price. The meta-model has to tell when that flow is persistent.

Validation: two halves by sessions

Lesson 5.4.2 explains why cross-validation with overlapping labels needs purging. Here a simpler split is enough: train on the first 123 sessions and evaluate on the next 124. No label crosses a session close, so no training label shares minutes with a test one.

ok = np.isfinite(X).all(axis=1).to_numpy()
mitad = n_ses // 2
ent = ok & (d < mitad)
pru = ok & (d >= mitad)
y, t = ev.y.to_numpy(), ev.tendencia.to_numpy()
print("training:", ent.sum(), " test:", pru.sum())
print("primary hit rate in the test set:", round(y[pru].mean(), 3))
print("events in trend in the test set:", round(t[pru].mean(), 3))
print("labels that cross sessions:", (ev.fin > 389).sum())

training: 3241  test: 3287
primary hit rate in the test set: 0.501
events in trend in the test set: 0.402
labels that cross sessions: 0

In the second half the primary is right 50.1% of the time and 40.2% of the events fall in trend: that is the population on which everything that follows is measured.

The one-line baseline and the ceiling

Before training anything it is worth knowing how much the simplest rule that would occur to someone who has read the README gives, and how much the one who knows the regime would give:

def resumen(sel, nombre):
    """Fraction traded, precision and its standard error, mean in bp."""
    yy = y[pru][sel]
    r = (ev.lado * ev.ret).to_numpy()[pru][sel]
    return {"filtro": nombre, "N": int(sel.sum()),
            "fraccion": sel.mean(), "precision": yy.mean(),
            "error": np.sqrt(yy.mean() * (1 - yy.mean()) / len(yy)),
            "media_pb": 1e4 * r.mean()}


filas = [resumen(np.ones(pru.sum(), bool), "all"),
         resumen(X.imb_lado.to_numpy()[pru] > 0, "imb_lado > 0"),
         resumen(t[pru], "oracle: trend")]
pd.DataFrame(filas).set_index("filtro").round(3)

                  N  fraccion  precision  error  media_pb
filtro
all            3287     1.000      0.501  0.009     0.467
imb_lado > 0   2301     0.700      0.535  0.010     1.885
oracle: trend  1323     0.402      0.597  0.013     4.925

The rule imb_lado > 0 trades 70% of the events with 53.5% precision (standard error of 1.0 points) and a mean of 1.9 bp per trade. The oracle, which only trades in trend, reaches 59.7% and 4.9 bp: it is the ceiling. A meta-model that beat it in the second half would be measuring noise or using information it does not have.

The secondary model

A logistic regression with the standardized features (lesson 5.5.1), trained on the first half:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

meta = make_pipeline(StandardScaler(), LogisticRegression())
meta.fit(X[ent], y[ent])
p = meta.predict_proba(X[pru])[:, 1]
coef = meta[-1].coef_[0].round(3).tolist()
print("coefficients:", dict(zip(X.columns, coef)))
print("AUC against the hit:", round(roc_auc_score(y[pru], p), 3))
print("AUC against the regime:", round(roc_auc_score(t[pru], p), 3))
filas.insert(2, resumen(p > 0.5, "logistic p > 0.5"))
pd.DataFrame(filas).set_index("filtro").round(3)

coefficients: {'imb_lado': 0.244, 'imb_abs': 0.028, 'vol_rel': 0.074, 'rv_rel': -0.011}
AUC against the hit: 0.575
AUC against the regime: 0.707
                     N  fraccion  precision  error  media_pb
filtro
all               3287     1.000      0.501  0.009     0.467
imb_lado > 0      2301     0.700      0.535  0.010     1.885
logistic p > 0.5  1729     0.526      0.545  0.012     2.502
oracle: trend     1323     0.402      0.597  0.013     4.925

The numbers match the README reference ones. The coefficient of the imbalance in favor of the side (0.244) dominates the other three: relative volume contributes something (0.074) and realized volatility and unsigned intensity, almost nothing. The AUC against the hit is 0.575, modest, but against the true regime it is 0.707: the model does not see the regime and has learned to recognize it considerably better than it predicts each trade, because within each regime the outcome is still noisy.

Trading only when $p > 0.5$, precision goes to 54.5% (error of 1.2 points) with half of the events, and the mean per trade from 0.5 to 2.5 bp. It sits between the primary and the oracle, as it should, and barely above the one-line rule: the difference between 53.5% and 54.5% cannot be told apart from noise.

The proof that it has learned the regime shows up by deciles of its probability:

dec = pd.DataFrame({"p": p, "tendencia": t[pru], "acierto": y[pru],
                    "decil": pd.qcut(p, 10, labels=False) + 1})
dec.groupby("decil").mean().round(3).T

decil         1      2      3      4      5      6      7      8      9      10
p          0.407  0.443  0.462  0.479  0.496  0.513  0.533  0.556  0.589  0.650
tendencia  0.286  0.195  0.235  0.246  0.316  0.348  0.407  0.503  0.705  0.784
acierto    0.401  0.432  0.503  0.429  0.498  0.448  0.520  0.527  0.647  0.605

In the low deciles, between 20% and 29% of the events are in trend; in the highest, 78%. Precision rises with them: from 40% to 60-65%. And the probability is reasonably calibrated: in decile 10 the model says 0.65 and is right 60.5% of the time; in decile 1 it says 0.41 and is right 40.1% of the time. If it is going to be used as a size, that calibration matters as much as the AUC; with tree models it is worth checking and correcting it if needed.

Precision, recall and the threshold

The 0.5 threshold has nothing special about it. Raising it trades less and with more precision; lowering it, the opposite. The figure in the header goes through all the thresholds; the table, some of them:

yp = y[pru]
r = (ev.lado * ev.ret).to_numpy()[pru]
filas_u = []
for umbral in (0.45, 0.48, 0.50, 0.52, 0.55, 0.58):
    s = p > umbral
    prec, rec = yp[s].mean(), yp[s].sum() / yp.sum()
    filas_u.append({"umbral": umbral, "operadas": s.mean(),
                    "precision": prec, "recall": rec,
                    "F1": 2 * prec * rec / (prec + rec),
                    "media_pb": 1e4 * r[s].mean(),
                    "error_pb": 1e4 * r[s].std() / np.sqrt(s.sum())})
pd.DataFrame(filas_u).set_index("umbral").round(3)

        operadas  precision  recall     F1  media_pb  error_pb
umbral
0.45       0.814      0.521   0.847  0.645     1.311     0.415
0.48       0.647      0.532   0.687  0.600     2.006     0.468
0.50       0.526      0.545   0.572  0.558     2.502     0.525
0.52       0.411      0.570   0.468  0.514     3.917     0.594
0.55       0.275      0.602   0.330  0.427     5.128     0.734
0.58       0.172      0.631   0.217  0.323     6.423     0.969

• With a 0.45 threshold, 81% is traded with 52.1%; with 0.55, 27.5% with 60.2%, already at the oracle's level but with less than a third of the events and a larger standard error (0.73 bp on the mean).
• The F1 drops as the threshold rises: 0.645 with 0.45 and 0.427 with 0.55. Trading everything (recall 1, precision 0.501) would give the highest F1 of all, 0.668. The F1 weighs a missed winning trade the same as a losing trade taken; in trading, with costs, they do not weigh the same. The threshold is chosen with the economic metric (lesson 5.1.3): net mean per trade, number of trades and risk.
• The mean per trade rises from 1.3 to 6.4 bp, but the number of trades falls to a fifth. The net total, which is what gets paid, has an intermediate maximum.

A more flexible model?

If the logistic regression learns the regime with four features, a nonlinear model might learn more:

import lightgbm as lgb

gbm = lgb.LGBMClassifier(
    n_estimators=200, learning_rate=0.02, num_leaves=8,
    min_child_samples=50, subsample=0.8, subsample_freq=1,
    random_state=0, n_jobs=4, deterministic=True,
    force_row_wise=True, verbose=-1)
gbm.fit(X[ent], y[ent])
p_gbm = gbm.predict_proba(X[pru])[:, 1]
print("AUC against the hit:", round(roc_auc_score(y[pru], p_gbm), 3))
print("AUC against the regime:", round(roc_auc_score(t[pru], p_gbm), 3))
pd.DataFrame([resumen(p_gbm > 0.5, "LightGBM p > 0.5")]).set_index(
    "filtro").round(3)

AUC against the hit: 0.567
AUC against the regime: 0.653
                     N  fraccion  precision  error  media_pb
filtro
LightGBM p > 0.5  1676      0.51      0.548  0.012     2.509

LightGBM (lesson 5.5.3), with 200 small trees, comes out worse than the logistic regression in AUC (0.567 versus 0.575) and in recognizing the regime (0.653 versus 0.707), and almost the same in precision with $p > 0.5$ (54.8%). The relationship that matters here is almost linear in one feature; flexibility only adds variance. It is the usual result with little signal: the simple baseline and the linear model are the bar a complex model has to clear on data it has not seen, not on the training data.

From probability to size

The meta-model's probability can decide the size, not just the yes or no. López de Prado's rule (2018, chapter 10) treats $p$ as the result of a test of the hypothesis “the trade is a coin flip”:

with $\Phi$ the normal cumulative distribution function and $m = 0$ when $p \le 0.5$. The size is rounded to multiples of a step so as not to change the position over minimal differences in probability:

from scipy.stats import norm


def tamano(p, paso=0.05):
    """2·Φ(z) - 1 with z = (p - 1/2) / √(p(1 - p)); 0 if p ≤ 1/2;
    rounded to multiples of `paso`."""
    z = (p - 0.5) / np.sqrt(p * (1 - p))
    m = np.clip(2 * norm.cdf(z) - 1, 0, None)
    return np.round(m / paso) * paso


for q in (0.50, 0.55, 0.60, 0.65, 0.70):
    print(f"p = {q:.2f}  size = {tamano(np.array([q]))[0]:.2f}")
tam = tamano(p)
print("mean size:", round(tam.mean(), 3), " with size > 0:",
      (tam > 0).sum())

p = 0.50  size = 0.00
p = 0.55  size = 0.10
p = 0.60  size = 0.15
p = 0.65  size = 0.25
p = 0.70  size = 0.35
mean size: 0.056  with size > 0: 1441

The rule is cautious: with $p = 0.6$ it bets 0.15 of the maximum size and with 0.7 only 0.35. With the FNX probabilities, almost all between 0.40 and 0.65, the mean size is 0.056 and 1,441 events receive a size greater than zero. To compare it with the filter it has to be brought to the same total exposure: multiply the sizes by 9.4. Lesson 4.2.2 explains why the absolute size is a separate risk decision.

The economic result in the second half

The P&L of each trade is the size times the side's return up to the exit, minus a cost. The assumed cost is 0.6 bp per round-trip trade and unit of size: one FNX tick is about 0.47 bp at the year's average price (a one-tick bid-ask spread, paid half on entry and half on exit) plus a commission. It is an assumption, not market data; lesson 2.4.7 explains how to model it.

COSTE = 0.6e-4        # per round-trip trade and unit of size
ses_pru = d[pru]


def pyg(tam_op):
    """Net P&L per trade and annualized Sharpe of the daily P&L."""
    neto = tam_op * r - COSTE * tam_op
    diario = pd.Series(neto).groupby(ses_pru).sum()
    sr = diario.mean() / diario.std() * np.sqrt(252)
    return {"exposicion": tam_op.sum(),
            "bruto_pb_por_unidad": 1e4 * (tam_op * r).sum()
            / tam_op.sum(),
            "neto_total_pb": 1e4 * neto.sum(), "sharpe": sr,
            "error_sharpe": np.sqrt((1 + sr**2 / 504) / len(diario)
                                    * 252)}


escala = (p > 0.5).sum() / tam.sum()   # same total exposure
res = pd.DataFrame({
    "all": pyg(np.ones(len(p))),
    "logistic p > 0.5": pyg((p > 0.5).astype(float)),
    "size 2Φ(z) - 1": pyg(tam * escala),
    "oracle": pyg(t[pru].astype(float)),
}).T
print(f"size scale: x{escala:.1f}")
res.round(2)

size scale: x9.4
                  exposicion  bruto_pb_por_unidad  neto_total_pb  sharpe  error_sharpe
all                   3287.0                 0.47        -436.04   -0.45          1.43
logistic p > 0.5      1729.0                 2.50        3288.91    3.75          1.45
size 2Φ(z) - 1        1729.0                 4.51        6768.89    5.12          1.46
oracle                1323.0                 4.93        5722.38    8.01          1.51

• Trading everything loses: 0.47 bp gross per trade does not pay for 0.6 of cost.
• Filtering with $p > 0.5$ earns 2.5 bp gross per trade and 3,289 bp net over 124 sessions, with an annualized Sharpe of 3.75.
• Sizing with $2\Phi(z) - 1$, at the same total exposure, earns 4.5 bp per unit of size, very close to the oracle (4.9), and 6,769 bp net: it concentrates the size on the clearest events.
• The oracle trades less (1,323 units) with the best result per unit and a Sharpe of 8.0.

Two warnings. The standard error of each Sharpe is about 1.45 with 124 sessions: the difference between the filter and the sizing (3.75 versus 5.12) is one standard error and does not allow choosing between them with this sample. And the regime effect is planted with a large size so that the method shows: in a real market the signal is weaker, it changes over time and others exploit it. What the lesson shows is that meta-labeling recovers a structure that exists; not that it exists in any real future.

Common mistakes

• Features with later information. Any window that includes the minute after the event, or a full “daily” volume, leaks the outcome. In the meta-model it is especially easy, because the features are computed after the events are known.
• Training the secondary with the same labels the primary was tuned on. If the CUSUM threshold or the barriers were chosen by looking at the hit rate, the secondary learns on a selected sample.
• Using the probability as a size without checking calibration. A poorly calibrated probability gives systematically large or small sizes.
• Measuring with accuracy or F1. The goal is economic; the meta-model can worsen the F1 and improve the result.
• Forgetting the overlap. The meta-labels of nearby events share minutes; cross-validation within the training set needs purging and embargo (5.4.2), and bagging needs uniqueness weights (5.3.4).

Exercises

1. Swap the halves: train the meta-model on the second and evaluate it on the first. Do the AUC and the precision with $p > 0.5$ hold?

meta2 = make_pipeline(StandardScaler(), LogisticRegression())
meta2.fit(X[pru], y[pru])
p2 = meta2.predict_proba(X[ent])[:, 1]
print("AUC against the hit:", round(roc_auc_score(y[ent], p2), 3))
print("AUC against the regime:", round(roc_auc_score(t[ent], p2), 3))
s2 = p2 > 0.5
print(f"traded {s2.mean():.3f}  precision {y[ent][s2].mean():.3f}"
      f"  (all: {y[ent].mean():.3f})")

AUC against the hit: 0.574
AUC against the regime: 0.674
traded 0.535  precision 0.564  (all: 0.513)

2. Add as a feature whether the previous event in the same session had the same side (+1, −1 or 0 if it is the first). In trend, consecutive events tend to repeat the side. Does the out-of-sample AUC improve?

misma = (ev.sesion == ev.sesion.shift()).to_numpy()
prev = np.where(misma, ev.lado * ev.lado.shift().fillna(0), 0)
print("same side as the previous one, in trend / reversion:",
      round((prev[t] == 1).mean(), 3), round((prev[~t] == 1).mean(), 3))
X5 = X.assign(prev_igual=prev)
meta5 = make_pipeline(StandardScaler(), LogisticRegression())
meta5.fit(X5[ent], y[ent])
p5 = meta5.predict_proba(X5[pru])[:, 1]
print("AUC with the feature:", round(roc_auc_score(y[pru], p5), 3),
      " without it:", round(roc_auc_score(y[pru], p), 3))

same side as the previous one, in trend / reversion: 0.6 0.434
AUC with the feature: 0.574  without it: 0.575

3. An asset manager has an analyst who recommends buying or selling sector indices once a week. How would you apply meta-labeling to their recommendations and what would you be especially careful about?