Price, volume and microstructure features.

A feature is a function of the past

In financial machine learning, a feature (explanatory variable, predictor) is a number computed for each asset and each date with information available on that date, and expected to be related to what will happen next: the return, its sign, its magnitude. The definition has two parts and both matter. “With information available” excludes any later data, even if only by one minute (lesson 0.1.1). “Related to what will happen” is a hypothesis that has to be measured, and almost always the answer is “a little”.

This lesson works with the families that come from prices and volumes alone. Fundamentals and alternative data are lesson 5.2.2; leakage-free cross-sectional normalization, 5.2.4.

The path from the data to the model is always the same, and leaks creep in through the arrows on the left:

flowchart LR
  P[Prices and volume] --> F[Features at t]
  F --> N[Normalization]
  N --> M[Model]
  E[Label t to t+h] --> M

The panel and the recomputed features

The data are the synthetic level 5 panel: 400 fictitious stocks, 3,772 NYSE sessions from 2011 to 2025 and 22 weekly features published with their definition (data README). The generator knows the true expected return of each stock and week, so every measurement in this lesson can be compared with the answer. They are synthetic data: the effects are planted and no figure is evidence of what happens in a real market.

Before using a feature published by someone else (a vendor, a colleague, the generator itself), it is worth recomputing it from the prices. If it comes out different, either the definition is not what you think or there is an alignment problem:

import numpy as np
import pandas as pd

px = pd.read_parquet("panel5_precios.parquet")
C = px.pivot(index="fecha", columns="ticker", values="cierre")
V = px.pivot(index="fecha", columns="ticker", values="volumen")
R = C.pct_change()

fe = pd.read_parquet("panel5_features.parquet")
fechas = pd.DatetimeIndex(fe["fecha"].unique())
print(C.shape, len(fechas), "weeks")

mias = {
    "ret_1s": C / C.shift(5) - 1,
    "mom_12_1": C.shift(21) / C.shift(252) - 1,
    "vol_21d": R.rolling(21).std() * np.sqrt(252),
    "volumen_rel": np.log(V.rolling(5).mean()
                          / V.rolling(63).mean()),
}
for nombre, X in mias.items():
    pub = fe.pivot(index="fecha", columns="ticker", values=nombre)
    dif = (X.loc[fechas] - pub).abs().max().max()
    print(f"{nombre:12s} maximum difference {dif:.4f}")

(3772, 400) 730 weeks
ret_1s       maximum difference 0.0001
mom_12_1     maximum difference 0.0001
vol_21d      maximum difference 0.0001
volumen_rel  maximum difference 0.0007

All four match to the fourth decimal place, which is the precision with which they are published; relative volume differs somewhat more because the volume in the file is rounded to three significant figures. Two details make the computation point-in-time. All the windows (rolling, shift) end on the row of the date, that is, at that day's close: the feature is known at the close and can only be traded on in the next session. And the mom_12_1 momentum measures from $t-252$ to $t-21$: it skips the last month on purpose, for the reason shown in the next section.

Returns over several horizons

The measure used throughout the module is the rank IC (information coefficient): each week, the Spearman correlation between the feature and the following week's return across the 400 stocks; then its mean over the 729 weeks and a $t$ statistic that divides it by its standard error,

Using ranks protects against extreme values and measures what matters for ranking stocks. The weekly labels do not overlap, so treating the weeks as distinct observations is reasonable (with 21-session labels it would not be: lesson 5.3.4). The README oracle, which knows the true expected return, reaches 0.029: that is the ceiling of any feature or model on these data (lesson 5.1.1).

et = pd.read_parquet("panel5_etiquetas.parquet")
Y = et.pivot(index="fecha", columns="ticker",
             values="ret_sig_semana").loc[fechas]

def ic(X, y=Y):
    """Weekly rank IC: mean, t and number of weeks."""
    X = X.reindex(index=fechas, columns=y.columns)
    v = X.rank(axis=1).corrwith(y.rank(axis=1), axis=1)
    v = v.dropna()
    t = v.mean() / v.std() * np.sqrt(len(v))
    return {"ic": round(v.mean(), 4), "t": round(t, 1),
            "semanas": len(v)}

tabla = {f"{h} sessions": ic(C / C.shift(h) - 1)
         for h in [1, 2, 5, 10, 21, 63, 126, 252]}
tabla["mom_12_1"] = ic(C.shift(21) / C.shift(252) - 1)
pd.DataFrame(tabla).T.astype({"semanas": int})

                  ic    t  semanas
1 sessions   -0.0040 -1.0      729
2 sessions   -0.0095 -2.4      729
5 sessions   -0.0159 -4.3      729
10 sessions  -0.0109 -2.9      729
21 sessions  -0.0077 -2.1      729
63 sessions  -0.0050 -1.6      729
126 sessions -0.0000 -0.0      729
252 sessions  0.0053  1.7      729
mom_12_1      0.0077  2.4      729

The header figure plots the table. The generator plants a 5-session reversal (ret_1s) and a 12-month momentum that skips the last one (mom_12_1); the other horizons have no effect of their own. Even so:

• Neighbors inherit. The 2-session return is part of the 5-session one, and the 10-session one contains it entirely: both come out negative and significant (t = −2.4 and −2.9) with no effect of their own. The 1-session one, with less of the week's information, stays at t = −1.0.
• Effects mask each other. At 126 sessions, the recent reversal and the older momentum offset each other and the IC is zero. At 252, momentum shows up (0.0053, t = 1.7), but contaminated by the last month; removing it, mom_12_1 rises to 0.0077 (t = 2.4). That is the reason for the classic “12-1” of Jegadeesh and Titman (1993)11Note 1 · N. Jegadeesh and S. Titman, “Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency”, Journal of Finance 48(1), 1993, pp. 65-91, doi:10.1111/j.1540-6261.1993.tb04702.x., who separated momentum from the one-month reversal documented by Jegadeesh (1990)22Note 2 · N. Jegadeesh, “Evidence of Predictable Behavior of Security Returns”, Journal of Finance 45(3), 1990, pp. 881-898, doi:10.1111/j.1540-6261.1990.tb05110.x..
• A significant IC is not a tradable signal. The best, −0.016, is a little over half of the oracle's ceiling, and the long-short portfolio built with signals like this does not cover the costs of its weekly turnover (data README; lesson 5.1.3).

With 22 features and 8 horizons tested, moreover, a $t$ of 2 is not impressive: it is a multiple search (lesson 2.1.8). Here it can be checked against the truth; on real data, it cannot.

The second common variant divides the return by the stock's volatility, so that a 5% rise in a quiet stock weighs more than in a volatile one:

vol63 = R.rolling(63).std()
for h in [5, 21, 252]:
    bruto = C / C.shift(h) - 1
    ajustado = bruto / (vol63 * np.sqrt(h))
    print(f"{h:3d} sessions: raw {ic(bruto)['ic']:+.4f}, "
          f"/vol {ic(ajustado)['ic']:+.4f} "
          f"(t = {ic(ajustado)['t']:+.1f})")

  5 sessions: raw -0.0159, /vol -0.0154 (t = -4.7)
 21 sessions: raw -0.0077, /vol -0.0064 (t = -2.0)
252 sessions: raw +0.0053, /vol +0.0066 (t = +2.2)

Dividing by volatility barely changes the IC at 5 sessions (−0.0159 against −0.0154, with a somewhat larger $t$) and improves the 252-session one somewhat (from 0.0053 to 0.0066). With ranks within each week, the effect of the normalization is small because the scale no longer matters; what changes is the order between stocks of different volatility. In a model that uses the values (a regression, a network), normalization matters much more: without it, volatile stocks dominate the loss. Lesson 5.2.4 covers how to normalize without using future information.

Volatility: it anticipates the magnitude, not the direction

Realized volatility is the easiest feature to predict and the one most often misunderstood. It is very persistent (volatility clustering, lesson 1.6.2), so the volatility of the last few weeks anticipates the next one well. What it does not anticipate is the sign:

for w in [5, 21, 63]:
    v = R.rolling(w).std()
    con_abs = ic(v, Y.abs())
    con_signo = ic(v)
    print(f"vol {w:2d} sessions: IC with |r| {con_abs['ic']:.3f} "
          f"(t = {con_abs['t']:.0f}); with r {con_signo['ic']:+.4f} "
          f"(t = {con_signo['t']:+.1f})")

vol  5 sessions: IC with |r| 0.171 (t = 56); with r +0.0024 (t = +0.5)
vol 21 sessions: IC with |r| 0.220 (t = 69); with r +0.0020 (t = +0.4)
vol 63 sessions: IC with |r| 0.227 (t = 76); with r +0.0032 (t = +0.6)

All three windows rank stocks very well by the magnitude of the following week's move: an IC of 0.22 with 63 sessions, about 30 times the oracle's ceiling for direction, with a $t$ of 76. With the signed return, the IC is zero. A risk feature like this is used to normalize others (the previous section), to condition them (an interaction) and to size positions (lesson 4.2.2), not as a direction predictor.

The generator plants a small, negative low-volatility effect in vol_63d, and it does not show: volatile stocks have more beta and, since the market rises on average, beta offsets the premium (the README recovers it by removing the true beta). It is a good reminder that the univariate IC mixes the effect of a feature with that of everything that resembles it.

Volume and interactions

Volume in shares is not comparable across companies (a 2 USD stock trades many more shares than an 800 USD one). The usual approach is relative volume: the volume of the last few days against that of a longer period for the same stock, in logarithms. In the panel, volumen_rel = log(mean volume over 5 sessions / over 63 sessions) has no effect of its own, but the generator uses it in an interaction: the week's reversal is stronger when the move happened on low volume, $\alpha \propto g(\text{ret\_1s})\,(1 - 0.5\,g(\text{volumen\_rel}))$. It is the economic idea that a move without volume is usually liquidity noise, and one with volume, information. It shows up when the sample is split by terciles:

r1s = C / C.shift(5) - 1
vrel = np.log(V.rolling(5).mean() / V.rolling(63).mean())
q = vrel.loc[fechas].rank(axis=1, pct=True)
tramos = {"low": (0, 1 / 3), "mid": (1 / 3, 2 / 3),
          "high": (2 / 3, 1)}
serie = {}
for nombre, (lo, hi) in tramos.items():
    X = r1s.loc[fechas].where((q > lo) & (q <= hi))
    serie[nombre] = X.rank(axis=1).corrwith(Y.rank(axis=1), axis=1)
    r = ic(X)
    print(f"relative volume {nombre:4s}: IC {r['ic']:+.4f} "
          f"(t = {r['t']:+.1f})")
dif = (serie["low"] - serie["high"]).dropna()
t_dif = dif.mean() / dif.std() * np.sqrt(len(dif))
print(f"low - high: {dif.mean():+.4f} (t = {t_dif:+.1f})")

centrado = lambda X: X.rank(axis=1, pct=True) - 0.5
inter = centrado(r1s.loc[fechas]) * (1 - centrado(vrel.loc[fechas]))
r = ic(inter)
print(f"interaction: IC {r['ic']:+.4f} (t = {r['t']:+.1f})")

relative volume low : IC -0.0240 (t = -5.1)
relative volume mid : IC -0.0162 (t = -3.3)
relative volume high: IC -0.0080 (t = -1.9)
low - high: -0.0160 (t = -3.4)
interaction: IC -0.0170 (t = -4.6)

The reversal with low volume is three times the one with high volume (−0.024 against −0.008), and the difference, measured week by week over the same dates, is −0.016 with $t$ = −3.4: it is not noise. The hand-built interaction feature, with centered ranks, gives an IC of −0.017, somewhat better than the 5-session return alone.

A decision tree or a boosting model (lessons 5.5.2 and 5.5.3) find interactions like this without building them; a linear model does not, unless it is given them. Building them by hand has the advantage that they can be reasoned about and tested one by one, and the drawback that each interaction tried is one more test in the multiple search.

Normalized indicators

Technical indicators (lesson 1.7.1) are almost always built in price units: the distance to a moving average in dollars, a MACD, a Bollinger band. In a panel, those units make the indicator measure mostly how much the stock is worth, not what you wanted to measure. And many indicators are, deep down, the same number on another scale:

sma50 = C.rolling(50).mean()
en_precio = (C - sma50).loc[fechas]
en_tanto = (C / sma50 - 1).loc[fechas]
en_vol = (en_tanto / (vol63.loc[fechas] * np.sqrt(50)))
nivel = C.loc[fechas].rank(axis=1)

for nombre, X in [("C - SMA50 (USD)", en_precio),
                  ("C / SMA50 - 1", en_tanto),
                  ("(C / SMA50 - 1) / vol", en_vol)]:
    rho = X.abs().rank(axis=1).corrwith(nivel, axis=1).mean()
    print(f"{nombre:22s} corr. of |X| with price: {rho:+.2f}")

rsi = fe.pivot(index="fecha", columns="ticker", values="rsi_14")
r14 = (C / C.shift(14) - 1).loc[fechas]
rho = rsi.rank(axis=1).corrwith(r14.rank(axis=1), axis=1).mean()
print(f"RSI 14 against 14-session return: {rho:.2f}")

C - SMA50 (USD)        corr. of |X| with price: +0.72
C / SMA50 - 1          corr. of |X| with price: +0.01
(C / SMA50 - 1) / vol  corr. of |X| with price: +0.00
RSI 14 against 14-session return: 0.95

The distance to the 50-session average in dollars has a rank correlation of 0.72 between its absolute value and the price: in an 800 USD stock, a “large” distance is simply that of an expensive stock. Divided by the average, the relationship disappears (0.01); divided also by the expected volatility over 50 sessions, it becomes a number of standard deviations, comparable across stocks and across dates. The three rules: divide by the price, by the volatility and, for volume, by the stock's own history.

The last line is the redundancy. The 14-session RSI and the 14-session return have a mean rank correlation of 0.95: they are almost the same feature. That is why rsi_14 comes out with a negative IC (−0.008, t = −2.6 in the README) without having an effect of its own: it inherits the reversal. Putting into a model twenty indicators that are transformations of the same return adds no information and does add noise and collinearity (lesson 5.2.5 measures importance by groups of correlated features).

Microstructure: order imbalance

Daily price and volume features summarize what has already happened in the price. Microstructure looks at who pushed: whether the volume of each interval was initiated by aggressive buyers or sellers (those who cross the bid-ask spread, lesson 1.1.1). The order imbalance of an interval is

It is the trade-based version of the order book order flow imbalance of lesson 4.3.2. Its practical problem is that the aggressor side is not always in the data: the consolidated tape of US equities does not carry it, and that is why classifiers exist. The tick rule assigns each trade to the side of the last price change, and the Lee and Ready (1991)33Note 3 · C. M. C. Lee and M. J. Ready, “Inferring Trade Direction from Intraday Data”, Journal of Finance 46(2), 1991, pp. 733-746, doi:10.1111/j.1540-6261.1991.tb02683.x. algorithm combines it with the prevailing quote. With bars, without individual trades, the bulk volume classification (BVC) of Easley, López de Prado and O'Hara44Note 4 · D. Easley, M. López de Prado and M. O'Hara, “Flow Toxicity and Liquidity in a High-frequency World”, Review of Financial Studies 25(5), 2012, pp. 1457-1493, doi:10.1093/rfs/hhs053 (VPIN and bulk volume classification); by the same authors, “Discerning information from trade data”, Journal of Financial Economics 120(2), 2016, pp. 269-285, doi:10.1016/j.jfineco.2016.01.018. splits the volume of each bar according to the standardized price change: $V_{\mathrm{buy}} = V\,\Phi(\Delta p/\sigma_{\Delta p})$.

The data are one year of one-minute bars of FNX, a fictitious future on an index (247 sessions of 390 minutes), with the true buy volume of each minute. The generator alternates two hidden regimes: trend, with informed flow that pushes in one direction for an hour on average, and 30-minute mean reversion. The buy fraction follows the price in both regimes and, only in trend, also carries a persistent bias toward the direction of the episode (0.5 + 0.12·tanh(r/σ) + 0.03·D + noise). That bias is the footprint of informed flow, and it is not in the price.

from scipy.stats import norm

b = pd.read_parquet("hf5_barras_1min.parquet")
b["sesion"] = b["ts"].dt.date
g = b.groupby("sesion")
b["r"] = np.log(b["cierre"]).groupby(b["sesion"]).diff()

# true side (the aggressor of each trade)
b["vc"] = b["volumen_comprador"]
# tick rule: the whole minute to the side of the price change
signo = np.sign(b["cierre"].groupby(b["sesion"]).diff())
b["vc_tick"] = b["volumen"] * (signo.fillna(0) + 1) / 2
# BVC: buy fraction = Φ(r / σ), σ from the previous minutes
sigma = g["r"].transform(
    lambda s: s.ewm(span=100, min_periods=10).std().shift(1))
frac = pd.Series(norm.cdf(b["r"] / sigma), index=b.index)
b["vc_bvc"] = b["volumen"] * frac.fillna(0.5)

d = {m: 2 * b[c] / b["volumen"] - 1
     for m, c in [("verdad", "vc"), ("tick", "vc_tick"),
                  ("bvc", "vc_bvc")]}
d = pd.DataFrame(d)[b["r"].notna() & sigma.notna()]
print(len(b), "minutes;", b["sesion"].nunique(), "sessions")
d.corr().round(3)

96330 minutes; 247 sessions
        verdad   tick    bvc
verdad   1.000  0.591  0.658
tick     0.591  1.000  0.861
bvc      0.658  0.861  1.000

Minute by minute, the two classifiers are very similar to each other (0.86) and much less similar to the truth (0.59 for the tick rule and 0.66 for BVC). That is to be expected: both infer the side from the price move, so they rebuild the part of the flow that follows the price and lose the rest. The question for a model is whether what they lose matters. The trend regime is what you want to detect, and the 30-minute imbalance is the natural feature:

from sklearn.metrics import roc_auc_score

verdad = pd.read_parquet("hf5_verdad.parquet")
b["regimen"] = verdad["regimen"].to_numpy()  # same row, same ts

def roll30(col):
    return g[col].transform(lambda s: s.rolling(30).sum())

vol30 = roll30("volumen")
for m, c in [("verdad", "vc"), ("tick", "vc_tick"),
             ("bvc", "vc_bvc")]:
    b[f"di_{m}"] = 2 * roll30(c) / vol30 - 1
b["r30"] = g["r"].transform(lambda s: s.rolling(30).sum())

ok = b[["di_verdad", "r30"]].notna().all(axis=1)
tend = b.loc[ok, "regimen"] == "tendencia"
print(f"minutes in trend: {tend.mean():.1%}")
for c in ["r30", "di_verdad", "di_tick", "di_bvc"]:
    auc = roc_auc_score(tend, b.loc[ok, c].abs())
    print(f"AUC of |{c}| for the regime: {auc:.3f}")

minutes in trend: 35.1%
AUC of |r30| for the regime: 0.603
AUC of |di_verdad| for the regime: 0.682
AUC of |di_tick| for the regime: 0.556
AUC of |di_bvc| for the regime: 0.564

The AUC measures how well the absolute value of each feature separates trend minutes from reversion minutes (0.5 is chance, 1 is perfect). The return of the last 30 minutes already separates them somewhat (0.603), because in trend the price moves more in one direction. The imbalance with the true side separates them more (0.682): it adds the persistent bias that the price does not have. The two classified versions end up below the return itself (0.556 and 0.564): they are a noisy version of the price change, and on top of that the classification noise accumulates over 30 minutes.

Does it anticipate the following 30 minutes?

Distinguishing the regime is not yet anticipating the price. The honest test is the one from the 5.4 lessons: one observation every 30 minutes (the 30-minute windows do not overlap), a model fitted on the first half of the sessions and evaluated on the second, and the comparison with the simplest baseline, the return of the previous 30 minutes (intraday momentum). The target is the return of the following 30 minutes divided by the expected volatility:

import statsmodels.api as sm

b["minuto"] = g.cumcount()
b["sigma"] = g["r"].transform(
    lambda s: s.ewm(span=100, min_periods=10).std())
b["fut30"] = g["r"].transform(
    lambda s: s[::-1].rolling(30).sum()[::-1].shift(-1))
b["y"] = b["fut30"] / (b["sigma"] * np.sqrt(30))
b["r30z"] = b["r30"] / (b["sigma"] * np.sqrt(30))

cols = ["y", "r30z", "di_verdad", "di_tick", "di_bvc"]
m = b[(b["minuto"] % 30 == 0)
      & b[cols].notna().all(axis=1)].copy()
sesiones = np.sort(m["sesion"].unique())
ent = m["sesion"] < sesiones[len(sesiones) // 2]
print(f"{ent.sum()} training observations, "
      f"{(~ent).sum()} test; standard error ≈ "
      f"{1 / np.sqrt((~ent).sum()):.3f}")

for x in [["r30z"], ["di_tick"], ["di_bvc"], ["di_verdad"],
          ["r30z", "di_verdad"]]:
    X = sm.add_constant(m[x])
    fit = sm.OLS(m.loc[ent, "y"], X[ent]).fit()
    pred = fit.predict(X[~ent])
    c = np.corrcoef(pred, m.loc[~ent, "y"])[0, 1]
    print(f"{' + '.join(x):18s} out-of-sample corr. {c:+.3f}")

1353 training observations, 1364 test; standard error ≈ 0.027
r30z               out-of-sample corr. +0.111
di_tick            out-of-sample corr. +0.049
di_bvc             out-of-sample corr. +0.061
di_verdad          out-of-sample corr. +0.182
r30z + di_verdad   out-of-sample corr. +0.182

With 1,364 test observations, the standard error of each correlation is about 0.027. The past return gives 0.111: there is intraday momentum, because part of the time is trend. The two classified imbalances end up below it (0.049 and 0.061): they add nothing to the price they come from. The true imbalance reaches 0.182, and combined with the past return it does not improve: it contains it. The figure in the previous section summarizes the two measures.

VPIN: imbalance in volume time

The VPIN (volume-synchronized probability of informed trading) of Easley, López de Prado and O'Hara (2012)⁴ measures the same imbalance in volume time: it splits trading into buckets of a fixed volume $V$ and averages the absolute imbalance of the last $n$ buckets,

Measuring in volume buckets makes the indicator advance quickly when there is a lot of activity and slowly when there is little, like the volume bars of lesson 1.4.5. Its authors presented it as a measure of flow toxicity for market makers and as a short-term volatility indicator.

flowchart LR
  B[1-min bars] --> C[Volume buckets]
  C --> D[Imbalance]
  D --> M[Mean of 10 buckets]

The bucket size is set with earlier data (here, 1/50 of the mean daily volume of the first 20 sessions) and is not recomputed with the whole sample, which would be looking at the future. A minute that crosses the edge of a bucket is split between the two in proportion to its volume:

def vpin(vol, vc, cubo, ventana=10):
    """VPIN per bucket: rolling mean of |2·V_buy − V| / V."""
    cum_v = np.cumsum(vol)
    bordes = np.arange(cubo, cum_v[-1], cubo)
    # cumulative buy volume at each edge (linear split of the
    # minute that crosses the edge between the two buckets)
    cum_c = np.interp(bordes, cum_v, np.cumsum(vc))
    comp = np.diff(np.r_[0.0, cum_c])
    desequilibrio = np.abs(2 * comp - cubo) / cubo
    vp = pd.Series(desequilibrio).rolling(ventana).mean()
    cierre = np.searchsorted(cum_v, bordes)  # closing minute
    return vp.to_numpy(), cierre

vol = b["volumen"].to_numpy(float)
# bucket set with the first 20 sessions: 1/50 of daily volume
cubo = b.groupby("sesion")["volumen"].sum().iloc[:20].mean() / 50
print(f"bucket of {cubo:,.0f} contracts")

res = {}
for m_, c in [("verdad", "vc"), ("tick", "vc_tick"),
              ("bvc", "vc_bvc")]:
    vp, cierre = vpin(vol, b[c].to_numpy(float), cubo)
    res[m_] = vp
res = pd.DataFrame(res)
res["regimen"] = b["regimen"].to_numpy()[cierre]
res = res.dropna()
print(len(res), "buckets; mean VPIN with the true side",
      round(res["verdad"].mean(), 3))
print(res.groupby("regimen")[["verdad", "tick", "bvc"]]
      .mean().round(3))
tend = res["regimen"] == "tendencia"
for m_ in ["verdad", "tick", "bvc"]:
    print(f"AUC of the VPIN ({m_}): "
          f"{roc_auc_score(tend, res[m_]):.3f}")

bucket of 23,644 contracts
12687 buckets; mean VPIN with the true side 0.084
           verdad   tick    bvc
regimen
reversion   0.079  0.311  0.197
tendencia   0.091  0.337  0.210
AUC of the VPIN (verdad): 0.639
AUC of the VPIN (tick): 0.575
AUC of the VPIN (bvc): 0.565

With the true side, the mean VPIN is 0.084: 0.079 in reversion and 0.091 in trend, with an AUC of 0.639 for separating the regimes, somewhat less than the 30-minute imbalance (the data README, with buckets of 1/50 of the generator's nominal daily volume, 22,000 contracts, gives 0.094 and 0.080: the level depends on the bucket size). With the classifiers two things happen. The level shoots up: 0.32 with the tick rule and 0.20 with BVC, because each classification error moves volume from one side to the other and, in absolute value, almost never cancels out. And the separation falls to 0.575 and 0.565. A VPIN threshold set with one classification method is not valid with another, and a published level cannot be compared with one computed with other buckets.

The figure shows a session chosen as an illustration (the mean difference between regimes is much smaller than on this day). In the first trend episode, a drop of almost 80 points in an hour and a half, the VPIN rises to 0.15; in the afternoon reversion it stays between 0.04 and 0.07.

The conclusions of the original VPIN paper were widely debated. Andersen and Bondarenko55Note 5 · T. G. Andersen and O. Bondarenko, “VPIN and the flash crash”, Journal of Financial Markets 17, 2014, pp. 1-46, doi:10.1016/j.finmar.2013.05.005, and “Reflecting on the VPIN dispute”, Journal of Financial Markets 17, 2014, pp. 53-64, doi:10.1016/j.finmar.2013.08.002 (abstract accessed on October 23, 2026). showed with S&P 500 futures data that the VPIN is correlated by construction with volume and volatility, that after controlling for them they find no additional predictive power for future volatility and that its properties depend heavily on the trade classification. For a model, the practical lesson is that of the whole section: a microstructure feature is worth what the aggressor-side data that feeds it is worth, and it is always compared with what the price, the volume and the volatility already say.

Common mistakes

• Computing the feature with the day's close and trading at that same close. The feature is known at the close; the order goes to the next session (or to the closing auction, if it is computed a few minutes earlier with prices from that moment).
• Horizons that repeat each other. Returns of 1, 2, 5 and 10 sessions share almost all of their information; with nested windows, the model spreads one effect across several columns and the importance of each one is misleading.
• Price units in a panel. A distance in dollars, an unnormalized MACD or a volume in shares rank by the size or the price of the stock, not by what you wanted to measure.
• Confusing risk with direction. Volatility anticipates the magnitude of the move; if the model predicts signed returns, its high IC with the absolute value says nothing.
• Classifying the aggressor side and believing you have the flow. The tick rule and BVC rebuild what is already in the price. If the signal depends on the flow, you need the aggressor-side data.
• Windows that cross sessions. A 30-minute imbalance at 9:35 that includes the end of the previous session mixes two days; in FNX the rolling windows are grouped by session (the VPIN, on the other hand, does carry over from one day to the next, as in its original definition).
• Evaluating with overlapping windows. One observation per minute with a 30-minute target inflates the number of observations about 30 times and makes everything look significant.

Exercises

1. dist_max_52s and rsi_14 have no effect of their own in the panel, but their univariate IC is not zero. Remove, week by week, the part of each one explained by the 5-session return (rank regression) and compute the IC of what remains. Which features are redundant?

def residuo_rangos(X, Z):
    # rank of X minus its linear projection onto the rank of Z
    rx, rz = X.rank(axis=1), Z.rank(axis=1)
    rx = rx.sub(rx.mean(axis=1), axis=0)
    rz = rz.sub(rz.mean(axis=1), axis=0)
    beta = (rx * rz).sum(axis=1) / (rz * rz).sum(axis=1)
    return rx - rz.mul(beta, axis=0)

r1 = fe.pivot(index="fecha", columns="ticker", values="ret_1s")
for nombre in ["rsi_14", "dist_max_52s"]:
    X = fe.pivot(index="fecha", columns="ticker", values=nombre)
    antes, despues = ic(X), ic(residuo_rangos(X, r1))
    print(f"{nombre:13s} IC {antes['ic']:+.4f} (t = {antes['t']:+.1f})"
          f" -> without ret_1s {despues['ic']:+.4f} "
          f"(t = {despues['t']:+.1f})")

rsi_14        IC -0.0083 (t = -2.6) -> without ret_1s -0.0005 (t = -0.2)
dist_max_52s  IC -0.0031 (t = -1.0) -> without ret_1s +0.0019 (t = +0.7)

2. Repeat the VPIN with the true side with buckets twice as large (1/25 of daily volume) and with a window of 50 buckets. How do the mean level and the regime AUC change? Which parameters would you choose, and how, without looking at the test sample?

vc = b["vc"].to_numpy(float)
for factor, ventana in [(1, 10), (2, 10), (1, 50), (2, 50)]:
    vp, cierre = vpin(vol, vc, cubo * factor, ventana)
    t = pd.DataFrame({"vp": vp,
                      "reg": b["regimen"].to_numpy()[cierre]})
    t = t.dropna()
    auc = roc_auc_score(t["reg"] == "tendencia", t["vp"])
    print(f"bucket x{factor}, window {ventana:2d}: mean "
          f"{t['vp'].mean():.3f}, AUC {auc:.3f}")

bucket x1, window 10: mean 0.084, AUC 0.639
bucket x2, window 10: mean 0.064, AUC 0.613
bucket x1, window 50: mean 0.084, AUC 0.556
bucket x2, window 50: mean 0.064, AUC 0.543

3. Your data vendor for a future gives you one-minute OHLCV bars, without the aggressor side. A colleague proposes building the imbalance with BVC and training a model with it. What would you tell them, and what data would you ask for?