--- title: "5.2.1 Price, volume and microstructure features" description: "Returns over several horizons, volatility, normalized indicators, order imbalance and VPIN, measured against the known answer." url: "https://pythonparatrading.com/en/python-trading-course/level-5/price-volume-microstructure-features/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/price-volume-microstructure-features.png" --- # 5.2.1 · Price, volume and microstructure features > Almost every trading model starts by turning prices and volumes into numbers that are comparable across assets and across dates. This lesson builds the classic features on a panel of 400 stocks with the published ground truth, measures how much each one anticipates and moves on to microstructure: the order imbalance and the VPIN of a future in one-minute bars, computed with the true aggressor side and with the classifiers you have to use when you do not have it. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.2 · Feature engineering - Duration: 70 min - Prerequisites: [1.4.5 · Building bars: time, tick, volume, dollar and imbalance](https://pythonparatrading.com/en/python-trading-course/level-1/time-tick-volume-and-dollar-bars/), [1.7.1 · Technical indicators in Python](https://pythonparatrading.com/en/python-trading-course/level-1/technical-indicators-in-python/), [2.3.1 · Trend following and momentum](https://pythonparatrading.com/en/python-trading-course/level-2/trend-following-and-momentum/), [4.3.2 · Order flow imbalance, microstructure signals and latency](https://pythonparatrading.com/en/python-trading-course/level-4/order-flow-imbalance-and-latency/), [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/) ## By the end of this lesson - Build returns over several horizons, volatilities, relative volume and normalized indicators with data known at the close of each date, and check them against the published features. - Measure the rank IC of each feature with its error and separate its own effect from the one inherited from a neighboring feature. - Distinguish a risk feature (it anticipates the magnitude of the move) from a direction feature, and build an interaction with volume. - Compute the order imbalance and the VPIN with the true buy volume, with the tick rule and with bulk volume classification (BVC), and measure how much information is lost by classifying. - Evaluate a microstructure feature without overlap and out of sample, against the baseline of the past return. ## A feature is a function of the past In financial machine learning, a **feature** (explanatory variable, predictor) is a number computed for each asset and each date with information available on that date, and expected to be related to what will happen next: the return, its sign, its magnitude. The definition has two parts and both matter. “With information available” excludes any later data, even if only by one minute (lesson 0.1.1). “Related to what will happen” is a hypothesis that has to be measured, and almost always the answer is “a little”. This lesson works with the families that come from prices and volumes alone. Fundamentals and alternative data are lesson 5.2.2; leakage-free cross-sectional normalization, 5.2.4. | Family | Examples | What it tries to capture | Typical trap | |---|---|---|---| | Returns | 1, 5, 21, 63 sessions; 12-1 momentum | Short-term reversal, medium-term trend | Overlapping horizons that repeat the same information | | Volatility | 21- and 63-session standard deviation | Risk, regime | Confusing anticipating the magnitude with anticipating the direction | | Volume | Relative volume, turnover | Attention, liquidity, confirmation | Absolute units (shares) that are not comparable | | Indicators | RSI, distance to the high, moving averages | The same as returns, on another scale | Price units; redundancy among them | | Microstructure | Order imbalance, VPIN | Informed flow, toxicity | Classifying the aggressor side without having it | The path from the data to the model is always the same, and leaks creep in through the arrows on the left: ```mermaid flowchart LR P[Prices and volume] --> F[Features at t] F --> N[Normalization] N --> M[Model] E[Label t to t+h] --> M ``` ## The panel and the recomputed features The data are the synthetic level 5 panel: 400 fictitious stocks, 3,772 NYSE sessions from 2011 to 2025 and 22 weekly features published with their definition (data README). The generator knows the true expected return of each stock and week, so every measurement in this lesson can be compared with the answer. They are synthetic data: the effects are planted and no figure is evidence of what happens in a real market. Before using a feature published by someone else (a vendor, a colleague, the generator itself), it is worth recomputing it from the prices. If it comes out different, either the definition is not what you think or there is an alignment problem: ```python import numpy as np import pandas as pd px = pd.read_parquet("panel5_precios.parquet") C = px.pivot(index="fecha", columns="ticker", values="cierre") V = px.pivot(index="fecha", columns="ticker", values="volumen") R = C.pct_change() fe = pd.read_parquet("panel5_features.parquet") fechas = pd.DatetimeIndex(fe["fecha"].unique()) print(C.shape, len(fechas), "weeks") mias = { "ret_1s": C / C.shift(5) - 1, "mom_12_1": C.shift(21) / C.shift(252) - 1, "vol_21d": R.rolling(21).std() * np.sqrt(252), "volumen_rel": np.log(V.rolling(5).mean() / V.rolling(63).mean()), } for nombre, X in mias.items(): pub = fe.pivot(index="fecha", columns="ticker", values=nombre) dif = (X.loc[fechas] - pub).abs().max().max() print(f"{nombre:12s} maximum difference {dif:.4f}") ``` ```output (3772, 400) 730 weeks ret_1s maximum difference 0.0001 mom_12_1 maximum difference 0.0001 vol_21d maximum difference 0.0001 volumen_rel maximum difference 0.0007 ``` All four match to the fourth decimal place, which is the precision with which they are published; relative volume differs somewhat more because the volume in the file is rounded to three significant figures. Two details make the computation point-in-time. All the windows (`rolling`, `shift`) end on the row of the date, that is, at that day's close: the feature is known at the close and can only be traded on in the next session. And the `mom_12_1` momentum measures from $t-252$ to $t-21$: it skips the last month on purpose, for the reason shown in the next section. ## Returns over several horizons The measure used throughout the module is the **rank IC** (*information coefficient*): each week, the Spearman correlation between the feature and the following week's return across the 400 stocks; then its mean over the 729 weeks and a $t$ statistic that divides it by its standard error, $$ t = \frac{\overline{\mathrm{IC}}}{s_{\mathrm{IC}}/\sqrt{n}}. $$ Using ranks protects against extreme values and measures what matters for ranking stocks. The weekly labels do not overlap, so treating the weeks as distinct observations is reasonable (with 21-session labels it would not be: lesson 5.3.4). The README oracle, which knows the true expected return, reaches 0.029: that is the ceiling of any feature or model on these data (lesson 5.1.1). ```python et = pd.read_parquet("panel5_etiquetas.parquet") Y = et.pivot(index="fecha", columns="ticker", values="ret_sig_semana").loc[fechas] def ic(X, y=Y): """Weekly rank IC: mean, t and number of weeks.""" X = X.reindex(index=fechas, columns=y.columns) v = X.rank(axis=1).corrwith(y.rank(axis=1), axis=1) v = v.dropna() t = v.mean() / v.std() * np.sqrt(len(v)) return {"ic": round(v.mean(), 4), "t": round(t, 1), "semanas": len(v)} tabla = {f"{h} sessions": ic(C / C.shift(h) - 1) for h in [1, 2, 5, 10, 21, 63, 126, 252]} tabla["mom_12_1"] = ic(C.shift(21) / C.shift(252) - 1) pd.DataFrame(tabla).T.astype({"semanas": int}) ``` ```output ic t semanas 1 sessions -0.0040 -1.0 729 2 sessions -0.0095 -2.4 729 5 sessions -0.0159 -4.3 729 10 sessions -0.0109 -2.9 729 21 sessions -0.0077 -2.1 729 63 sessions -0.0050 -1.6 729 126 sessions -0.0000 -0.0 729 252 sessions 0.0053 1.7 729 mom_12_1 0.0077 2.4 729 ``` The header figure plots the table. The generator plants a 5-session reversal (`ret_1s`) and a 12-month momentum that skips the last one (`mom_12_1`); the other horizons have no effect of their own. Even so: - **Neighbors inherit.** The 2-session return is part of the 5-session one, and the 10-session one contains it entirely: both come out negative and significant (t = −2.4 and −2.9) with no effect of their own. The 1-session one, with less of the week's information, stays at t = −1.0. - **Effects mask each other.** At 126 sessions, the recent reversal and the older momentum offset each other and the IC is zero. At 252, momentum shows up (0.0053, t = 1.7), but contaminated by the last month; removing it, `mom_12_1` rises to 0.0077 (t = 2.4). That is the reason for the classic “12-1” of Jegadeesh and Titman (1993)[^1], who separated momentum from the one-month reversal documented by Jegadeesh (1990)[^2]. - **A significant IC is not a tradable signal.** The best, −0.016, is a little over half of the oracle's ceiling, and the long-short portfolio built with signals like this does not cover the costs of its weekly turnover (data README; lesson 5.1.3). With 22 features and 8 horizons tested, moreover, a $t$ of 2 is not impressive: it is a multiple search (lesson 2.1.8). Here it can be checked against the truth; on real data, it cannot. The second common variant divides the return by the stock's volatility, so that a 5% rise in a quiet stock weighs more than in a volatile one: ```python vol63 = R.rolling(63).std() for h in [5, 21, 252]: bruto = C / C.shift(h) - 1 ajustado = bruto / (vol63 * np.sqrt(h)) print(f"{h:3d} sessions: raw {ic(bruto)['ic']:+.4f}, " f"/vol {ic(ajustado)['ic']:+.4f} " f"(t = {ic(ajustado)['t']:+.1f})") ``` ```output 5 sessions: raw -0.0159, /vol -0.0154 (t = -4.7) 21 sessions: raw -0.0077, /vol -0.0064 (t = -2.0) 252 sessions: raw +0.0053, /vol +0.0066 (t = +2.2) ``` Dividing by volatility barely changes the IC at 5 sessions (−0.0159 against −0.0154, with a somewhat larger $t$) and improves the 252-session one somewhat (from 0.0053 to 0.0066). With ranks within each week, the effect of the normalization is small because the scale no longer matters; what changes is the order between stocks of different volatility. In a model that uses the values (a regression, a network), normalization matters much more: without it, volatile stocks dominate the loss. Lesson 5.2.4 covers how to normalize without using future information. ## Volatility: it anticipates the magnitude, not the direction Realized volatility is the easiest feature to predict and the one most often misunderstood. It is very persistent (volatility clustering, lesson 1.6.2), so the volatility of the last few weeks anticipates the next one well. What it does not anticipate is the sign: ```python for w in [5, 21, 63]: v = R.rolling(w).std() con_abs = ic(v, Y.abs()) con_signo = ic(v) print(f"vol {w:2d} sessions: IC with |r| {con_abs['ic']:.3f} " f"(t = {con_abs['t']:.0f}); with r {con_signo['ic']:+.4f} " f"(t = {con_signo['t']:+.1f})") ``` ```output vol 5 sessions: IC with |r| 0.171 (t = 56); with r +0.0024 (t = +0.5) vol 21 sessions: IC with |r| 0.220 (t = 69); with r +0.0020 (t = +0.4) vol 63 sessions: IC with |r| 0.227 (t = 76); with r +0.0032 (t = +0.6) ``` All three windows rank stocks very well by the **magnitude** of the following week's move: an IC of 0.22 with 63 sessions, about 30 times the oracle's ceiling for direction, with a $t$ of 76. With the signed return, the IC is zero. A risk feature like this is used to normalize others (the previous section), to condition them (an interaction) and to size positions (lesson 4.2.2), not as a direction predictor. The generator plants a small, negative low-volatility effect in `vol_63d`, and it does not show: volatile stocks have more beta and, since the market rises on average, beta offsets the premium (the README recovers it by removing the true beta). It is a good reminder that the univariate IC mixes the effect of a feature with that of everything that resembles it. ## Volume and interactions Volume in shares is not comparable across companies (a 2 USD stock trades many more shares than an 800 USD one). The usual approach is **relative volume**: the volume of the last few days against that of a longer period for the same stock, in logarithms. In the panel, `volumen_rel` = log(mean volume over 5 sessions / over 63 sessions) has no effect of its own, but the generator uses it in an **interaction**: the week's reversal is stronger when the move happened on low volume, $\alpha \propto g(\text{ret\_1s})\,(1 - 0.5\,g(\text{volumen\_rel}))$. It is the economic idea that a move without volume is usually liquidity noise, and one with volume, information. It shows up when the sample is split by terciles: ```python r1s = C / C.shift(5) - 1 vrel = np.log(V.rolling(5).mean() / V.rolling(63).mean()) q = vrel.loc[fechas].rank(axis=1, pct=True) tramos = {"low": (0, 1 / 3), "mid": (1 / 3, 2 / 3), "high": (2 / 3, 1)} serie = {} for nombre, (lo, hi) in tramos.items(): X = r1s.loc[fechas].where((q > lo) & (q <= hi)) serie[nombre] = X.rank(axis=1).corrwith(Y.rank(axis=1), axis=1) r = ic(X) print(f"relative volume {nombre:4s}: IC {r['ic']:+.4f} " f"(t = {r['t']:+.1f})") dif = (serie["low"] - serie["high"]).dropna() t_dif = dif.mean() / dif.std() * np.sqrt(len(dif)) print(f"low - high: {dif.mean():+.4f} (t = {t_dif:+.1f})") centrado = lambda X: X.rank(axis=1, pct=True) - 0.5 inter = centrado(r1s.loc[fechas]) * (1 - centrado(vrel.loc[fechas])) r = ic(inter) print(f"interaction: IC {r['ic']:+.4f} (t = {r['t']:+.1f})") ``` ```output relative volume low : IC -0.0240 (t = -5.1) relative volume mid : IC -0.0162 (t = -3.3) relative volume high: IC -0.0080 (t = -1.9) low - high: -0.0160 (t = -3.4) interaction: IC -0.0170 (t = -4.6) ``` The reversal with low volume is three times the one with high volume (−0.024 against −0.008), and the difference, measured week by week over the same dates, is −0.016 with $t$ = −3.4: it is not noise. The hand-built interaction feature, with centered ranks, gives an IC of −0.017, somewhat better than the 5-session return alone. *Fig. — IC of the 5-session return by tercile of the week's relative volume: the planted reversal is stronger when the move happened on low volume.* Three negative horizontal bars. With low relative volume, highlighted, IC −0.024 (t −5.1); with mid volume, −0.016 (t −3.3); with high volume, −0.008 (t −1.9). The ±1.96 standard-error whiskers of the low and high terciles barely touch. A decision tree or a boosting model (lessons 5.5.2 and 5.5.3) find interactions like this without building them; a linear model does not, unless it is given them. Building them by hand has the advantage that they can be reasoned about and tested one by one, and the drawback that each interaction tried is one more test in the multiple search. ## Normalized indicators Technical indicators (lesson 1.7.1) are almost always built in price units: the distance to a moving average in dollars, a MACD, a Bollinger band. In a panel, those units make the indicator measure mostly **how much the stock is worth**, not what you wanted to measure. And many indicators are, deep down, the same number on another scale: ```python sma50 = C.rolling(50).mean() en_precio = (C - sma50).loc[fechas] en_tanto = (C / sma50 - 1).loc[fechas] en_vol = (en_tanto / (vol63.loc[fechas] * np.sqrt(50))) nivel = C.loc[fechas].rank(axis=1) for nombre, X in [("C - SMA50 (USD)", en_precio), ("C / SMA50 - 1", en_tanto), ("(C / SMA50 - 1) / vol", en_vol)]: rho = X.abs().rank(axis=1).corrwith(nivel, axis=1).mean() print(f"{nombre:22s} corr. of |X| with price: {rho:+.2f}") rsi = fe.pivot(index="fecha", columns="ticker", values="rsi_14") r14 = (C / C.shift(14) - 1).loc[fechas] rho = rsi.rank(axis=1).corrwith(r14.rank(axis=1), axis=1).mean() print(f"RSI 14 against 14-session return: {rho:.2f}") ``` ```output C - SMA50 (USD) corr. of |X| with price: +0.72 C / SMA50 - 1 corr. of |X| with price: +0.01 (C / SMA50 - 1) / vol corr. of |X| with price: +0.00 RSI 14 against 14-session return: 0.95 ``` The distance to the 50-session average in dollars has a rank correlation of 0.72 between its absolute value and the price: in an 800 USD stock, a “large” distance is simply that of an expensive stock. Divided by the average, the relationship disappears (0.01); divided also by the expected volatility over 50 sessions, it becomes a number of standard deviations, comparable across stocks and across dates. The three rules: divide by the price, by the volatility and, for volume, by the stock's own history. The last line is the redundancy. The 14-session RSI and the 14-session return have a mean rank correlation of 0.95: they are almost the same feature. That is why `rsi_14` comes out with a negative IC (−0.008, t = −2.6 in the README) without having an effect of its own: it inherits the reversal. Putting into a model twenty indicators that are transformations of the same return adds no information and does add noise and collinearity (lesson 5.2.5 measures importance by groups of correlated features). ## Microstructure: order imbalance Daily price and volume features summarize what has already happened in the price. Microstructure looks at **who pushed**: whether the volume of each interval was initiated by aggressive buyers or sellers (those who cross the bid-ask spread, lesson 1.1.1). The **order imbalance** of an interval is $$ \mathrm{DI} = \frac{V_{\mathrm{buy}} - V_{\mathrm{sell}}}{V} = \frac{2\,V_{\mathrm{buy}}}{V} - 1 \in [-1, 1]. $$ It is the trade-based version of the order book *order flow imbalance* of lesson 4.3.2. Its practical problem is that the aggressor side is not always in the data: the consolidated tape of US equities does not carry it, and that is why classifiers exist. The **tick rule** assigns each trade to the side of the last price change, and the Lee and Ready (1991)[^3] algorithm combines it with the prevailing quote. With bars, without individual trades, the **bulk volume classification** (BVC) of Easley, López de Prado and O'Hara[^4] splits the volume of each bar according to the standardized price change: $V_{\mathrm{buy}} = V\,\Phi(\Delta p/\sigma_{\Delta p})$. The data are one year of one-minute bars of **FNX**, a fictitious future on an index (247 sessions of 390 minutes), with the **true buy volume** of each minute. The generator alternates two hidden regimes: **trend**, with informed flow that pushes in one direction for an hour on average, and 30-minute **mean reversion**. The buy fraction follows the price in both regimes and, only in trend, also carries a persistent bias toward the direction of the episode (0.5 + 0.12·tanh(r/σ) + 0.03·D + noise). That bias is the footprint of informed flow, and it is not in the price. ```python from scipy.stats import norm b = pd.read_parquet("hf5_barras_1min.parquet") b["sesion"] = b["ts"].dt.date g = b.groupby("sesion") b["r"] = np.log(b["cierre"]).groupby(b["sesion"]).diff() # true side (the aggressor of each trade) b["vc"] = b["volumen_comprador"] # tick rule: the whole minute to the side of the price change signo = np.sign(b["cierre"].groupby(b["sesion"]).diff()) b["vc_tick"] = b["volumen"] * (signo.fillna(0) + 1) / 2 # BVC: buy fraction = Φ(r / σ), σ from the previous minutes sigma = g["r"].transform( lambda s: s.ewm(span=100, min_periods=10).std().shift(1)) frac = pd.Series(norm.cdf(b["r"] / sigma), index=b.index) b["vc_bvc"] = b["volumen"] * frac.fillna(0.5) d = {m: 2 * b[c] / b["volumen"] - 1 for m, c in [("verdad", "vc"), ("tick", "vc_tick"), ("bvc", "vc_bvc")]} d = pd.DataFrame(d)[b["r"].notna() & sigma.notna()] print(len(b), "minutes;", b["sesion"].nunique(), "sessions") d.corr().round(3) ``` ```output 96330 minutes; 247 sessions verdad tick bvc verdad 1.000 0.591 0.658 tick 0.591 1.000 0.861 bvc 0.658 0.861 1.000 ``` Minute by minute, the two classifiers are very similar to each other (0.86) and much less similar to the truth (0.59 for the tick rule and 0.66 for BVC). That is to be expected: both infer the side **from the price move**, so they rebuild the part of the flow that follows the price and lose the rest. The question for a model is whether what they lose matters. The trend regime is what you want to detect, and the 30-minute imbalance is the natural feature: ```python from sklearn.metrics import roc_auc_score verdad = pd.read_parquet("hf5_verdad.parquet") b["regimen"] = verdad["regimen"].to_numpy() # same row, same ts def roll30(col): return g[col].transform(lambda s: s.rolling(30).sum()) vol30 = roll30("volumen") for m, c in [("verdad", "vc"), ("tick", "vc_tick"), ("bvc", "vc_bvc")]: b[f"di_{m}"] = 2 * roll30(c) / vol30 - 1 b["r30"] = g["r"].transform(lambda s: s.rolling(30).sum()) ok = b[["di_verdad", "r30"]].notna().all(axis=1) tend = b.loc[ok, "regimen"] == "tendencia" print(f"minutes in trend: {tend.mean():.1%}") for c in ["r30", "di_verdad", "di_tick", "di_bvc"]: auc = roc_auc_score(tend, b.loc[ok, c].abs()) print(f"AUC of |{c}| for the regime: {auc:.3f}") ``` ```output minutes in trend: 35.1% AUC of |r30| for the regime: 0.603 AUC of |di_verdad| for the regime: 0.682 AUC of |di_tick| for the regime: 0.556 AUC of |di_bvc| for the regime: 0.564 ``` The AUC measures how well the absolute value of each feature separates trend minutes from reversion minutes (0.5 is chance, 1 is perfect). The return of the last 30 minutes already separates them somewhat (0.603), because in trend the price moves more in one direction. The imbalance with the true side separates them more (0.682): it adds the persistent bias that the price does not have. The two classified versions end up **below the return itself** (0.556 and 0.564): they are a noisy version of the price change, and on top of that the classification noise accumulates over 30 minutes. *Fig. — FNX, 1-minute bars: what separates the trend regime (AUC of the absolute value) and what anticipates the next 30 minutes' return out of sample, depending on how the 30-minute imbalance is measured.* Two panels of horizontal bars. Left, AUC for separating the trend regime: 30-minute return 0.603, imbalance with the true side 0.682, tick rule 0.556, BVC 0.564. Right, out-of-sample correlation with the next 30 minutes' return: 30-minute return 0.111, true side 0.182 (highlighted), tick rule 0.049, BVC 0.061. ## Does it anticipate the following 30 minutes? Distinguishing the regime is not yet anticipating the price. The honest test is the one from the 5.4 lessons: one observation every 30 minutes (the 30-minute windows do not overlap), a model fitted on the first half of the sessions and evaluated on the second, and the comparison with the simplest baseline, the return of the previous 30 minutes (intraday momentum). The target is the return of the following 30 minutes divided by the expected volatility: ```python import statsmodels.api as sm b["minuto"] = g.cumcount() b["sigma"] = g["r"].transform( lambda s: s.ewm(span=100, min_periods=10).std()) b["fut30"] = g["r"].transform( lambda s: s[::-1].rolling(30).sum()[::-1].shift(-1)) b["y"] = b["fut30"] / (b["sigma"] * np.sqrt(30)) b["r30z"] = b["r30"] / (b["sigma"] * np.sqrt(30)) cols = ["y", "r30z", "di_verdad", "di_tick", "di_bvc"] m = b[(b["minuto"] % 30 == 0) & b[cols].notna().all(axis=1)].copy() sesiones = np.sort(m["sesion"].unique()) ent = m["sesion"] < sesiones[len(sesiones) // 2] print(f"{ent.sum()} training observations, " f"{(~ent).sum()} test; standard error ≈ " f"{1 / np.sqrt((~ent).sum()):.3f}") for x in [["r30z"], ["di_tick"], ["di_bvc"], ["di_verdad"], ["r30z", "di_verdad"]]: X = sm.add_constant(m[x]) fit = sm.OLS(m.loc[ent, "y"], X[ent]).fit() pred = fit.predict(X[~ent]) c = np.corrcoef(pred, m.loc[~ent, "y"])[0, 1] print(f"{' + '.join(x):18s} out-of-sample corr. {c:+.3f}") ``` ```output 1353 training observations, 1364 test; standard error ≈ 0.027 r30z out-of-sample corr. +0.111 di_tick out-of-sample corr. +0.049 di_bvc out-of-sample corr. +0.061 di_verdad out-of-sample corr. +0.182 r30z + di_verdad out-of-sample corr. +0.182 ``` With 1,364 test observations, the standard error of each correlation is about 0.027. The past return gives 0.111: there is intraday momentum, because part of the time is trend. The two classified imbalances end up below it (0.049 and 0.061): they add nothing to the price they come from. The true imbalance reaches 0.182, and combined with the past return it does not improve: it contains it. The figure in the previous section summarizes the two measures. ## VPIN: imbalance in volume time The VPIN (*volume-synchronized probability of informed trading*) of Easley, López de Prado and O'Hara (2012)[^4] measures the same imbalance in **volume time**: it splits trading into buckets of a fixed volume $V$ and averages the absolute imbalance of the last $n$ buckets, $$ \mathrm{VPIN} = \frac{1}{n}\sum_{\tau=1}^{n} \frac{\lvert V^{\mathrm{buy}}_\tau - V^{\mathrm{sell}}_\tau \rvert}{V}. $$ Measuring in volume buckets makes the indicator advance quickly when there is a lot of activity and slowly when there is little, like the volume bars of lesson 1.4.5. Its authors presented it as a measure of flow toxicity for market makers and as a short-term volatility indicator. ```mermaid flowchart LR B[1-min bars] --> C[Volume buckets] C --> D[Imbalance] D --> M[Mean of 10 buckets] ``` The bucket size is set with earlier data (here, 1/50 of the mean daily volume of the first 20 sessions) and is not recomputed with the whole sample, which would be looking at the future. A minute that crosses the edge of a bucket is split between the two in proportion to its volume: ```python def vpin(vol, vc, cubo, ventana=10): """VPIN per bucket: rolling mean of |2·V_buy − V| / V.""" cum_v = np.cumsum(vol) bordes = np.arange(cubo, cum_v[-1], cubo) # cumulative buy volume at each edge (linear split of the # minute that crosses the edge between the two buckets) cum_c = np.interp(bordes, cum_v, np.cumsum(vc)) comp = np.diff(np.r_[0.0, cum_c]) desequilibrio = np.abs(2 * comp - cubo) / cubo vp = pd.Series(desequilibrio).rolling(ventana).mean() cierre = np.searchsorted(cum_v, bordes) # closing minute return vp.to_numpy(), cierre vol = b["volumen"].to_numpy(float) # bucket set with the first 20 sessions: 1/50 of daily volume cubo = b.groupby("sesion")["volumen"].sum().iloc[:20].mean() / 50 print(f"bucket of {cubo:,.0f} contracts") res = {} for m_, c in [("verdad", "vc"), ("tick", "vc_tick"), ("bvc", "vc_bvc")]: vp, cierre = vpin(vol, b[c].to_numpy(float), cubo) res[m_] = vp res = pd.DataFrame(res) res["regimen"] = b["regimen"].to_numpy()[cierre] res = res.dropna() print(len(res), "buckets; mean VPIN with the true side", round(res["verdad"].mean(), 3)) print(res.groupby("regimen")[["verdad", "tick", "bvc"]] .mean().round(3)) tend = res["regimen"] == "tendencia" for m_ in ["verdad", "tick", "bvc"]: print(f"AUC of the VPIN ({m_}): " f"{roc_auc_score(tend, res[m_]):.3f}") ``` ```output bucket of 23,644 contracts 12687 buckets; mean VPIN with the true side 0.084 verdad tick bvc regimen reversion 0.079 0.311 0.197 tendencia 0.091 0.337 0.210 AUC of the VPIN (verdad): 0.639 AUC of the VPIN (tick): 0.575 AUC of the VPIN (bvc): 0.565 ``` With the true side, the mean VPIN is 0.084: 0.079 in reversion and 0.091 in trend, with an AUC of 0.639 for separating the regimes, somewhat less than the 30-minute imbalance (the data README, with buckets of 1/50 of the generator's nominal daily volume, 22,000 contracts, gives 0.094 and 0.080: the level depends on the bucket size). With the classifiers two things happen. The **level** shoots up: 0.32 with the tick rule and 0.20 with BVC, because each classification error moves volume from one side to the other and, in absolute value, almost never cancels out. And the **separation** falls to 0.575 and 0.565. A VPIN threshold set with one classification method is not valid with another, and a published level cannot be compared with one computed with other buckets. The figure shows a session chosen as an illustration (the mean difference between regimes is much smaller than on this day). In the first trend episode, a drop of almost 80 points in an hour and a half, the VPIN rises to 0.15; in the afternoon reversion it stays between 0.04 and 0.07. *Fig. — FNX, session of 3 September 2024 (chosen as an illustration): price on top and VPIN with the true side below; in grey, the trend episodes of the true regime.* Two panels with the minute of the session on the horizontal axis. Top, the FNX price falls from about 5,480 to about 5,400 during the first trend episode (minutes 1 to 109) and rises during the second (124 to 161). Bottom, the VPIN, highlighted, rises above 0.12 during the first episode and returns to levels of 0.05 to 0.08 in the afternoon reversal; a dashed line marks the year average, 0.084. The conclusions of the original VPIN paper were widely debated. Andersen and Bondarenko[^5] showed with S&P 500 futures data that the VPIN is correlated by construction with volume and volatility, that after controlling for them they find no additional predictive power for future volatility and that its properties depend heavily on the trade classification. For a model, the practical lesson is that of the whole section: a microstructure feature is worth what the aggressor-side data that feeds it is worth, and it is always compared with what the price, the volume and the volatility already say. ## Common mistakes - **Computing the feature with the day's close and trading at that same close.** The feature is known at the close; the order goes to the next session (or to the closing auction, if it is computed a few minutes earlier with prices from that moment). - **Horizons that repeat each other.** Returns of 1, 2, 5 and 10 sessions share almost all of their information; with nested windows, the model spreads one effect across several columns and the importance of each one is misleading. - **Price units in a panel.** A distance in dollars, an unnormalized MACD or a volume in shares rank by the size or the price of the stock, not by what you wanted to measure. - **Confusing risk with direction.** Volatility anticipates the magnitude of the move; if the model predicts signed returns, its high IC with the absolute value says nothing. - **Classifying the aggressor side and believing you have the flow.** The tick rule and BVC rebuild what is already in the price. If the signal depends on the flow, you need the aggressor-side data. - **Windows that cross sessions.** A 30-minute imbalance at 9:35 that includes the end of the previous session mixes two days; in FNX the `rolling` windows are grouped by session (the VPIN, on the other hand, does carry over from one day to the next, as in its original definition). - **Evaluating with overlapping windows.** One observation per minute with a 30-minute target inflates the number of observations about 30 times and makes everything look significant. ## Exercises 1. `dist_max_52s` and `rsi_14` have no effect of their own in the panel, but their univariate IC is not zero. Remove, week by week, the part of each one explained by the 5-session return (rank regression) and compute the IC of what remains. Which features are redundant? **Solution.** ```python def residuo_rangos(X, Z): # rank of X minus its linear projection onto the rank of Z rx, rz = X.rank(axis=1), Z.rank(axis=1) rx = rx.sub(rx.mean(axis=1), axis=0) rz = rz.sub(rz.mean(axis=1), axis=0) beta = (rx * rz).sum(axis=1) / (rz * rz).sum(axis=1) return rx - rz.mul(beta, axis=0) r1 = fe.pivot(index="fecha", columns="ticker", values="ret_1s") for nombre in ["rsi_14", "dist_max_52s"]: X = fe.pivot(index="fecha", columns="ticker", values=nombre) antes, despues = ic(X), ic(residuo_rangos(X, r1)) print(f"{nombre:13s} IC {antes['ic']:+.4f} (t = {antes['t']:+.1f})" f" -> without ret_1s {despues['ic']:+.4f} " f"(t = {despues['t']:+.1f})") ``` ```output rsi_14 IC -0.0083 (t = -2.6) -> without ret_1s -0.0005 (t = -0.2) dist_max_52s IC -0.0031 (t = -1.0) -> without ret_1s +0.0019 (t = +0.7) ``` The RSI loses all of its IC when the 5-session reversal is removed (from −0.0083 to −0.0005): it was the same feature on another scale. The distance to the 52-week high was not significant before (t = −1.0) and is not after either (t = 0.7): the small change of sign is noise. Both are redundant with the returns. On real data you do not have the truth to confirm it, but the procedure is the same: measure the **incremental** IC over the features that are already in the model, not the univariate one. 2. Repeat the VPIN with the true side with buckets twice as large (1/25 of daily volume) and with a window of 50 buckets. How do the mean level and the regime AUC change? Which parameters would you choose, and how, without looking at the test sample? **Solution.** ```python vc = b["vc"].to_numpy(float) for factor, ventana in [(1, 10), (2, 10), (1, 50), (2, 50)]: vp, cierre = vpin(vol, vc, cubo * factor, ventana) t = pd.DataFrame({"vp": vp, "reg": b["regimen"].to_numpy()[cierre]}) t = t.dropna() auc = roc_auc_score(t["reg"] == "tendencia", t["vp"]) print(f"bucket x{factor}, window {ventana:2d}: mean " f"{t['vp'].mean():.3f}, AUC {auc:.3f}") ``` ```output bucket x1, window 10: mean 0.084, AUC 0.639 bucket x2, window 10: mean 0.064, AUC 0.613 bucket x1, window 50: mean 0.084, AUC 0.556 bucket x2, window 50: mean 0.064, AUC 0.543 ``` With buckets twice as large, each bucket aggregates more volume and the mean absolute imbalance falls (from 0.084 to 0.064): the level of the VPIN depends on the bucket, not only on the market. With a window of 50 buckets (a few whole sessions) the indicator is smoothed so much that it stops following one-hour episodes and the AUC falls to 0.556 and 0.543. The parameters are chosen with the horizon of the phenomenon you want to detect (here, one-hour episodes: windows of a few hours at most) and with validation on the first half of the data, never with the AUC of the second. Each combination tried counts in the multiple search. 3. Your data vendor for a future gives you one-minute OHLCV bars, without the aggressor side. A colleague proposes building the imbalance with BVC and training a model with it. What would you tell them, and what data would you ask for? **Solution.** That BVC only splits the volume according to the bar's price change: in FNX, the imbalance built this way separates the regime worse than the 30-minute return itself (AUC 0.564 against 0.603) and anticipates the price less (0.061 against 0.111). Before adding it you have to compare it with the price features it comes from; if it does not improve on them out of sample, it is superfluous. If the hypothesis is that the flow contains information that the price does not have yet, you need each trade's data with its aggressor side (many futures markets disseminate it in their trade feed) or, at the very least, the trades with the prevailing quote to apply Lee and Ready. And, in both cases, check the timestamp of each trade against the close of the bar. [^1]: N. Jegadeesh and S. Titman, “Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency”, *Journal of Finance* 48(1), 1993, pp. 65-91, [doi:10.1111/j.1540-6261.1993.tb04702.x](https://doi.org/10.1111/j.1540-6261.1993.tb04702.x). [^2]: N. Jegadeesh, “Evidence of Predictable Behavior of Security Returns”, *Journal of Finance* 45(3), 1990, pp. 881-898, [doi:10.1111/j.1540-6261.1990.tb05110.x](https://doi.org/10.1111/j.1540-6261.1990.tb05110.x). [^3]: C. M. C. Lee and M. J. Ready, “Inferring Trade Direction from Intraday Data”, *Journal of Finance* 46(2), 1991, pp. 733-746, [doi:10.1111/j.1540-6261.1991.tb02683.x](https://doi.org/10.1111/j.1540-6261.1991.tb02683.x). [^4]: D. Easley, M. López de Prado and M. O'Hara, “Flow Toxicity and Liquidity in a High-frequency World”, *Review of Financial Studies* 25(5), 2012, pp. 1457-1493, [doi:10.1093/rfs/hhs053](https://doi.org/10.1093/rfs/hhs053) (VPIN and bulk volume classification); by the same authors, “Discerning information from trade data”, *Journal of Financial Economics* 120(2), 2016, pp. 269-285, [doi:10.1016/j.jfineco.2016.01.018](https://doi.org/10.1016/j.jfineco.2016.01.018). [^5]: T. G. Andersen and O. Bondarenko, “VPIN and the flash crash”, *Journal of Financial Markets* 17, 2014, pp. 1-46, [doi:10.1016/j.finmar.2013.05.005](https://doi.org/10.1016/j.finmar.2013.05.005), and “Reflecting on the VPIN dispute”, *Journal of Financial Markets* 17, 2014, pp. 53-64, [doi:10.1016/j.finmar.2013.08.002](https://doi.org/10.1016/j.finmar.2013.08.002) (abstract accessed on October 23, 2026). ## Self-assessment 1. In the panel, the 5-session return has an IC of −0.016 and the 10-session one, −0.011, although only the 5-session one has a planted effect. Why does the 10-session one also come out negative and significant? - A. Because the 10-session return contains the return of the last 5: it inherits their effect even though it has none of its own - B. Because the generator also plants a 10-session reversal - C. Because the rank IC is biased toward negative values - D. Because the labels overlap *Answer A: Because the 10-session return contains the return of the last 5: it inherits their effect even though it has none of its own.* A 10-session return is the return of the last 5 plus that of the 5 before them. The recent part carries the planted reversal; the older part only adds noise, which is why the IC falls but does not disappear. A univariate IC does not distinguish a feature's own effect from an inherited one: you need a model with both features at once or to know the truth, as here. 2. The 63-session volatility has an IC of 0.23 with the absolute value of the following week's return and of 0.003 with the signed return. What use does it have in a direction model? - A. None: if it does not predict direction it has to be removed - B. It is the best direction feature in the panel - C. Scaling or conditioning other features (a reversal weighs differently in a quiet stock than in a volatile one) and sizing positions; on its own it does not rank by expected return - D. Replacing the label *Answer C: Scaling or conditioning other features (a reversal weighs differently in a quiet stock than in a volatile one) and sizing positions; on its own it does not rank by expected return.* Volatility is very persistent and anticipates the magnitude of moves, not their sign. It is used to normalize (return divided by volatility), for interactions and for position sizing. In the panel, moreover, it has a small negative planted effect that beta hides in the raw return (data README). 3. With the one-minute bars of FNX, the 30-minute imbalance computed with the true buy volume distinguishes the trend regime with an AUC of 0.68; rebuilt with the tick rule, with 0.56, less than the 30-minute return itself (0.60). Why such a difference? - A. Because the tick rule has a programming error - B. Because the tick rule infers the side from the price move, so it only recovers the part of the flow that is already in the price; the part that cannot be seen in it (the persistent bias of informed flow) is lost - C. Because the AUC cannot be compared across features - D. Because the true buy volume has leakage *Answer B: Because the tick rule infers the side from the price move, so it only recovers the part of the flow that is already in the price; the part that cannot be seen in it (the persistent bias of informed flow) is lost.* In the data, the buy fraction is 0.5 + 0.12·tanh(r/σ) + 0.03·D in trend + noise. A classifier that looks at the price can only rebuild the first term. The 0.03·D term, the footprint of informed flow, is only in the aggressor side of each trade. 4. The VPIN of FNX computed with BVC comes out around 0.20 and with the true side around 0.08. What do you conclude? - A. That the market is more toxic than it looks - B. That BVC is more accurate - C. That the VPIN does not depend on the bucket size - D. That the level of the VPIN depends on the classification method: the classification noise adds up as apparent imbalance, and a threshold set with one method is not valid with another *Answer D: That the level of the VPIN depends on the classification method: the classification noise adds up as apparent imbalance, and a threshold set with one method is not valid with another.* Each volume bucket measures |V_buy − V_sell|/V. A classification error shifts volume from one side to the other and almost never cancels out in absolute value: the measured VPIN goes up. That is why published VPIN levels cannot be compared if the classification or the bucket size changes. 5. To evaluate whether the 30-minute imbalance anticipates the return of the following 30 minutes, why take one observation every 30 minutes and not one per minute? - A. Because with one per minute the windows overlap: 30 consecutive observations share almost all of the future and the standard error computed as if they were independent would be about five times too small - B. To make the computation faster - C. Because one-minute bars have no buy volume - D. Because the correlation does not accept more than 3,000 points *Answer A: Because with one per minute the windows overlap: 30 consecutive observations share almost all of the future and the standard error computed as if they were independent would be about five times too small.* With overlapping windows, the mean correlation can be estimated, but the effective number of observations is about 30 times smaller. Sampling without overlap gives an honest standard error of 1/√n (lessons 5.3.4 and 5.4.1). ## Key points - A feature is a function of the past known at the close of its date. Recomputing it from prices and comparing it with the published one is the cheapest way to find definition and alignment errors. - In the panel, the planted 5-session reversal gives the strongest IC among the returns (−0.016, t = −4.3); the neighboring horizons inherit part of its effect. The 12-month momentum skips the last month so that it does not mix with the reversal. - Volatility anticipates the magnitude (IC of 0.23 with the absolute return) and not the direction: it is used to normalize, condition and size. - Interactions matter: the reversal is three times stronger with low relative volume (IC −0.024) than with high volume (−0.008). - Indicators in price units are not comparable across stocks: they are normalized by price and by volatility. Many classic indicators are almost the same feature (the 14-session RSI and the 14-session return have a rank correlation of 0.95). - The order imbalance with the true aggressor side carries information that the price does not have: in FNX it distinguishes the regime (AUC 0.68) and anticipates the following 30 minutes (out-of-sample correlation of 0.18, against 0.11 for the past return). Classified from the bars, it loses almost all of that; the VPIN, moreover, changes level with the method. ## Resources ### Research - [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min - [Limit Order Book - Aggregated Market Liquidity](https://pythonparatrading.com/en/limit-order-book-and-market-liquidity/) · Research-0008 · 11 min ### Documentation - [Easley, López de Prado and O'Hara (2012): Flow Toxicity and Liquidity in a High-frequency World, Review of Financial Studies 25(5)](https://doi.org/10.1093/rfs/hhs053) · doi.org - [Andersen and Bondarenko (2014): VPIN and the flash crash, Journal of Financial Markets 17](https://doi.org/10.1016/j.finmar.2013.05.005) · doi.org - [Easley, López de Prado and O'Hara (2016): Discerning information from trade data, Journal of Financial Economics 120(2)](https://doi.org/10.1016/j.jfineco.2016.01.018) · doi.org - [Lee and Ready (1991): Inferring Trade Direction from Intraday Data, Journal of Finance 46(2)](https://doi.org/10.1111/j.1540-6261.1991.tb02683.x) · doi.org ### Data - [Level 5 data: daily prices of the 400-stock panel (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_precios.parquet) · Download · panel5_precios.parquet - [Level 5 data: one-minute bars of FNX with buy volume (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/hf5_barras_1min.parquet) · Download · hf5_barras_1min.parquet Previous lesson: [5.1.3 · Economic metrics for trading models: beyond accuracy](https://pythonparatrading.com/en/python-trading-course/level-5/economic-metrics-for-trading-models/) Next lesson: [5.2.2 · Point-in-time fundamentals, macro and alternative data](https://pythonparatrading.com/en/python-trading-course/level-5/point-in-time-fundamentals-macro-alternative-data/)