Lesson 5.2.3 · 60 min
Fractional differentiation for features
Fractional differentiation promises the best of both worlds: a stationary series that keeps the memory of the price level. This lesson uses it as a model feature and measures what lies behind the promise: which part of the memory is past returns and which part is level, why passing the ADF in training does not keep the test from falling out of range, how the level drags its defects along and how much it adds to a LightGBM against a well-chosen set of returns.
- Module
- 5.2 · Feature engineering
- Duration
- 60 min
- Prerequisites
- 1.4.3 · Adjusting for splits and dividends: adjusted and unadjusted prices, 2.1.2 · Stationarity: ADF, KPSS, differencing and fractional differencing, 5.2.1 · Price, volume and microstructure features
- Author
- Jesús Cuesta
- Updated
Level 5 lessons
5.1 · ML fundamentals applied to finance
5.2 · Feature engineering
5.3 · Labelling and sampling
5.4 · Validation for financial series
5.5 · Classical models
- 5.5.1 Regularized regressions: Ridge, Lasso, Elastic Net and logistic
- 5.5.2 Decision trees, Random Forest and bagging
- 5.5.3 Gradient boosting: XGBoost, LightGBM and CatBoost
- 5.5.4 Optuna and the risk of overfitting the validation
- 5.5.5 Unsupervised learning: PCA, clustering and autoencoders
- 5.5.6 Learning to rank for stock selection
5.6 · Deep learning with Keras 3
5.7 · Deep learning with PyTorch
- 5.7.1 PyTorch: tensors, autograd and the training loop
- 5.7.2 Time series architectures: TCN, N-BEATS, PatchTST and iTransformer
- 5.7.3 Probabilistic models: quantile loss, DeepAR and intervals
- 5.7.4 Graph neural networks for asset relationships
- 5.7.5 End-to-end portfolio optimization: differentiable layers and Sharpe loss
5.8 · Time series foundation models
- 5.8.1 Time series foundation models: pretrain once, forecast without training
- 5.8.2 Foundation model families: architecture, data and license
- 5.8.3 In practice: context, quantiles, covariates and fine-tuning
- 5.8.4 Foundation models in finance: what the evidence says
- 5.8.5 Lab: foundation models vs GARCH, HAR and LightGBM
5.9 · Reinforcement learning
5.10 · Interpretability, robustness and model risk
What you already know and what is missing
Lesson 2.1.2 explained fractional differentiation: the operator with real , its weights , the fixed-width window version (FFD) that cuts the weights below a threshold, and the sweep over that looks for the smallest one that passes the ADF while keeping the correlation with the level. There the goal was to have a stationary regressor. Here the question is a different one, the one that matters for a machine learning model: is an FFD a better feature than returns?
López de Prado (2018)1Note 1 · M. López de Prado, Advances in Financial Machine Learning, Wiley, 2018, chapter 5 (“Fractionally Differentiated Features”); the fractional operator comes from J. R. M. Hosking, “Fractional differencing”, Biometrika 68(1), 1981, pp. 165-176, doi:10.1093/biomet/68.1.165. proposes it precisely for that: features made stationary by integer differencing lose all their memory, and the FFD keeps part of it. To measure that promise you need to know exactly what the memory it keeps is. The functions from lesson 2.1.2 are rewritten and one calculation is added:
import numpy as np
import pandas as pd
def pesos(d, umbral=1e-4, kmax=100_000):
"""w_0 = 1, w_k = -w_(k-1)·(d - k + 1)/k until |w_k| < threshold."""
w = [1.0]
for k in range(1, kmax):
wk = -w[-1] * (d - k + 1) / k
if abs(wk) < umbral:
break
w.append(wk)
return np.array(w)
def ffd(x, d, umbral=1e-4):
"""Fixed-window fractional difference (no future data)."""
w = pesos(d, umbral)
v = np.convolve(x.to_numpy(), w, mode="valid")
return pd.Series(v, index=x.index[len(w) - 1:])
for d in [0.1, 0.3, 0.5, 0.7]:
w = pesos(d)
c = np.cumsum(w) # partial sums: weight of each return
print(f"d={d:.1f}: {len(w):3d} weights; returns from "
f"1, 5, 21 and 63 sessions ago {np.round(c[[1, 5, 21, 63]], 3)}; "
f"level remainder {c[-1]:.3f}")d=0.1: 503 weights; returns from 1, 5, 21 and 63 sessions ago [0.9 0.79 0.689 0.618]; level remainder 0.502
d=0.3: 388 weights; returns from 1, 5, 21 and 63 sessions ago [0.7 0.466 0.308 0.222]; level remainder 0.129
d=0.5: 200 weights; returns from 1, 5, 21 and 63 sessions ago [0.5 0.246 0.122 0.071]; level remainder 0.040
d=0.7: 97 weights; returns from 1, 5, 21 and 63 sessions ago [0.3 0.106 0.039 0.018]; level remainder 0.014Grouping the terms of the sum, an FFD with weights can be written as past returns plus a level remainder:
The weight of the return from sessions ago is the partial sum , which starts at 1 (today's) and decays like a power, : with , the return from 21 sessions ago still weighs 0.31 and the one from 63 sessions ago, 0.22; with , 0.04 and 0.02. The last partial sum, which is the sum of all the weights, is the weight left to the level of the series at the start of the window: 0.50 with , 0.13 with 0.3, 0.04 with 0.5. With an infinite window it would be zero for any ; the fixed window leaves it there.
So the “memory” of an FFD has two parts of a different nature. One is an average of returns with slowly decaying weights, something that can be approximated with returns over several horizons (lesson 5.2.1). The other is the level from sessions ago, and that one is new: no set of returns contains it. The figure plots the first part; the following sections measure both.
Stationary in the ADF, out of range in the test
The criterion of lesson 2.1.2 chooses the smallest that passes the ADF. The ADF tests whether the series returns
to its mean within the sample. A model needs something else: that the values it will see in production resemble
the ones it saw in training. With the vol5 index (log of the price since 2008, built by accumulating returns
forward: point-in-time), training on 2008-2017 and testing on 2018-2025, we measure for each the training ADF, the
fraction of test values outside the training range and the memory (correlation with the log of the price in the test):
from statsmodels.tsa.stattools import adfuller
v5 = pd.read_parquet("vol5_diario.parquet")
r_ind = v5[v5["activo"] == "indice"].set_index("fecha")["rendimiento"]
lp = np.log1p(r_ind).cumsum() # log of the price, 0 on the first day
filas = []
for d in np.round(np.arange(0, 1.01, 0.05), 2):
x = ffd(lp, d) if d > 0 else lp
ent, prueba = x.loc[:"2017-12-31"], x.loc["2018-01-01":]
adf = adfuller(ent, maxlag=1, regression="c", autolag=None,
result_object=True)
fuera = ((prueba < ent.min()) | (prueba > ent.max())).mean()
memoria = np.corrcoef(prueba, lp.loc[prueba.index])[0, 1]
filas.append({"d": d, "adf": adf.statistic, "fuera": fuera,
"memoria": memoria})
t = pd.DataFrame(filas).set_index("d")
critico = adf.critical_values["5%"]
d_adf = t.index[t["adf"] < critico].min()
d_rango = t.index[t["fuera"] < 0.05].min()
print(f"5% critical value: {critico:.2f}; minimum d by the "
f"ADF: {d_adf}; by the range (< 5% outside): {d_rango}")
t.loc[[0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.7, 1.0]].round(3)5% critical value: -2.86; minimum d by the ADF: 0.1; by the range (< 5% outside): 0.5
adf fuera memoria
d
0.00 -2.630 0.879 1.000
0.05 -2.506 0.832 0.999
0.10 -3.328 0.769 0.996
0.20 -5.110 0.656 0.975
0.30 -7.499 0.546 0.928
0.40 -10.968 0.321 0.847
0.50 -15.390 0.033 0.725
0.70 -25.172 0.000 0.409
1.00 -36.912 0.000 0.018The header figure plots the table. With the FFD already passes the ADF (−3.33 against −2.86) and keeps a correlation of 0.996 with the price, but 77% of its 2018-2025 values are outside the 2008-2017 range: the index rose and the FFD, with half its weight on the level, rose with it. For less than 5% of the test to fall outside, you need , and then the memory drops to 0.725. Between the two there is an interval of in which the series is “stationary” according to the test and is not for a model.
The ADF is not wrong: with ten years of data and a level remainder that changes slowly, the series mean-reverts enough within the sample to reject the unit root. The problem is that the level of the test is not in the sample. What a model does with that depends on the model; a tree does it in a very visible way:
import lightgbm as lgb
y21 = lp.shift(-21) - lp # log return of the next 21
def arbol(d):
x = (ffd(lp, d) if d > 0 else lp).rename("x")
df = pd.concat([x, y21.rename("y")], axis=1, sort=True).dropna()
ent = df.loc[:"2017-12-31"]
prueba = df.loc["2018-01-01":"2025-10-31"]
m = lgb.LGBMRegressor(n_estimators=100, learning_rate=0.05,
num_leaves=8, min_child_samples=100,
random_state=0, n_jobs=4, deterministic=True,
force_row_wise=True, verbose=-1)
m.fit(ent[["x"]], ent["y"])
pred = m.predict(prueba[["x"]])
fuera = prueba["x"] > ent["x"].max()
return fuera.mean(), np.unique(pred[fuera.to_numpy()]).size
for d in [0.0, 0.1, 0.5]:
f, n = arbol(d)
print(f"d={d:.2f}: {f:.0%} of the test above the training "
f"maximum; distinct predictions there: {n}")d=0.00: 88% of the test above the training maximum; distinct predictions there: 1
d=0.10: 76% of the test above the training maximum; distinct predictions there: 1
d=0.50: 3% of the test above the training maximum; distinct predictions there: 1A tree splits the feature axis at thresholds learned from the training data. Everything that falls above the last
threshold receives the prediction of the last leaf: with the log of the price, 88% of the test receives a single
prediction; with , 76%. The model does not extrapolate, it stays at the edge. With only 3% falls
there. A linear regression would do the opposite, extrapolate without limit, which is no better. The target of this
cell (the return of the following 21 sessions) is unpredictable in vol5 by construction (data README): the cell does
not measure whether the model is right, but how it treats what it has not seen.
The memory of the level inherits its defects
The level remainder is the part of the FFD that cannot be built with returns. If the level carries information, the FFD has it; if the level carries a defect, so does the FFD. The stock panel lets you see this in two versions of the same price: the published one, which accumulates returns forward from an initial price and is point-in-time, and the same series normalized by its last price (), which is what someone does when dividing each series by its last value to compare them, and what, in essence, a backward adjustment does. For each one the full FFD and the FFD without the level remainder are computed, with their IC against the following week's return:
from scipy.signal import lfilter
px = pd.read_parquet("panel5_precios.parquet")
LP = np.log(px.pivot(index="fecha", columns="ticker", values="cierre"))
fe = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
fechas = pd.DatetimeIndex(fe["fecha"].unique())
Y = et.pivot(index="fecha", columns="ticker",
values="ret_sig_semana").loc[fechas]
def ic(X):
"""Weekly rank IC with the following week: mean and t."""
X = X.reindex(index=fechas, columns=Y.columns)
v = X.rank(axis=1).corrwith(Y.rank(axis=1), axis=1).dropna()
t = v.mean() / v.std() * np.sqrt(len(v))
return round(float(v.mean()), 4), round(float(t), 1)
def ffd_panel(L, d):
"""FFD by columns: full and without the level remainder."""
w = pesos(d)
completa = pd.DataFrame(lfilter(w, [1.0], L.to_numpy(), axis=0),
index=L.index, columns=L.columns)
completa.iloc[:len(w) - 1] = np.nan
return completa, completa - w.sum() * L.shift(len(w) - 1)
LN = LP - LP.iloc[-1] # normalized: 0 on the last day of the sample
for nombre, L in [("published", LP), ("normalized", LN)]:
print(f"{nombre}: log of the price, IC, t = {ic(L)}")
for d in [0.3, 0.5, 0.7, 1.0]:
completa, rendim = ffd_panel(L, d)
print(f" d={d:.1f}: full {ic(completa)}, "
f"returns only {ic(rendim)}")published: log of the price, IC, t = (-0.0002, -0.1)
d=0.3: full (-0.0008, -0.3), returns only (-0.0014, -0.4)
d=0.5: full (-0.0039, -1.4), returns only (-0.0058, -1.6)
d=0.7: full (-0.0061, -1.9), returns only (-0.0086, -2.3)
d=1.0: full (-0.004, -1.0), returns only (-0.004, -1.0)
normalized: log of the price, IC, t = (-0.0661, -15.5)
d=0.3: full (-0.0553, -15.0), returns only (-0.0014, -0.4)
d=0.5: full (-0.0455, -12.7), returns only (-0.0058, -1.6)
d=0.7: full (-0.032, -8.7), returns only (-0.0086, -2.3)
d=1.0: full (-0.004, -1.0), returns only (-0.004, -1.0)With the published price, the level says nothing about the following week (IC −0.0002) and the full FFD gives almost the same as its returns part: small, negative ICs (−0.009 at most, with ), the planted weekly reversal seen through an average of returns. The level remainder adds no information, but it does no harm either.
With the price normalized at the end, the log of the price moves to an IC of −0.066 with = −15.5: more than twice the ceiling of the README oracle (0.029), which knows the true expected return. Out of sample, no feature can beat the oracle, so this is a leak. The full FFD inherits it (−0.055 with , −0.046 with 0.5, −0.032 with 0.7) even though its level remainder weighs 0.13, 0.04 or 0.01, because the level varies much more across stocks than a sum of returns. The returns part is identical in the two versions: normalizing subtracts a constant per stock, which cancels out in the returns.
Where does the leak come from? Subtracting the log of the last price makes today's level exactly minus the return from today to the end of the sample:
hasta_el_final = LP.iloc[-1] - LP # log return from t to 2025-12-31
for f in ["2013-06-28", "2016-06-30", "2019-06-28", "2022-06-30"]:
pub = np.corrcoef(LP.loc[f], hasta_el_final.loc[f])[0, 1]
nor = np.corrcoef(LN.loc[f], hasta_el_final.loc[f])[0, 1]
print(f"{f}: corr. with the return to the end: "
f"published {pub:+.2f}, normalized {nor:+.2f}")2013-06-28: corr. with the return to the end: published +0.03, normalized -1.00
2016-06-30: corr. with the return to the end: published -0.02, normalized -1.00
2019-06-28: corr. with the return to the end: published +0.01, normalized -1.00
2022-06-30: corr. with the return to the end: published -0.02, normalized -1.00With the published price, the cross-sectional correlation between the level and the return up to 2025 is nil (between −0.02 and +0.03); with the normalized one it is −1 by construction. The following week is part of that future return, and that is where the impossible IC comes from. It is not a laboratory example: the first version of these synthetic data rescaled each series so that its last price fell in a plausible range, had exactly this defect and was caught this way, with a level feature that beat the oracle.
On real data the same problem appears in a milder form. A price backward-adjusted for dividends and splits (lesson
1.4.3) multiplies the past by factors that depend on later dividends and splits; a continuous futures series with
proportional adjustment (lesson 1.4.6), by the jumps of later rolls; a series normalized to 100 on its last day, by
the whole future return. The returns of all of them are point-in-time; the level is not. Any feature that keeps memory
of the level (an FFD, a distance to an all-time high in dollars, the price itself as a column of a tree) has to be
built on a level that was known on each date: unadjusted prices, or returns accumulated forward from a fixed day, like
the panel's published price or the lp of the vol5 index. And the IC of any new feature is compared with the
oracle's ceiling or, on real data, with what is plausible.
Does it add anything in a model? FNX with LightGBM
The main question remains: with a point-in-time level and a nonlinear model, does the FFD add anything over returns? The FNX data (lesson 5.2.1) have a structure that favors a feature with memory: in the reversion regime, the price returns toward its 30-minute exponential average, that is, toward a weighted average of its own past, which is what the returns part of an FFD measures. The level within the year, on the other hand, says nothing about the next minute.
The setup: features computed within each session (the window does not cross the night: threshold , windows of 26 to 66 minutes), the target is the log return of the following 10 minutes, training on the first half of the sessions and testing on the second, with one observation every 10 minutes so that the targets do not overlap. Four sets are compared: five returns (1, 5, 10, 30 and 60 minutes), three full FFDs ( = 0.3, 0.5 and 0.7), the same without the level remainder, and returns plus full FFDs; each one alone and with the 30-minute order imbalance of lesson 5.2.1.
b = pd.read_parquet("hf5_barras_1min.parquet")
n_ses = b["ts"].dt.date.nunique()
P = np.log(b["cierre"].to_numpy()).reshape(n_ses, 390) # session × min
VC = b["volumen_comprador"].to_numpy().reshape(n_ses, 390)
VT = b["volumen"].to_numpy().reshape(n_ses, 390)
def por_sesion(w):
"""Filter with the weights w within each session (no crossing)."""
X = lfilter(w, [1.0], P, axis=1)
X[:, :len(w) - 1] = np.nan
return X
def retardo(A, k):
out = np.full_like(A, np.nan, dtype=float)
out[:, k:] = A[:, :-k]
return out
H = 10
feats = {}
for k in [1, 5, 10, 30, 60]:
feats[f"r{k}"] = P - retardo(P, k)
for d in [0.3, 0.5, 0.7]:
w = pesos(d, umbral=1e-3) # windows of 26 to 66 minutes
completa = por_sesion(w)
feats[f"ffd{d}"] = completa
feats[f"ffdr{d}"] = completa - w.sum() * retardo(P, len(w) - 1)
cv, ct = np.cumsum(VC, axis=1), np.cumsum(VT, axis=1)
feats["di30"] = 2 * (cv - retardo(cv, 30)) / (ct - retardo(ct, 30)) - 1
y = np.full_like(P, np.nan)
y[:, :-H] = P[:, H:] - P[:, :-H] # 10-minute log return
X = pd.DataFrame({k: v.reshape(-1) for k, v in feats.items()})
y = y.reshape(-1)
sesion = np.repeat(np.arange(n_ses), 390)
minuto = np.tile(np.arange(390), n_ses)
ok = X.notna().all(axis=1).to_numpy() & np.isfinite(y)
ent = ok & (sesion < n_ses // 2)
prueba = ok & (sesion >= n_ses // 2) & (minuto % H == 0)
et_ = 1 / np.sqrt(prueba.sum())
print(f"{ent.sum()} training rows; {prueba.sum()} test rows "
f"without overlap; standard error ≈ {et_:.3f}")38745 training rows; 3844 test rows without overlap; standard error ≈ 0.016With 3,844 test observations, the standard error of a correlation is about 0.016.
conjuntos = {
"returns": ["r1", "r5", "r10", "r30", "r60"],
"full ffd": ["ffd0.3", "ffd0.5", "ffd0.7"],
"ffd without level": ["ffdr0.3", "ffdr0.5", "ffdr0.7"],
"returns + full ffd": ["r1", "r5", "r10", "r30", "r60",
"ffd0.3", "ffd0.5", "ffd0.7"],
}
res = {}
for nombre, cols in conjuntos.items():
for extra in [[], ["di30"]]:
m = lgb.LGBMRegressor(
n_estimators=200, learning_rate=0.03, num_leaves=15,
min_child_samples=200, subsample=0.5, subsample_freq=1,
random_state=0, n_jobs=4, deterministic=True,
force_row_wise=True, verbose=-1)
m.fit(X.loc[ent, cols + extra], y[ent])
pred = m.predict(X.loc[prueba, cols + extra])
etiqueta = "with di30" if extra else "alone"
res[(nombre, etiqueta)] = np.corrcoef(pred, y[prueba])[0, 1]
pd.Series(res).unstack().round(3) alone with di30
ffd without level 0.069 0.102
full ffd -0.012 0.039
returns 0.060 0.094
returns + full ffd 0.032 0.074Three conclusions, in order of importance:
- The level remainder does harm. The full FFDs give −0.012, worse than nothing, and added to the returns they make them worse, from 0.060 to 0.032: the tree splits on the level of FNX, which has no relationship with the target, and learns noise from the training data.
- The useful part of the FFD is an average of returns. Without the remainder, the three FFDs give 0.069, the same as the five returns (0.060) within the standard error. The FFD is a compact way of giving the model an average of returns with weights that decay like a power; a set of returns over several horizons contains almost the same information.
- What improves the model is new information. The order imbalance lifts every set by some 0.03-0.05 (0.094 with returns, 0.102 with FFD without level). Transforming the same prices in another way moves the result much less than adding a piece of data that the price does not have.
Outside these data, the result will depend on the asset, the horizon and the model: the FFD is worth testing as one more feature, with the same validation as any other, and not as an automatic substitute for returns.
When it is not needed: a series that is already stationary
The other extreme: a series that is already stationary and has long memory. Realized variance is the typical case;
its logarithm is persistent but returns to its mean, and Corsi's HAR model2Note 2 · F. Corsi, “A Simple Approximate Long-Memory Model of Realized Volatility”, Journal of Financial Econometrics
7(2), 2009, pp. 174-196, doi:10.1093/jjfinec/nbp001. approximates that long memory with three
averages (1, 5 and 22 days). In vol5, the HAR is the true model. Fractionally differencing the log of the realized
variance to predict the next day's:
from sklearn.linear_model import LinearRegression
def r2_fuera(X, y):
df = pd.concat([X, y.rename("y")], axis=1, sort=True).dropna()
a = df.loc["2009-01-01":"2017-12-31"]
b_ = df.loc["2018-01-01":]
m = LinearRegression().fit(a.drop(columns="y"), a["y"])
e = b_["y"] - m.predict(b_.drop(columns="y"))
return 1 - (e ** 2).sum() / ((b_["y"] - a["y"].mean()) ** 2).sum()
filas = {}
for activo, g in v5.groupby("activo"):
lrv = np.log(g.set_index("fecha")["rv_5min"])
obj = lrv.shift(-1)
fila = {f"d={d}": r2_fuera((ffd(lrv, d, 1e-3) if d else lrv)
.to_frame("x"), obj)
for d in [0, 0.2, 0.5, 1.0]}
har = pd.concat({"d1": lrv, "d5": lrv.rolling(5).mean(),
"d22": lrv.rolling(22).mean()}, axis=1)
fila["HAR"] = r2_fuera(har, obj)
filas[activo] = fila
pd.DataFrame(filas).T.round(3) d=0 d=0.2 d=0.5 d=1.0 HAR
accion 0.275 0.117 0.025 0.003 0.393
bono 0.443 0.217 0.052 0.009 0.543
cripto 0.339 0.125 0.022 0.003 0.488
divisa 0.375 0.186 0.044 0.004 0.466
indice 0.505 0.293 0.082 0.014 0.568
oro 0.480 0.235 0.054 0.007 0.580Each step of toward 1 destroys predictive power: in the index, the out-of-sample goes from 0.505 with the log of the variance itself () to 0.293 with , 0.082 with 0.5 and 0.014 with 1. The memory was the information, and the series did not need to be “more stationary”. The HAR, with three averages of the undifferenced series, reaches 0.568. It is the rule of lesson 2.1.2 (do not difference what is already stationary) seen through a model.
The lesson's decisions, in a diagram:
Mermaid source
flowchart TB
S{Stationary?} -- yes --> N[Do not difference]
S -- no --> O{Does the level matter?}
O -- no --> R[Returns]
O -- yes --> P{PIT level?}
P -- no --> R
P -- yes --> F[FFD with fixed d]Common mistakes
- Choosing with the whole sample. The sweep over is one more fit: it is done on the training data and kept fixed in the test.
- Trusting the ADF as the criterion for a model. It passes with small values of that leave most of the test outside the training range. You also have to measure the stability of the distribution.
- Applying the FFD to a level that is not point-in-time. Backward-adjusted prices, continuous series with proportional adjustment or series normalized by their last value carry the future in the level, and the FFD keeps it.
- Windows that cross sessions or gaps. With intraday data, a window of hundreds of minutes mixes days and drags in the opening jump; it is computed by session with a higher threshold or with a fixed window.
- Differencing what is already stationary. The realized log-variance, a cointegrated spread or an interest rate in a bounded range lose information when differenced.
- Not comparing with a baseline. An FFD only deserves to enter the model if it beats a simple set of returns, out of sample and beyond its standard error.
Exercises
- Check with the
vol5control that an FFD does not create predictability where there is none: for each asset, compute the out-of-sample (2018-2025) of the next day's return with the FFD of the log of the price () and with today's return ().
Solution
filas = {}
for activo, g in v5.groupby("activo"):
r = g.set_index("fecha")["rendimiento"]
logp = np.log1p(r).cumsum()
obj = r.shift(-1)
filas[activo] = {f"d={d}": r2_fuera(ffd(logp, d).to_frame("x"), obj)
for d in [0.5, 1.0]}
(pd.DataFrame(filas).T * 100).round(2) # in % d=0.5 d=1.0
accion 0.11 -0.05
bono -0.05 -0.08
cripto -0.06 -0.26
divisa 0.10 -0.13
indice -4.54 -0.26
oro -0.36 0.13In five of the six assets, both values are between −0.4% and +0.1%: nothing, as befits returns that are unpredictable by construction. The index gives −4.5% with the FFD: the regression learns in 2008-2017 a slope with the level remainder and extrapolates it to a 2018-2025 level it had not seen, so it predicts worse than the historical mean. An FFD does not manufacture a signal; what it can manufacture is an extrapolation error.
- Choose for gold (
oro) and crypto (cripto) invol5with a criterion that does not look at the test: the smallest (in steps of 0.05) that leaves less than 5% of 2015-2017 outside the 2008-2014 range. Then measure the fraction out of range of 2018-2025 relative to 2008-2017 with that .
Solution
def fuera(x, a, b):
# fraction of x in stretch b outside the range of x in stretch a
xa, xb = x.loc[a[0]:a[1]], x.loc[b[0]:b[1]]
return ((xb < xa.min()) | (xb > xa.max())).mean()
for activo in ["oro", "cripto"]:
r = v5[v5["activo"] == activo].set_index("fecha")["rendimiento"]
logp = np.log1p(r).cumsum()
for d in np.round(np.arange(0.05, 1.01, 0.05), 2):
x = ffd(logp, d)
if fuera(x, ("2008", "2014"), ("2015", "2017")) < 0.05:
break
f = fuera(x, ("2008", "2017"), ("2018", "2025"))
print(f"{activo}: chosen d {d}; out of range in 2018-2025: "
f"{f:.1%}")oro: chosen d 0.05; out of range in 2018-2025: 86.1%
cripto: chosen d 0.25; out of range in 2018-2025: 31.6%For gold, the 2015-2017 validation fits entirely within the 2008-2014 range with , but in 2018-2025 86% of the values fall outside: the gold in the data leaves its historical range in the test, and a single validation block could not know it. For crypto the criterion chooses and even so 32% falls outside. A range criterion evaluated once does not protect against a regime change; in a walk-forward you have to recompute the out-of-range fraction in each block, retrain (and revise ) when it grows, and prefer a somewhat larger than the minimum when the model is a tree.
- The continuous series of a level 2 future (
futuros2_continuos.parquet, columncierre_ajustado) has a backward proportional adjustment. A colleague computes its FFD with for a trend model. What problem does it have and how would you fix it?
Solution
The backward proportional adjustment multiplies the whole past by the ratio of each later roll: the level on a date
depends on jumps between contracts that had not yet happened (lesson 1.4.6). With the FFD keeps 13% of the
level at the start of the window, and with it that information from the future, like the normalized price in this
lesson. The returns of the adjusted series, on the other hand, are those of the contract that was held and are
point-in-time. The fix is to build the level forward: accumulate the series' returns from a fixed date (like the lp
of vol5) and compute the FFD on that level, which on each date only uses the past. Then, as in the lesson, choose
with the training data, check the range in the test and compare it with a set of returns.
Footnotes
-
M. López de Prado, Advances in Financial Machine Learning, Wiley, 2018, chapter 5 (“Fractionally Differentiated Features”); the fractional operator comes from J. R. M. Hosking, “Fractional differencing”, Biometrika 68(1), 1981, pp. 165-176, doi:10.1093/biomet/68.1.165. ↩
-
F. Corsi, “A Simple Approximate Long-Memory Model of Realized Volatility”, Journal of Financial Econometrics 7(2), 2009, pp. 174-196, doi:10.1093/jjfinec/nbp001. ↩
Each answer is folded under its question
Self-assessment
With a threshold of 0.0001, the fractional difference with d = 0.3 uses 388 weights and the sum of its weights is 0.13. What does that sum mean?
- AThat the series is not stationary
- BThat the feature is 0.13 times the price level of 387 sessions ago plus a weighted sum of the 387 later returns: a level remainder is left
- CThat 13% of the weights are positive
- DNothing: the sum of the weights is always 0.13
Show answer
Answer B. That the feature is 0.13 times the price level of 387 sessions ago plus a weighted sum of the 387 later returns: a level remainder is left
Grouping terms, the sum of w_k·y_(t−k) equals the sum of c_j·r_(t−j) plus c_(L−1)·y_(t−L+1), with c_j the partial sums of the weights. The last partial sum is the sum of all the weights: the weight left to the level of the series at the start of the window. With an infinite window it would be zero for any d > 0.
In the
vol5index, the fractional difference with d = 0.1 already passes the ADF at 5% in 2008-2017, but 77% of its 2018-2025 values fall outside the training range. What will a tree trained with it do?- AAssign all those values to the leaf at the edge of the training range, as if they were the maximum observed
- BExtrapolate the learned trend
- CRaise an error
- DIgnore the feature
Show answer
Answer A. Assign all those values to the leaf at the edge of the training range, as if they were the maximum observed
A tree splits the axis at thresholds learned from the training data; any value above the last threshold falls into the same leaf. The ADF measures whether the series returns to its mean within the sample, not whether the future sample is within the range of the past one.
In the panel, the logarithm of the price normalized by its last value has an IC of −0.066 with the following week's return, more than twice the oracle's ceiling (0.029); that of the published price, −0.0002. What does it indicate?
- AA very strong low-price effect
- BThat the oracle is miscalculated
- CA leak: normalizing by the last price makes today's level equal to minus the return from today to the end of the sample, which includes the following week; a fractional difference with a small d inherits it
- DThat more data are needed
Show answer
Answer C. A leak: normalizing by the last price makes today's level equal to minus the return from today to the end of the sample, which includes the following week; a fractional difference with a small d inherits it
No feature can beat the true expected return out of sample. The level of a price normalized by its last value (like a backward-adjusted series) is not point-in-time; the returns are. The fractional difference keeps the memory of the level, with its defects.
In FNX, a LightGBM with three full fractional differences has an out-of-sample correlation of −0.012 and one with five returns, 0.060. With only the returns part of the fractional differences, 0.069. What do you conclude?
- AThat the fractional difference is better than returns
- BThat LightGBM is not suitable for intraday data
- CThat d = 1 has to be used
- DThat the level remainder is noise for this target and does harm; the useful part of the fractional difference is a weighted sum of returns, equivalent to a good set of returns (the difference between 0.060 and 0.069 is within the standard error of 0.016)
Show answer
Answer D. That the level remainder is noise for this target and does harm; the useful part of the fractional difference is a weighted sum of returns, equivalent to a good set of returns (the difference between 0.060 and 0.069 is within the standard error of 0.016)
The level of FNX within the year says nothing about the return of the following 10 minutes, and a tree that uses it to split learns noise. The useful memory here is that of the recent returns, which the fractional difference summarizes with weights that decay like a power.
Key points
- A fixed-window fractional difference is a sum of past returns with weights that decay like a power plus a remainder of the initial level of the window (the sum of the weights: 0.13 with d = 0.3 and 0.04 with d = 0.5).
- The training ADF is not enough: in the index, d = 0.1 passes the ADF and leaves 77% of the test outside the training range. For a tree, the useful criterion is the stability of the distribution between training and test.
- The memory of the level inherits the defects of the level. With the panel price normalized by its last value (like a backward adjustment), the level gives an impossible IC (−0.066, more than twice the oracle) and the fractional difference inherits it (−0.055 with d = 0.3); with the published, point-in-time price, the level adds nothing.
- As a feature of a LightGBM on FNX, the returns part of the fractional difference performs like five returns (0.069 against 0.060, standard error 0.016); the level remainder makes the model worse.
- If the series is already stationary (the realized log-variance), it is not differenced: in the index, with d = 0 it explains 50% of the next day's variance and with d = 0.5 only 8%.