---
title: "5.2.3 Fractional differentiation for features"
description: "Fractional differencing as a feature: the memory it keeps, why the ADF is not enough, the leak it inherits from the level and what it adds over returns."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/fractional-differentiation-for-features/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/fractional-differentiation-for-features.png"
---

# 5.2.3 · Fractional differentiation for features

> Fractional differentiation promises the best of both worlds: a stationary series that keeps the memory of the price level. This lesson uses it as a model feature and measures what lies behind the promise: which part of the memory is past returns and which part is level, why passing the ADF in training does not keep the test from falling out of range, how the level drags its defects along and how much it adds to a LightGBM against a well-chosen set of returns.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.2 · Feature engineering
- Duration: 60 min
- Prerequisites: [1.4.3 · Adjusting for splits and dividends: adjusted and unadjusted prices](https://pythonparatrading.com/en/python-trading-course/level-1/adjusting-prices-for-splits-and-dividends/), [2.1.2 · Stationarity: ADF, KPSS, differencing and fractional differencing](https://pythonparatrading.com/en/python-trading-course/level-2/stationarity-adf-kpss-fractional-differencing/), [5.2.1 · Price, volume and microstructure features](https://pythonparatrading.com/en/python-trading-course/level-5/price-volume-microstructure-features/)

## By the end of this lesson

- Decompose a fixed-window fractional difference into a weighted sum of past returns plus a level remainder, and compute the weight of each part for each d.
- Measure, in addition to the ADF, the fraction of test values outside the training range and explain what a tree does with them.
- Detect that a feature inherits a leak from a price level that is not point-in-time by comparing its IC with the oracle's ceiling.
- Compare out of sample a model with fractional differences, one with returns and one with both, against their standard error.
- Decide when it is worth fractionally differencing a series and when it is not needed.

## What you already know and what is missing

Lesson 2.1.2 explained fractional differentiation: the operator $(1-B)^d$ with real $d$, its weights
$w_k = -w_{k-1}(d-k+1)/k$, the fixed-width window version (FFD) that cuts the weights below a threshold, and the sweep
over $d$ that looks for the smallest one that passes the ADF while keeping the correlation with the level. There the
goal was to have a stationary regressor. Here the question is a different one, the one that matters for a machine
learning model: is an FFD a **better feature** than returns?

López de Prado (2018)[^1] proposes it precisely for that: features made stationary by integer differencing lose all
their memory, and the FFD keeps part of it. To measure that promise you need to know exactly what the memory it keeps
is. The functions from lesson 2.1.2 are rewritten and one calculation is added:

```python
import numpy as np
import pandas as pd

def pesos(d, umbral=1e-4, kmax=100_000):
    """w_0 = 1, w_k = -w_(k-1)·(d - k + 1)/k until |w_k| < threshold."""
    w = [1.0]
    for k in range(1, kmax):
        wk = -w[-1] * (d - k + 1) / k
        if abs(wk) < umbral:
            break
        w.append(wk)
    return np.array(w)

def ffd(x, d, umbral=1e-4):
    """Fixed-window fractional difference (no future data)."""
    w = pesos(d, umbral)
    v = np.convolve(x.to_numpy(), w, mode="valid")
    return pd.Series(v, index=x.index[len(w) - 1:])

for d in [0.1, 0.3, 0.5, 0.7]:
    w = pesos(d)
    c = np.cumsum(w)  # partial sums: weight of each return
    print(f"d={d:.1f}: {len(w):3d} weights; returns from "
          f"1, 5, 21 and 63 sessions ago {np.round(c[[1, 5, 21, 63]], 3)}; "
          f"level remainder {c[-1]:.3f}")
```
```output
d=0.1: 503 weights; returns from 1, 5, 21 and 63 sessions ago [0.9   0.79  0.689 0.618]; level remainder 0.502
d=0.3: 388 weights; returns from 1, 5, 21 and 63 sessions ago [0.7   0.466 0.308 0.222]; level remainder 0.129
d=0.5: 200 weights; returns from 1, 5, 21 and 63 sessions ago [0.5   0.246 0.122 0.071]; level remainder 0.040
d=0.7:  97 weights; returns from 1, 5, 21 and 63 sessions ago [0.3   0.106 0.039 0.018]; level remainder 0.014
```

Grouping the terms of the sum, an FFD with $L$ weights can be written as past returns plus a level remainder:

$$
\sum_{k=0}^{L-1} w_k\, y_{t-k} \;=\; \sum_{j=0}^{L-2} c_j\, r_{t-j} \;+\; c_{L-1}\, y_{t-L+1},
\qquad c_j = \sum_{k=0}^{j} w_k,\quad r_t = y_t - y_{t-1}.
$$

The weight of the return from $j$ sessions ago is the partial sum $c_j$, which starts at 1 (today's) and decays like a
power, $c_j \approx j^{-d}/\Gamma(1-d)$: with $d = 0.3$, the return from 21 sessions ago still weighs 0.31 and the one
from 63 sessions ago, 0.22; with $d = 0.7$, 0.04 and 0.02. The last partial sum, which is the sum of all the weights,
is the weight left to the **level** of the series at the start of the window: 0.50 with $d = 0.1$, 0.13 with 0.3, 0.04
with 0.5. With an infinite window it would be zero for any $d > 0$; the fixed window leaves it there.

So the “memory” of an FFD has two parts of a different nature. One is an average of returns with slowly decaying
weights, something that can be approximated with returns over several horizons (lesson 5.2.1). The other is the level
from $L$ sessions ago, and that one is new: no set of returns contains it. The figure plots the first part; the
following sections measure both.

*Fig. — Weight of each past return in the fixed-width fractional difference (partial sums of the weights, threshold 1e-4, log scales); the last value of each curve is the level rest: 0.13, 0.04 and 0.01.* Three decreasing curves on logarithmic scales, age from 1 to 1,000 sessions and weight from 0.01 to 1. With d 0.3 the return from 21 sessions ago weighs 0.31 and the window ends at 387 sessions with a level rest of 0.13. With d 0.5, highlighted, it weighs 0.12 at 21 sessions and ends at 199 with rest 0.04. With d 0.7 it weighs 0.04 and ends at 96 with rest 0.01.

## Stationary in the ADF, out of range in the test

The criterion of lesson 2.1.2 chooses the smallest $d$ that passes the ADF. The ADF tests whether the series returns
to its mean **within the sample**. A model needs something else: that the values it will see in production resemble
the ones it saw in training. With the `vol5` index (log of the price since 2008, built by accumulating returns
forward: point-in-time), training on 2008-2017 and testing on 2018-2025, we measure for each $d$ the training ADF, the
fraction of test values outside the training range and the memory (correlation with the log of the price in the test):

```python
from statsmodels.tsa.stattools import adfuller

v5 = pd.read_parquet("vol5_diario.parquet")
r_ind = v5[v5["activo"] == "indice"].set_index("fecha")["rendimiento"]
lp = np.log1p(r_ind).cumsum()  # log of the price, 0 on the first day

filas = []
for d in np.round(np.arange(0, 1.01, 0.05), 2):
    x = ffd(lp, d) if d > 0 else lp
    ent, prueba = x.loc[:"2017-12-31"], x.loc["2018-01-01":]
    adf = adfuller(ent, maxlag=1, regression="c", autolag=None,
                   result_object=True)
    fuera = ((prueba < ent.min()) | (prueba > ent.max())).mean()
    memoria = np.corrcoef(prueba, lp.loc[prueba.index])[0, 1]
    filas.append({"d": d, "adf": adf.statistic, "fuera": fuera,
                  "memoria": memoria})
t = pd.DataFrame(filas).set_index("d")
critico = adf.critical_values["5%"]
d_adf = t.index[t["adf"] < critico].min()
d_rango = t.index[t["fuera"] < 0.05].min()
print(f"5% critical value: {critico:.2f}; minimum d by the "
      f"ADF: {d_adf}; by the range (< 5% outside): {d_rango}")
t.loc[[0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.7, 1.0]].round(3)
```
```output
5% critical value: -2.86; minimum d by the ADF: 0.1; by the range (< 5% outside): 0.5
         adf  fuera  memoria
d
0.00  -2.630  0.879    1.000
0.05  -2.506  0.832    0.999
0.10  -3.328  0.769    0.996
0.20  -5.110  0.656    0.975
0.30  -7.499  0.546    0.928
0.40 -10.968  0.321    0.847
0.50 -15.390  0.033    0.725
0.70 -25.172  0.000    0.409
1.00 -36.912  0.000    0.018
```

The header figure plots the table. With $d = 0.1$ the FFD already passes the ADF (−3.33 against −2.86) and keeps a
correlation of 0.996 with the price, but 77% of its 2018-2025 values are outside the 2008-2017 range: the index rose
and the FFD, with half its weight on the level, rose with it. For less than 5% of the test to fall outside, you need
$d = 0.5$, and then the memory drops to 0.725. Between the two there is an interval of $d$ in which the series is
“stationary” according to the test and is not for a model.

The ADF is not wrong: with ten years of data and a level remainder that changes slowly, the series mean-reverts
enough within the sample to reject the unit root. The problem is that the level of the test is not in the sample.
What a model does with that depends on the model; a tree does it in a very visible way:

```python
import lightgbm as lgb

y21 = lp.shift(-21) - lp  # log return of the next 21

def arbol(d):
    x = (ffd(lp, d) if d > 0 else lp).rename("x")
    df = pd.concat([x, y21.rename("y")], axis=1, sort=True).dropna()
    ent = df.loc[:"2017-12-31"]
    prueba = df.loc["2018-01-01":"2025-10-31"]
    m = lgb.LGBMRegressor(n_estimators=100, learning_rate=0.05,
                          num_leaves=8, min_child_samples=100,
                          random_state=0, n_jobs=4, deterministic=True,
                          force_row_wise=True, verbose=-1)
    m.fit(ent[["x"]], ent["y"])
    pred = m.predict(prueba[["x"]])
    fuera = prueba["x"] > ent["x"].max()
    return fuera.mean(), np.unique(pred[fuera.to_numpy()]).size

for d in [0.0, 0.1, 0.5]:
    f, n = arbol(d)
    print(f"d={d:.2f}: {f:.0%} of the test above the training "
          f"maximum; distinct predictions there: {n}")
```
```output
d=0.00: 88% of the test above the training maximum; distinct predictions there: 1
d=0.10: 76% of the test above the training maximum; distinct predictions there: 1
d=0.50: 3% of the test above the training maximum; distinct predictions there: 1
```

A tree splits the feature axis at thresholds learned from the training data. Everything that falls above the last
threshold receives the prediction of the last leaf: with the log of the price, 88% of the test receives a single
prediction; with $d = 0.1$, 76%. The model does not extrapolate, it stays at the edge. With $d = 0.5$ only 3% falls
there. A linear regression would do the opposite, extrapolate without limit, which is no better. The target of this
cell (the return of the following 21 sessions) is unpredictable in `vol5` by construction (data README): the cell does
not measure whether the model is right, but how it treats what it has not seen.

<aside class="callout" data-kind="nota">

The practical criterion for a feature of a tree model: choose $d$ with the training data (never with the whole sample)
and check, in a walk-forward scheme (lesson 5.4.3), what fraction of each test block falls outside the range of its
training data. The ADF is a necessary condition, not a sufficient one. The PSI of lesson 5.10.2 measures the same thing
more finely.

</aside>

## The memory of the level inherits its defects

The level remainder is the part of the FFD that cannot be built with returns. If the level carries information, the
FFD has it; if the level carries a defect, so does the FFD. The stock panel lets you see this in two versions of the
same price: the published one, which accumulates returns forward from an initial price and is point-in-time, and the
same series **normalized by its last price** ($\log P_t - \log P_T$), which is what someone does when dividing each
series by its last value to compare them, and what, in essence, a backward adjustment does. For each one the full FFD
and the FFD without the level remainder are computed, with their IC against the following week's return:

```python
from scipy.signal import lfilter

px = pd.read_parquet("panel5_precios.parquet")
LP = np.log(px.pivot(index="fecha", columns="ticker", values="cierre"))
fe = pd.read_parquet("panel5_features.parquet")
et = pd.read_parquet("panel5_etiquetas.parquet")
fechas = pd.DatetimeIndex(fe["fecha"].unique())
Y = et.pivot(index="fecha", columns="ticker",
             values="ret_sig_semana").loc[fechas]

def ic(X):
    """Weekly rank IC with the following week: mean and t."""
    X = X.reindex(index=fechas, columns=Y.columns)
    v = X.rank(axis=1).corrwith(Y.rank(axis=1), axis=1).dropna()
    t = v.mean() / v.std() * np.sqrt(len(v))
    return round(float(v.mean()), 4), round(float(t), 1)

def ffd_panel(L, d):
    """FFD by columns: full and without the level remainder."""
    w = pesos(d)
    completa = pd.DataFrame(lfilter(w, [1.0], L.to_numpy(), axis=0),
                            index=L.index, columns=L.columns)
    completa.iloc[:len(w) - 1] = np.nan
    return completa, completa - w.sum() * L.shift(len(w) - 1)

LN = LP - LP.iloc[-1]  # normalized: 0 on the last day of the sample
for nombre, L in [("published", LP), ("normalized", LN)]:
    print(f"{nombre}: log of the price, IC, t = {ic(L)}")
    for d in [0.3, 0.5, 0.7, 1.0]:
        completa, rendim = ffd_panel(L, d)
        print(f"  d={d:.1f}: full {ic(completa)}, "
              f"returns only {ic(rendim)}")
```
```output
published: log of the price, IC, t = (-0.0002, -0.1)
  d=0.3: full (-0.0008, -0.3), returns only (-0.0014, -0.4)
  d=0.5: full (-0.0039, -1.4), returns only (-0.0058, -1.6)
  d=0.7: full (-0.0061, -1.9), returns only (-0.0086, -2.3)
  d=1.0: full (-0.004, -1.0), returns only (-0.004, -1.0)
normalized: log of the price, IC, t = (-0.0661, -15.5)
  d=0.3: full (-0.0553, -15.0), returns only (-0.0014, -0.4)
  d=0.5: full (-0.0455, -12.7), returns only (-0.0058, -1.6)
  d=0.7: full (-0.032, -8.7), returns only (-0.0086, -2.3)
  d=1.0: full (-0.004, -1.0), returns only (-0.004, -1.0)
```

With the published price, the level says nothing about the following week (IC −0.0002) and the full FFD gives almost
the same as its returns part: small, negative ICs (−0.009 at most, with $d = 0.7$), the planted weekly reversal seen
through an average of returns. The level remainder adds no information, but it does no harm either.

With the price normalized at the end, the log of the price moves to an IC of −0.066 with $t$ = −15.5: more than twice
the ceiling of the README oracle (0.029), which knows the true expected return. Out of sample, no feature can beat the
oracle, so this is a leak. The full FFD inherits it (−0.055 with $d = 0.3$, −0.046 with 0.5, −0.032 with 0.7) even
though its level remainder weighs 0.13, 0.04 or 0.01, because the level varies much more across stocks than a sum of
returns. The returns part is identical in the two versions: normalizing subtracts a constant per stock, which cancels
out in the returns.

*Fig. — 400-stock panel: absolute IC of the fractional difference of the log price with next-week return; with the price normalised by its last value it exceeds the oracle ceiling because it inherits a leak from the level.* Three curves against d. With the price normalised by its last value, highlighted, the absolute IC is 0.066 at d 0, 0.055 at d 0.3, 0.046 at d 0.5 and 0.032 at d 0.7, above the dashed line of the oracle ceiling (0.029) up to d 0.7; at d 1 it falls to 0.004. With the published, point-in-time price it stays between 0.000 and 0.006, and the returns part, common to both, between 0.001 and 0.009.

Where does the leak come from? Subtracting the log of the last price makes today's level exactly minus the return from
today to the end of the sample:

```python
hasta_el_final = LP.iloc[-1] - LP  # log return from t to 2025-12-31
for f in ["2013-06-28", "2016-06-30", "2019-06-28", "2022-06-30"]:
    pub = np.corrcoef(LP.loc[f], hasta_el_final.loc[f])[0, 1]
    nor = np.corrcoef(LN.loc[f], hasta_el_final.loc[f])[0, 1]
    print(f"{f}: corr. with the return to the end: "
          f"published {pub:+.2f}, normalized {nor:+.2f}")
```
```output
2013-06-28: corr. with the return to the end: published +0.03, normalized -1.00
2016-06-30: corr. with the return to the end: published -0.02, normalized -1.00
2019-06-28: corr. with the return to the end: published +0.01, normalized -1.00
2022-06-30: corr. with the return to the end: published -0.02, normalized -1.00
```

With the published price, the cross-sectional correlation between the level and the return up to 2025 is nil
(between −0.02 and +0.03); with the normalized one it is −1 by construction. The following week is part of that future
return, and that is where the impossible IC comes from. It is not a laboratory example: the first version of these
synthetic data rescaled each series so that its last price fell in a plausible range, had exactly this defect and was
caught this way, with a level feature that beat the oracle.

On real data the same problem appears in a milder form. A price backward-adjusted for dividends and splits (lesson
1.4.3) multiplies the past by factors that depend on later dividends and splits; a continuous futures series with
proportional adjustment (lesson 1.4.6), by the jumps of later *rolls*; a series normalized to 100 on its last day, by
the whole future return. The returns of all of them are point-in-time; the level is not. Any feature that keeps memory
of the level (an FFD, a distance to an all-time high in dollars, the price itself as a column of a tree) has to be
built on a level that was known on each date: unadjusted prices, or returns accumulated forward from a fixed day, like
the panel's published price or the `lp` of the `vol5` index. And the IC of any new feature is compared with the
oracle's ceiling or, on real data, with what is plausible.

## Does it add anything in a model? FNX with LightGBM

The main question remains: with a point-in-time level and a nonlinear model, does the FFD add anything over returns?
The FNX data (lesson 5.2.1) have a structure that favors a feature with memory: in the reversion regime, the price
returns toward its 30-minute exponential average, that is, toward a weighted average of its own past, which is what
the returns part of an FFD measures. The level within the year, on the other hand, says nothing about the next minute.

The setup: features computed within each session (the window does not cross the night: threshold $10^{-3}$, windows of
26 to 66 minutes), the target is the log return of the following 10 minutes, training on the first half of the
sessions and testing on the second, with one observation every 10 minutes so that the targets do not overlap. Four
sets are compared: five returns (1, 5, 10, 30 and 60 minutes), three full FFDs ($d$ = 0.3, 0.5 and 0.7), the same
without the level remainder, and returns plus full FFDs; each one alone and with the 30-minute order imbalance of
lesson 5.2.1.

```python
b = pd.read_parquet("hf5_barras_1min.parquet")
n_ses = b["ts"].dt.date.nunique()
P = np.log(b["cierre"].to_numpy()).reshape(n_ses, 390)  # session × min
VC = b["volumen_comprador"].to_numpy().reshape(n_ses, 390)
VT = b["volumen"].to_numpy().reshape(n_ses, 390)

def por_sesion(w):
    """Filter with the weights w within each session (no crossing)."""
    X = lfilter(w, [1.0], P, axis=1)
    X[:, :len(w) - 1] = np.nan
    return X

def retardo(A, k):
    out = np.full_like(A, np.nan, dtype=float)
    out[:, k:] = A[:, :-k]
    return out

H = 10
feats = {}
for k in [1, 5, 10, 30, 60]:
    feats[f"r{k}"] = P - retardo(P, k)
for d in [0.3, 0.5, 0.7]:
    w = pesos(d, umbral=1e-3)  # windows of 26 to 66 minutes
    completa = por_sesion(w)
    feats[f"ffd{d}"] = completa
    feats[f"ffdr{d}"] = completa - w.sum() * retardo(P, len(w) - 1)
cv, ct = np.cumsum(VC, axis=1), np.cumsum(VT, axis=1)
feats["di30"] = 2 * (cv - retardo(cv, 30)) / (ct - retardo(ct, 30)) - 1
y = np.full_like(P, np.nan)
y[:, :-H] = P[:, H:] - P[:, :-H]  # 10-minute log return

X = pd.DataFrame({k: v.reshape(-1) for k, v in feats.items()})
y = y.reshape(-1)
sesion = np.repeat(np.arange(n_ses), 390)
minuto = np.tile(np.arange(390), n_ses)
ok = X.notna().all(axis=1).to_numpy() & np.isfinite(y)
ent = ok & (sesion < n_ses // 2)
prueba = ok & (sesion >= n_ses // 2) & (minuto % H == 0)
et_ = 1 / np.sqrt(prueba.sum())
print(f"{ent.sum()} training rows; {prueba.sum()} test rows "
      f"without overlap; standard error ≈ {et_:.3f}")
```
```output
38745 training rows; 3844 test rows without overlap; standard error ≈ 0.016
```

With 3,844 test observations, the standard error of a correlation is about 0.016.

```python
conjuntos = {
    "returns": ["r1", "r5", "r10", "r30", "r60"],
    "full ffd": ["ffd0.3", "ffd0.5", "ffd0.7"],
    "ffd without level": ["ffdr0.3", "ffdr0.5", "ffdr0.7"],
    "returns + full ffd": ["r1", "r5", "r10", "r30", "r60",
                           "ffd0.3", "ffd0.5", "ffd0.7"],
}
res = {}
for nombre, cols in conjuntos.items():
    for extra in [[], ["di30"]]:
        m = lgb.LGBMRegressor(
            n_estimators=200, learning_rate=0.03, num_leaves=15,
            min_child_samples=200, subsample=0.5, subsample_freq=1,
            random_state=0, n_jobs=4, deterministic=True,
            force_row_wise=True, verbose=-1)
        m.fit(X.loc[ent, cols + extra], y[ent])
        pred = m.predict(X.loc[prueba, cols + extra])
        etiqueta = "with di30" if extra else "alone"
        res[(nombre, etiqueta)] = np.corrcoef(pred, y[prueba])[0, 1]
pd.Series(res).unstack().round(3)
```
```output
                    alone  with di30
ffd without level   0.069      0.102
full ffd           -0.012      0.039
returns             0.060      0.094
returns + full ffd  0.032      0.074
```

Three conclusions, in order of importance:

- **The level remainder does harm.** The full FFDs give −0.012, worse than nothing, and added to the returns they make
  them worse, from 0.060 to 0.032: the tree splits on the level of FNX, which has no relationship with the target, and
  learns noise from the training data.
- **The useful part of the FFD is an average of returns.** Without the remainder, the three FFDs give 0.069, the same
  as the five returns (0.060) within the standard error. The FFD is a compact way of giving the model an average of
  returns with weights that decay like a power; a set of returns over several horizons contains almost the same
  information.
- **What improves the model is new information.** The order imbalance lifts every set by some 0.03-0.05 (0.094 with
  returns, 0.102 with FFD without level). Transforming the same prices in another way moves the result much less than
  adding a piece of data that the price does not have.

*Fig. — FNX: out-of-sample correlation of a LightGBM with the next 10 minutes' log return by feature set (whisker: ±1.96 standard errors); the level rest of the fractional difference hurts the model.* Two bar panels. Prices only: five returns 0.060; three full fractional differences, highlighted, −0.012; the same without the level rest 0.069; returns plus full differences 0.032. Adding the 30-minute order imbalance: 0.094, 0.039, 0.102 and 0.074. The ±0.032 error whiskers overlap between returns and differences without level.

Outside these data, the result will depend on the asset, the horizon and the model: the FFD is worth testing as one
more feature, with the same validation as any other, and not as an automatic substitute for returns.

## When it is not needed: a series that is already stationary

The other extreme: a series that is already stationary and has long memory. Realized variance is the typical case;
its logarithm is persistent but returns to its mean, and Corsi's HAR model[^2] approximates that long memory with three
averages (1, 5 and 22 days). In `vol5`, the HAR is the true model. Fractionally differencing the log of the realized
variance to predict the next day's:

```python
from sklearn.linear_model import LinearRegression

def r2_fuera(X, y):
    df = pd.concat([X, y.rename("y")], axis=1, sort=True).dropna()
    a = df.loc["2009-01-01":"2017-12-31"]
    b_ = df.loc["2018-01-01":]
    m = LinearRegression().fit(a.drop(columns="y"), a["y"])
    e = b_["y"] - m.predict(b_.drop(columns="y"))
    return 1 - (e ** 2).sum() / ((b_["y"] - a["y"].mean()) ** 2).sum()

filas = {}
for activo, g in v5.groupby("activo"):
    lrv = np.log(g.set_index("fecha")["rv_5min"])
    obj = lrv.shift(-1)
    fila = {f"d={d}": r2_fuera((ffd(lrv, d, 1e-3) if d else lrv)
                                .to_frame("x"), obj)
            for d in [0, 0.2, 0.5, 1.0]}
    har = pd.concat({"d1": lrv, "d5": lrv.rolling(5).mean(),
                     "d22": lrv.rolling(22).mean()}, axis=1)
    fila["HAR"] = r2_fuera(har, obj)
    filas[activo] = fila
pd.DataFrame(filas).T.round(3)
```
```output
          d=0  d=0.2  d=0.5  d=1.0    HAR
accion  0.275  0.117  0.025  0.003  0.393
bono    0.443  0.217  0.052  0.009  0.543
cripto  0.339  0.125  0.022  0.003  0.488
divisa  0.375  0.186  0.044  0.004  0.466
indice  0.505  0.293  0.082  0.014  0.568
oro     0.480  0.235  0.054  0.007  0.580
```

Each step of $d$ toward 1 destroys predictive power: in the index, the out-of-sample $R^2$ goes from 0.505 with the
log of the variance itself ($d = 0$) to 0.293 with $d = 0.2$, 0.082 with 0.5 and 0.014 with 1. The memory was the
information, and the series did not need to be “more stationary”. The HAR, with three averages of the undifferenced
series, reaches 0.568. It is the rule of lesson 2.1.2 (do not difference what is already stationary) seen through a
model.

The lesson's decisions, in a diagram:

```mermaid
flowchart TB
  S{Stationary?} -- yes --> N[Do not difference]
  S -- no --> O{Does the level matter?}
  O -- no --> R[Returns]
  O -- yes --> P{PIT level?}
  P -- no --> R
  P -- yes --> F[FFD with fixed d]
```

## Common mistakes

- **Choosing $d$ with the whole sample.** The sweep over $d$ is one more fit: it is done on the training data and kept
  fixed in the test.
- **Trusting the ADF as the criterion for a model.** It passes with small values of $d$ that leave most of the test
  outside the training range. You also have to measure the stability of the distribution.
- **Applying the FFD to a level that is not point-in-time.** Backward-adjusted prices, continuous series with
  proportional adjustment or series normalized by their last value carry the future in the level, and the FFD keeps it.
- **Windows that cross sessions or gaps.** With intraday data, a window of hundreds of minutes mixes days and drags in
  the opening jump; it is computed by session with a higher threshold or with a fixed window.
- **Differencing what is already stationary.** The realized log-variance, a cointegrated spread or an interest rate in
  a bounded range lose information when differenced.
- **Not comparing with a baseline.** An FFD only deserves to enter the model if it beats a simple set of returns, out
  of sample and beyond its standard error.

## Exercises

1. Check with the `vol5` control that an FFD does not create predictability where there is none: for each asset,
   compute the out-of-sample $R^2$ (2018-2025) of the next day's return with the FFD of the log of the price
   ($d = 0.5$) and with today's return ($d = 1$).

**Solution.**

```python
filas = {}
for activo, g in v5.groupby("activo"):
    r = g.set_index("fecha")["rendimiento"]
    logp = np.log1p(r).cumsum()
    obj = r.shift(-1)
    filas[activo] = {f"d={d}": r2_fuera(ffd(logp, d).to_frame("x"), obj)
                     for d in [0.5, 1.0]}
(pd.DataFrame(filas).T * 100).round(2)  # in %
```
```output
        d=0.5  d=1.0
accion   0.11  -0.05
bono    -0.05  -0.08
cripto  -0.06  -0.26
divisa   0.10  -0.13
indice  -4.54  -0.26
oro     -0.36   0.13
```

In five of the six assets, both $R^2$ values are between −0.4% and +0.1%: nothing, as befits returns that are
unpredictable by construction. The index gives −4.5% with the FFD: the regression learns in 2008-2017 a slope with the
level remainder and extrapolates it to a 2018-2025 level it had not seen, so it predicts worse than the historical
mean. An FFD does not manufacture a signal; what it can manufacture is an extrapolation error.

2. Choose $d$ for gold (`oro`) and crypto (`cripto`) in `vol5` with a criterion that does not look at the test: the
   smallest $d$ (in steps of 0.05) that leaves less than 5% of 2015-2017 outside the 2008-2014 range. Then measure the
   fraction out of range of 2018-2025 relative to 2008-2017 with that $d$.

**Solution.**

```python
def fuera(x, a, b):
    # fraction of x in stretch b outside the range of x in stretch a
    xa, xb = x.loc[a[0]:a[1]], x.loc[b[0]:b[1]]
    return ((xb < xa.min()) | (xb > xa.max())).mean()

for activo in ["oro", "cripto"]:
    r = v5[v5["activo"] == activo].set_index("fecha")["rendimiento"]
    logp = np.log1p(r).cumsum()
    for d in np.round(np.arange(0.05, 1.01, 0.05), 2):
        x = ffd(logp, d)
        if fuera(x, ("2008", "2014"), ("2015", "2017")) < 0.05:
            break
    f = fuera(x, ("2008", "2017"), ("2018", "2025"))
    print(f"{activo}: chosen d {d}; out of range in 2018-2025: "
          f"{f:.1%}")
```
```output
oro: chosen d 0.05; out of range in 2018-2025: 86.1%
cripto: chosen d 0.25; out of range in 2018-2025: 31.6%
```

For gold, the 2015-2017 validation fits entirely within the 2008-2014 range with $d = 0.05$, but in 2018-2025 86% of
the values fall outside: the gold in the data leaves its historical range in the test, and a single validation block
could not know it. For crypto the criterion chooses $d = 0.25$ and even so 32% falls outside. A range criterion
evaluated once does not protect against a regime change; in a walk-forward you have to recompute the out-of-range
fraction in each block, retrain (and revise $d$) when it grows, and prefer a $d$ somewhat larger than the minimum when
the model is a tree.

3. The continuous series of a level 2 future (`futuros2_continuos.parquet`, column `cierre_ajustado`) has a backward
   proportional adjustment. A colleague computes its FFD with $d = 0.3$ for a trend model. What problem does it have
   and how would you fix it?

**Solution.**

The backward proportional adjustment multiplies the whole past by the ratio of each later *roll*: the level on a date
depends on jumps between contracts that had not yet happened (lesson 1.4.6). With $d = 0.3$ the FFD keeps 13% of the
level at the start of the window, and with it that information from the future, like the normalized price in this
lesson. The returns of the adjusted series, on the other hand, are those of the contract that was held and are
point-in-time. The fix is to build the level forward: accumulate the series' returns from a fixed date (like the `lp`
of `vol5`) and compute the FFD on that level, which on each date only uses the past. Then, as in the lesson, choose
$d$ with the training data, check the range in the test and compare it with a set of returns.

[^1]: M. López de Prado, *Advances in Financial Machine Learning*, Wiley, 2018, chapter 5 (“Fractionally Differentiated
    Features”); the fractional operator comes from J. R. M. Hosking, “Fractional differencing”, *Biometrika* 68(1),
    1981, pp. 165-176, [doi:10.1093/biomet/68.1.165](https://doi.org/10.1093/biomet/68.1.165).
[^2]: F. Corsi, “A Simple Approximate Long-Memory Model of Realized Volatility”, *Journal of Financial Econometrics*
    7(2), 2009, pp. 174-196, [doi:10.1093/jjfinec/nbp001](https://doi.org/10.1093/jjfinec/nbp001).

## Self-assessment

1. With a threshold of 0.0001, the fractional difference with d = 0.3 uses 388 weights and the sum of its weights is 0.13. What does that sum mean?

   - A. That the series is not stationary
   - B. That the feature is 0.13 times the price level of 387 sessions ago plus a weighted sum of the 387 later returns: a level remainder is left
   - C. That 13% of the weights are positive
   - D. Nothing: the sum of the weights is always 0.13

   *Answer B: That the feature is 0.13 times the price level of 387 sessions ago plus a weighted sum of the 387 later returns: a level remainder is left.* Grouping terms, the sum of w_k·y_(t−k) equals the sum of c_j·r_(t−j) plus c_(L−1)·y_(t−L+1), with c_j the partial sums of the weights. The last partial sum is the sum of all the weights: the weight left to the level of the series at the start of the window. With an infinite window it would be zero for any d > 0.

2. In the `vol5` index, the fractional difference with d = 0.1 already passes the ADF at 5% in 2008-2017, but 77% of its 2018-2025 values fall outside the training range. What will a tree trained with it do?

   - A. Assign all those values to the leaf at the edge of the training range, as if they were the maximum observed
   - B. Extrapolate the learned trend
   - C. Raise an error
   - D. Ignore the feature

   *Answer A: Assign all those values to the leaf at the edge of the training range, as if they were the maximum observed.* A tree splits the axis at thresholds learned from the training data; any value above the last threshold falls into the same leaf. The ADF measures whether the series returns to its mean within the sample, not whether the future sample is within the range of the past one.

3. In the panel, the logarithm of the price normalized by its last value has an IC of −0.066 with the following week's return, more than twice the oracle's ceiling (0.029); that of the published price, −0.0002. What does it indicate?

   - A. A very strong low-price effect
   - B. That the oracle is miscalculated
   - C. A leak: normalizing by the last price makes today's level equal to minus the return from today to the end of the sample, which includes the following week; a fractional difference with a small d inherits it
   - D. That more data are needed

   *Answer C: A leak: normalizing by the last price makes today's level equal to minus the return from today to the end of the sample, which includes the following week; a fractional difference with a small d inherits it.* No feature can beat the true expected return out of sample. The level of a price normalized by its last value (like a backward-adjusted series) is not point-in-time; the returns are. The fractional difference keeps the memory of the level, with its defects.

4. In FNX, a LightGBM with three full fractional differences has an out-of-sample correlation of −0.012 and one with five returns, 0.060. With only the returns part of the fractional differences, 0.069. What do you conclude?

   - A. That the fractional difference is better than returns
   - B. That LightGBM is not suitable for intraday data
   - C. That d = 1 has to be used
   - D. That the level remainder is noise for this target and does harm; the useful part of the fractional difference is a weighted sum of returns, equivalent to a good set of returns (the difference between 0.060 and 0.069 is within the standard error of 0.016)

   *Answer D: That the level remainder is noise for this target and does harm; the useful part of the fractional difference is a weighted sum of returns, equivalent to a good set of returns (the difference between 0.060 and 0.069 is within the standard error of 0.016).* The level of FNX within the year says nothing about the return of the following 10 minutes, and a tree that uses it to split learns noise. The useful memory here is that of the recent returns, which the fractional difference summarizes with weights that decay like a power.

## Key points

- A fixed-window fractional difference is a sum of past returns with weights that decay like a power plus a remainder of the initial level of the window (the sum of the weights: 0.13 with d = 0.3 and 0.04 with d = 0.5).
- The training ADF is not enough: in the index, d = 0.1 passes the ADF and leaves 77% of the test outside the training range. For a tree, the useful criterion is the stability of the distribution between training and test.
- The memory of the level inherits the defects of the level. With the panel price normalized by its last value (like a backward adjustment), the level gives an impossible IC (−0.066, more than twice the oracle) and the fractional difference inherits it (−0.055 with d = 0.3); with the published, point-in-time price, the level adds nothing.
- As a feature of a LightGBM on FNX, the returns part of the fractional difference performs like five returns (0.069 against 0.060, standard error 0.016); the level remainder makes the model worse.
- If the series is already stationary (the realized log-variance), it is not differenced: in the index, with d = 0 it explains 50% of the next day's variance and with d = 0.5 only 8%.

## Resources

### Research

- [Stochastic Processes - Random Walks](https://pythonparatrading.com/en/stochastic-processes/) · Research-0005 · 11 min

### Documentation

- [Hosking (1981): Fractional differencing, Biometrika 68(1)](https://doi.org/10.1093/biomet/68.1.165) · doi.org
- [Corsi (2009): A Simple Approximate Long-Memory Model of Realized Volatility, Journal of Financial Econometrics 7(2)](https://doi.org/10.1093/jjfinec/nbp001) · doi.org
- [statsmodels: adfuller](https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.adfuller.html) · statsmodels.org
- [LightGBM: documentation](https://lightgbm.readthedocs.io/en/stable/) · lightgbm.readthedocs.io

### Books

- [López de Prado (2018): Advances in Financial Machine Learning, Wiley, chapter 5](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com

### Data

- [Level 5 data: returns and realized variance of six assets (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet

Previous lesson: [5.2.2 · Point-in-time fundamentals, macro and alternative data](https://pythonparatrading.com/en/python-trading-course/level-5/point-in-time-fundamentals-macro-alternative-data/)

Next lesson: [5.2.4 · Leakage-free normalization and cross-sectional rank](https://pythonparatrading.com/en/python-trading-course/level-5/leakage-free-normalization-cross-sectional-rank/)