--- title: "2.1.2 Stationarity: ADF, KPSS and differencing" description: "ADF and KPSS on prices, returns and spreads with a known answer, the four combinations and fractional differencing that preserves memory." url: "https://pythonparatrading.com/en/python-trading-course/level-2/stationarity-adf-kpss-fractional-differencing/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-2/stationarity-adf-kpss-fractional-differencing.png" --- # 2.1.2 · Stationarity: ADF, KPSS, differencing and fractional differencing > Almost every statistical method assumes that the series does not change its nature over time. Prices do not meet that condition; returns do, but in exchange they forget where the price was. This lesson measures stationarity with two tests with opposite hypotheses and looks for the middle ground: the smallest fractional difference that achieves it. - Course: [Level 2: Systematic strategies and rigorous backtesting](https://pythonparatrading.com/en/python-trading-course/level-2/) - Module: 2.1 · Statistics and econometrics for trading - Duration: 65 min - Prerequisites: [1.6.2 · Stylized facts: fat tails, clustering, autocorrelation and leverage](https://pythonparatrading.com/en/python-trading-course/level-1/stylized-facts-of-financial-markets/), [2.1.1 · Hypothesis tests, p-values, bootstrap and regression with robust standard errors](https://pythonparatrading.com/en/python-trading-course/level-2/hypothesis-testing-bootstrap-robust-errors/) ## By the end of this lesson - Explain what a stationary series is and why a price is not one, with the Dickey-Fuller regression and its special distribution. - Apply ADF (null: unit root) and KPSS (null: stationary) with statsmodels and read the four combinations of results. - Check on spreads with a known answer when the tests get it right, when they lack sample and when persistence confuses them. - Code fixed-window fractional differencing and find the minimum d that passes the ADF while preserving the correlation with the price. ## What stationary means A series is **stationary** (in the weak sense) if its mean, its variance and the covariance between two observations $k$ periods apart do not depend on the date. Put another way: a stretch from 2005 and one from 2023 are two samples from the same distribution. That is what makes it meaningful to estimate a mean, a volatility or a regression with past data and use it in the future. Prices are not stationary. An index that traded at 1,000 in 2003 and at 5,000 in 2024 does not have “a mean” to return to; its variance grows with the horizon. The simplest model that captures this is the **random walk**: today's price is yesterday's plus a shock, $p_t = p_{t-1} + \varepsilon_t$. Its difference, the return, is stationary. In the jargon, the price is **integrated of order 1**, $I(1)$, and the return, $I(0)$. ```mermaid flowchart LR P[Price I1] --> L[Logarithm] L --> D1[Difference d=1] L --> DF[Fractional diff.] D1 --> R[Return I0] DF --> M[I0 with memory] ``` Why it matters in a trading system: - **Spurious regressions.** Two independent integrated series give huge correlations and t's by chance (lesson 0.7.2). A regression of one price on another only makes sense if they are cointegrated (lesson 2.1.5). - **Mean reversion.** A rule that buys when the price is “below its mean” assumes there is a mean to return to. In an $I(1)$ price there is none; in a stationary spread, there is (lesson 2.3.3). - **Model features.** A model trained on the level of a price learns the relationship of a period that will not repeat (level 3). The data in this lesson are synthetic (level 2 README): an index future that is a random walk with drift, commodity futures that return very slowly to an anchor (half-lives of years) and pairs whose true spread is stationary or not, depending on the pair. The answer is known in advance; that makes it possible to see where the tests get it right and where they do not. ## Dickey-Fuller: testing the unit root The idea of the Dickey-Fuller test is to estimate how much the series “comes back”. In the regression $$ \Delta y_t = a + \gamma\, y_{t-1} + \sum_{i=1}^{p} \phi_i\, \Delta y_{t-i} + \varepsilon_t $$ a negative $\gamma$ means that, when $y$ is high, it tends to go down: reversion. If $\gamma = 0$, the series is a random walk (it has a **unit root**). The ADF test (*augmented*, because of the lags $\Delta y_{t-i}$, which absorb short-term autocorrelation) uses the null $H_0: \gamma = 0$ against $\gamma < 0$. The catch is that, under the null, $y_{t-1}$ is not stationary and the t-statistic of $\gamma$ does not follow a Student's t. It follows the Dickey-Fuller distribution, shifted further to the left. You can see it by simulating seeded random walks and computing the statistic on each one: ```python import numpy as np import pandas as pd from scipy import stats rng = np.random.default_rng(212) sims, n = 4000, 1000 y = rng.standard_normal((sims, n)).cumsum(axis=1) # walks dy, ylag = np.diff(y, axis=1), y[:, :-1] def t_gamma(dy, ylag): # OLS of dy = a + g·ylag, row by row xm = ylag - ylag.mean(1, keepdims=True) g = (xm * dy).sum(1) / (xm**2).sum(1) a = dy.mean(1) - g * ylag.mean(1) e = dy - a[:, None] - g[:, None] * ylag s2 = (e**2).sum(1) / (dy.shape[1] - 2) return g / np.sqrt(s2 / (xm**2).sum(1)) t = t_gamma(dy, ylag) print(f"5% percentile of the Dickey-Fuller t: " f"{np.percentile(t, 5):.2f} (normal: " f"{stats.norm.ppf(0.05):.2f})") print(f"“stationary” walks with the normal threshold: " f"{np.mean(t < stats.norm.ppf(0.05)):.1%}") ``` ```output 5% percentile of the Dickey-Fuller t: -2.88 (normal: -1.64) “stationary” walks with the normal threshold: 45.3% ``` *Fig. — Dickey-Fuller t statistic (regression with constant) in 4,000 simulated random walks of 1,000 steps, against the standard normal.* Histogram of the Dickey-Fuller statistic in 0.25 bins, centred at −1.5, with the standard normal curve overlaid, centred at 0. Two vertical lines: the simulation's 5th percentile, −2.88, highlighted, and the normal's, −1.64. 45% of the random walks fall left of −1.64. The lower 5% of the simulated statistic is at −2.88 (the tabulated critical value for the regression with a constant and this length is −2.86; the difference is the noise of 4,000 simulations), and not at the normal's −1.64. Anyone who uses the Student's t threshold will declare 45.3% of random walks stationary: almost half. `statsmodels.tsa.stattools.adfuller` already uses the correct critical values (MacKinnon's) and chooses the number of lags $p$ by information criterion (`autolag="AIC"`). ## Two opposite nulls: ADF and KPSS The ADF has a design problem: the null is the unit root, so “not rejecting” can mean that the series is a random walk or that the sample is too short to see it. The **KPSS** test (Kwiatkowski, Phillips, Schmidt and Shin, 1992) swaps the roles: its null is that the series **is** stationary around a constant, and it rejects it when the cumulative sum of the deviations grows too much. Using both together gives more information than either one separately. ```python import warnings from statsmodels.tsa.stattools import adfuller, kpss from statsmodels.tools.sm_exceptions import InterpolationWarning # kpss warns when the statistic falls outside its table of # p-values (it clips them to 0.01 or 0.10): the warning is expected warnings.filterwarnings("ignore", category=InterpolationWarning) def contrastes(x): x = pd.Series(x).dropna() a = adfuller(x, regression="c", autolag="AIC", result_object=True) k = kpss(x, regression="c", nlags="auto", result_object=True) return {"n": len(x), "adf": a.statistic, "p_adf": a.pvalue, "kpss": k.statistic, "p_kpss": k.pvalue} cont = pd.read_parquet("futuros2_continuos.parquet") ida = np.log(cont[cont["mercado"] == "IDA"] .set_index("fecha")["cierre_ajustado"]) pd.DataFrame({"IDA log price": contrastes(ida), "IDA return": contrastes(ida.diff())}).T.round(3) ``` ```output n adf p_adf kpss p_kpss IDA log price 6539.0 0.176 0.971 10.817 0.01 IDA return 6538.0 -26.296 0.000 0.264 0.10 ``` With `result_object=True`, both functions return an object with names (`statistic`, `pvalue`, `critical_values`) instead of a tuple whose length changes with the arguments; statsmodels warns with a `FutureWarning` that this will be the default behavior, and asking for it now silences it. `kpss` returns p-values interpolated in a table that runs from 0.01 to 0.10; outside that range it clips them and emits an `InterpolationWarning`. The warning does not signal an error, but that the real p-value is below 0.01 or above 0.10; only that category is silenced so that the output is clean. The KPSS critical values with a constant are 0.347 (10%), 0.463 (5%) and 0.739 (1%). The log price of IDA, a fictional equity index built as a random walk with drift, gives an ADF of 0.18 (p = 0.97): the unit root is not rejected. The KPSS is 10.82, far above 0.463: stationarity is rejected. The two say the same thing in opposite languages. The daily return gives the opposite: ADF of −26.3 (p ≈ 0) and KPSS of 0.26, below the threshold. It is the textbook case. ## Spreads with a known answer The pairs in `pares2.parquet` make it possible to go further. All their relationships are in logarithms: in pairs A, B and C, the spread $u = \ln y - \alpha - \beta \ln x$ is a **stationary** Ornstein-Uhlenbeck process with a half-life of 5, 15 and 40 sessions (a percentage, with a standard deviation of 2% to 4.5%); in E, the same spread is a **random walk**, even though the two assets have a return correlation of 0.86. Here the true $\alpha$ and $\beta$ from `pares2_verdad.csv` are used: estimating them changes the critical values (Engle-Granger, lesson 2.1.5). ```python pares = pd.read_parquet("pares2.parquet").pivot( index="fecha", columns="activo", values="precio") verdad = pd.read_csv("pares2_verdad.csv").set_index("par") def spread(par): # all relationships are in logarithms (formulacion = "log") v = verdad.loc[par] y, x = pares[v["activo_y"]], pares[v["activo_x"]] return np.log(y) - v["alfa"] - v["beta"] * np.log(x) tabla = {} for par in ["A", "B", "C", "E"]: vm = verdad.loc[par, "vida_media_sesiones"] s = spread(par) tabla[f"{par}, half-life {vm:g}"] = contrastes(s) tabla[f"{par}, last year"] = contrastes(s.iloc[-252:]) pd.DataFrame(tabla).T.round(3) ``` ```output n adf p_adf kpss p_kpss A, half-life 5 3018.0 -14.929 0.000 0.108 0.100 A, last year 252.0 -4.251 0.001 0.287 0.100 B, half-life 15 3018.0 -8.975 0.000 0.385 0.084 B, last year 252.0 -2.825 0.055 0.133 0.100 C, half-life 40 3018.0 -5.562 0.000 0.826 0.010 C, last year 252.0 -1.643 0.461 0.333 0.100 E, half-life nan 3018.0 -2.012 0.281 4.775 0.010 E, last year 252.0 -2.371 0.150 0.586 0.024 ``` With the full twelve years (3,018 sessions), the ADF gets all four right: it rejects the unit root in A, B and C (statistics of −14.9, −9.0 and −5.6) and does not reject it in E (p = 0.28). The KPSS gets A and E right, but rejects the stationarity of C (0.826 > 0.739, p ≤ 0.01), which is stationary. With a single year of data, the picture changes: the ADF no longer rejects either in B (p = 0.055) or in C (p = 0.46). With a half-life of 40 sessions, a year contains about six reversion cycles, and that is not enough to tell them apart from a walk. ### The four combinations | ADF rejects (unit root) | KPSS rejects (stationary) | Reading | Example in the data | |---|---|---|---| | Yes | No | Stationary | IDA return; spread A | | No | Yes | Unit root | IDA price; spread E | | Yes | Yes | Stationary but very persistent, with breaks or with long memory | Spread C, 12 years | | No | No | The sample does not have enough information | Spread C, last year | *Fig. — KPSS statistic (horizontal axis, log) against ADF (vertical) for IDA's price and return and for the true spreads of four pairs, with 12 years and with the last year.* Plane split into four quadrants by the 5% critical values (KPSS 0.463 and ADF −2.86). Bottom left, stationary: IDA's return (ADF −26.3), spreads A and B with 12 years and A with one year. Top right, unit root: IDA's price (KPSS 10.8) and spread E. Bottom right, both reject: spread C with 12 years, highlighted (ADF −5.6, KPSS 0.83). Top left, not enough data: spreads B and C with one year. The top two rows are the clear answer. The bottom two say that the test cannot answer with these data, and they are more frequent than it seems with short or persistent series, which are exactly the ones of interest for mean reversion. ```mermaid flowchart TB A{ADF rejects?} -- yes --> K1{KPSS rejects?} A -- no --> K2{KPSS rejects?} K1 -- no --> E[Stationary] K1 -- yes --> P[Very persistent] K2 -- yes --> U[Unit root] K2 -- no --> M[Not enough sample] ``` ```python h = spread("H") ruptura = verdad.loc["H", "fecha_ruptura"] pd.DataFrame({"full": contrastes(h), "before the break": contrastes(h[:ruptura].iloc[:-1]), }).T.round(3) ``` ```output n adf p_adf kpss p_kpss full 3018.0 -1.448 0.559 6.643 0.010 before the break 1875.0 -5.702 0.000 0.407 0.074 ``` ## Differencing: what you gain and what you lose Differencing once ($d = 1$) turns the price into a return and solves stationarity. The cost is **memory**: today's return says nothing about the level the price is at. It is measured with the correlation between the transformed series and the original: ```python r = ida.diff().dropna() print(f"correlation of the return with the log price: " f"{np.corrcoef(r, ida.loc[r.index])[0, 1]:.3f}") # over-differencing a spread that was already stationary ds = spread("A").diff().dropna() print(f"lag-1 autocorrelation of spread A: " f"{spread('A').autocorr(1):.3f}; of its difference: " f"{ds.autocorr(1):.3f}") ``` ```output correlation of the return with the log price: 0.025 lag-1 autocorrelation of spread A: 0.865; of its difference: -0.055 ``` The return has a correlation of 0.025 with the price level: all the information about where the market is has disappeared. And differencing what was already stationary damages the series: spread A has an autocorrelation of 0.865 at lag 1 (the theoretical Ornstein-Uhlenbeck value is $0.5^{1/5} = 0.871$); its difference has −0.055, close to the value of an over-differenced AR(1) ($-(1-\phi)/2 = -0.065$): the positive persistence that revealed the reversion has disappeared and what is left is a series with almost no memory. The rule: difference just enough, not by reflex. ## Fractional differencing Is there anything between $d = 0$ (the price, with all its memory and non-stationary) and $d = 1$ (the return, stationary and without memory)? Yes: the difference operator $(1-B)^d$, with $B$ the lag operator ($B\,y_t = y_{t-1}$), admits any real $d$. Its binomial expansion gives an infinite weighted sum of the present and the past, $$ (1-B)^d\, y_t = \sum_{k=0}^{\infty} w_k\, y_{t-k}, \qquad w_0 = 1,\quad w_k = -w_{k-1}\,\frac{d-k+1}{k} $$ With $d = 1$ the weights are $1, -1, 0, 0, \ldots$: the usual difference. With $0 < d < 1$ all the past weights are negative and decay slowly, like a power: the series remembers the level for a long time. ```python def pesos(d, umbral=1e-4, kmax=100_000): w = [1.0] for k in range(1, kmax): wk = -w[-1] * (d - k + 1) / k if abs(wk) < umbral: break w.append(wk) return np.array(w) for d in [0.1, 0.3, 0.5, 1.0]: w = pesos(d) print(f"d={d:.1f}: {len(w):4d} weights; first " f"{np.round(w[:5], 3).tolist()}") ``` ```output d=0.1: 503 weights; first [1.0, -0.1, -0.045, -0.028, -0.021] d=0.3: 388 weights; first [1.0, -0.3, -0.105, -0.06, -0.04] d=0.5: 200 weights; first [1.0, -0.5, -0.125, -0.062, -0.039] d=1.0: 2 weights; first [1.0, -1.0] ``` *Fig. — Fractional difference weights, in absolute value, for the first 20 lags at four values of d (log scale). With d = 1 there is a single weight, −1 at lag 1.* Four declining curves with the lag on the horizontal axis, from 1 to 20, and the absolute weight on a log scale. They start at d (0.1, 0.25, 0.5 and 0.75) at lag 1 and fall slowly, like a power law: at lag 20 they are between 0.001 and 0.005. The d = 0.25 curve, highlighted, is the minimum d that makes IDA's price stationary. The **fixed-window** version (*fixed-width window fractional differencing*, López de Prado, 2018, chapter 5) cuts the sum where the weights fall below a threshold (here $10^{-4}$) and applies the same window to every date. That way all values carry the same amount of history and none uses future data: the value on date $t$ only combines $y_t, y_{t-1}, \ldots$ ```python def ffd(x, d, umbral=1e-4): w = pesos(d, umbral) # “valid” convolution: the value at t uses y_t … y_(t-L+1) v = np.convolve(x.to_numpy(), w, mode="valid") return pd.Series(v, index=x.index[len(w) - 1:]) # check: d = 1 is the log return print(np.allclose(ffd(ida, 1.0), ida.diff().dropna())) x3 = ffd(ida, 0.3) print(f"d = 0.3: starts on {x3.index[0]:%Y-%m-%d}, " f"{len(ida) - len(x3)} sessions after the price") ``` ```output True d = 0.3: starts on 2001-07-17, 387 sessions after the price ``` Now $d$ is swept and, for each value, the ADF of the transformed series and its correlation with the log price are computed. Following López de Prado, the ADF uses a single lag (`maxlag=1`, with no automatic selection), so that all the $d$ values are compared with the same regression: ```python filas = [] for d in np.round(np.arange(0, 1.01, 0.05), 2): x = ffd(ida, d) a = adfuller(x, maxlag=1, regression="c", autolag=None, result_object=True) crit = a.critical_values filas.append({"d": d, "adf": a.statistic, "p": a.pvalue, "corr": np.corrcoef(x, ida.loc[x.index])[0, 1]}) barrido = pd.DataFrame(filas).set_index("d") d_min = barrido.index[barrido["adf"] < crit["5%"]].min() print(f"5% critical value: {crit['5%']:.2f}; minimum d: {d_min}") barrido.loc[[0.0, 0.1, 0.2, 0.25, 0.3, 0.5, 1.0]].round(3) ``` ```output 5% critical value: -2.86; minimum d: 0.25 adf p corr d 0.00 0.393 0.981 1.000 0.10 -0.877 0.795 0.998 0.20 -2.166 0.219 0.987 0.25 -3.152 0.023 0.977 0.30 -4.173 0.001 0.963 0.50 -12.092 0.000 0.845 1.00 -57.513 0.000 0.025 ``` The figure at the top draws the full sweep. The ADF falls continuously as $d$ rises and crosses the critical value (−2.86) at $d = 0.25$ (ADF of −3.15; with $d = 0.20$ it is still −2.17); with that $d$, the series is stationary according to the ADF and keeps a correlation of 0.977 with the log price. The return ($d = 1$) is far more stationary than necessary (ADF of −57.5) and has lost all its memory (0.025). Between the two there is a range of series that preserve information on the level without the problems of a non-stationary regressor. ## What this changes in a system | Situation | What to test | What to do if it fails | |---|---|---| | Reversion rule on a price | ADF and KPSS of the price | There is no mean to return to: look for a spread (lessons 2.1.5 and 2.3.3) | | Spread of a pair | ADF with Engle-Granger critical values | Longer sample; watch for breaks (2.1.7) | | Model feature (level, moving average) | ADF of the feature | Return, z-score or fractional difference with minimum $d$ | | Regression of a return on a level | Stationarity of the regressor | Difference or use robust errors (lesson 2.1.1) | Three warnings. The first, that “not rejecting” is not “accepting”: an ADF that does not reject with one year of data does not prove that the spread is a random walk. The second, that the tests in this lesson measure stationarity **of the past**: pair H was stationary until it stopped being so. The third, that sweeping $d$, windows or pairs until one passes the ADF is a multiple search like any other (lesson 2.1.8). ## Exercises 1. Repeat the sweep of $d$ with the gold (`MEA`) and light crude (`ENA`) futures. What minimum $d$ comes out for each one and what correlation does it keep? **Solution.** ```python def d_minima(serie, crit=-2.86): for d in np.round(np.arange(0, 1.01, 0.05), 2): x = ffd(serie, d) adf = adfuller(x, maxlag=1, regression="c", autolag=None, result_object=True).statistic if adf < crit: return d, adf, np.corrcoef(x, serie.loc[x.index])[0, 1] for m in ["IDA", "MEA", "ENA"]: s = np.log(cont[cont["mercado"] == m] .set_index("fecha")["cierre_ajustado"]) d, adf, c = d_minima(s) print(f"{m}: minimum d {d:.2f}, ADF {adf:.2f}, correlation {c:.3f}") ``` ```output IDA: minimum d 0.25, ADF -3.15, correlation 0.977 MEA: minimum d 0.15, ADF -3.05, correlation 0.981 ENA: minimum d 0.10, ADF -3.40, correlation 0.988 ``` Crude (0.10) and gold (0.15) pass with a smaller $d$ than the index (0.25) and keep a somewhat higher correlation with their level (0.988 and 0.981 versus 0.977). It is what you would expect: their prices revert, very slowly, to an anchor, and the index's does not. The minimum $d$ is not a constant of the market: it depends on the persistence of each series, on its roll adjustment and on the period. 2. The ADF has more power the longer the sample. For spread B (half-life of 15 sessions), compute the ADF p-value with the last 126, 252, 504 and 1,008 days. From what length does it reject the unit root at 5%? **Solution.** ```python sb = spread("B") for n in [126, 252, 504, 1008]: p = adfuller(sb.iloc[-n:], regression="c", autolag="AIC", result_object=True).pvalue print(f"{n:5d} sessions: p = {p:.4f}") ``` ```output 126 sessions: p = 0.2449 252 sessions: p = 0.0548 504 sessions: p = 0.0090 1008 sessions: p = 0.0000 ``` With half a year and with one year, the ADF does not reject at 5%; with two years, it does. A spread that reverts in 15 sessions needs on the order of several hundred sessions for the test to tell it apart from a random walk. For a pairs strategy, that is a real limit on how much you can re-estimate with short windows. 3. A colleague proposes a reversion rule: buy IDA when its price is 10% below its 200-session mean. What do the tests in this lesson say about that rule, and what would you test instead? **Solution.** The price of IDA has a unit root (ADF of 0.18, KPSS of 10.82): there is no mean it should return to. The distance to the 200-session mean can be stationary, because the moving average follows the price, but being stationary does not imply that it predicts anything: it is an oscillator, and what has to be tested is whether its value today predicts the return of the following days, with errors robust to the overlap (lesson 2.1.1) and out of sample (lesson 2.5.2). In these data, the known answer says that index futures do not revert to any anchor (only commodities, currencies and bonds do, and slowly) and that what there is is a small momentum, so the rule should fail. ## Self-assessment 1. The ADF of the log price of IDA gives a statistic of 0.18 and a p-value of 0.97. What do you conclude? - A. That the unit root cannot be rejected: the price behaves like a random walk and does not return to any level - B. That the price is stationary - C. That the price has a deterministic trend - D. That more lags should be used *Answer A: That the unit root cannot be rejected: the price behaves like a random walk and does not return to any level.* The null of the ADF is the unit root. A high p-value means the data are compatible with it. The KPSS, whose null is the opposite, rejects it strongly (10.82 versus a critical value of 0.46): the two agree that the price is not stationary. 2. Why does the ADF not use the critical values of Student's t? - A. Out of habit - B. Because the ADF is a two-sided test - C. Because returns have fat tails - D. Because under the unit root null the regressor (yesterday's level) is not stationary and the t-statistic follows the Dickey-Fuller distribution, shifted to the left: the 5% critical value is −2.86, not −1.65 *Answer D: Because under the unit root null the regressor (yesterday's level) is not stationary and the t-statistic follows the Dickey-Fuller distribution, shifted to the left: the 5% critical value is −2.86, not −1.65.* In the lesson's simulation, with 4,000 random walks, the lower 5% of the statistic is at −2.88 (the tabulated value is −2.86); with the normal's −1.64 you would reject the unit root in 45.3% of the random walks. 3. The spread of pair C (true half-life of 40 sessions) rejects both nulls: the ADF says stationary and the KPSS says non-stationary. How is it interpreted? - A. That the data have an error - B. That it is a random walk - C. That the series is stationary but very persistent: the KPSS, with its autocorrelation correction, mistakes slow reversion for a unit root - D. That it has to be differenced twice *Answer C: That the series is stationary but very persistent: the KPSS, with its autocorrelation correction, mistakes slow reversion for a unit root.* The known answer is a stationary Ornstein-Uhlenbeck. With a half-life of 40 sessions, the series spends months far from its mean and the KPSS takes it for drift. The “both reject” combination calls for looking at persistence, structural breaks or long memory. 4. The fractional difference with d = 0.25 of the log of IDA passes the ADF and keeps a correlation of 0.977 with the original. What advantage does it have over the return (d = 1)? - A. None - B. That it preserves information on the price level (correlation 0.977 versus 0.025) while being stationary: a model can use it without the problems of a non-stationary regressor - C. That it is easier to compute - D. That it does not need past data *Answer B: That it preserves information on the price level (correlation 0.977 versus 0.025) while being stationary: a model can use it without the problems of a non-stationary regressor.* Differencing with d = 1 makes the series stationary but erases almost all the relationship with the level. The fractional difference removes only the part of the memory that prevents stationarity. It is a tool for building features (level 3), not a strategy. 5. With the fixed window and a threshold of 0.0001, the series with d = 0.1 starts 502 sessions later than the price. Why? - A. Because of a pandas bug - B. Because each value is a weighted sum of the previous 503 observations (the weights take a long time to fall below the threshold when d is small): the first ones do not have enough history - C. Because holidays are discarded - D. Because ADF needs 500 observations *Answer B: Because each value is a weighted sum of the previous 503 observations (the weights take a long time to fall below the threshold when d is small): the first ones do not have enough history.* With a small d the weights decay slowly, so the window is long. It is the price of memory: each fractional data point depends on two years of prices. And it is causal: it only uses the present and the past. ## Key points - Stationary (in the weak sense) = mean, variance and autocovariances that do not depend on the date. Prices are not; returns and cointegrated spreads are. - ADF and KPSS have opposite nulls: using them together gives four combinations, and two of them (both reject or neither does) say that the sample or the model is not enough. - The Dickey-Fuller statistic does not follow a t: its 5% critical value is −2.86. With the normal's −1.64, 45% of random walks are declared stationary. - Power depends on persistence and on the sample: the spread with a 40-session half-life cannot be told apart from a random walk with one year of data. - Differencing with d = 1 buys stationarity at the price of memory (correlation 0.025 with the level); the fractional difference with d = 0.25 passes the ADF and keeps a correlation of 0.977. ## Resources ### Research - [Stochastic Processes - Random Walks](https://pythonparatrading.com/en/stochastic-processes/) · Research-0005 · 11 min ### Documentation - [statsmodels: adfuller](https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.adfuller.html) · statsmodels.org - [statsmodels: kpss](https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.kpss.html) · statsmodels.org ### Books - [Kwiatkowski, D., Phillips, P. C. B., Schmidt, P. and Shin, Y. (1992): Testing the null hypothesis of stationarity against the alternative of a unit root. Journal of Econometrics 54(1-3), 159-178](https://doi.org/10.1016/0304-4076(92)90104-Y) · doi.org - [Dickey, D. A. and Fuller, W. A. (1979): Distribution of the Estimators for Autoregressive Time Series with a Unit Root. Journal of the American Statistical Association 74(366), 427-431](https://doi.org/10.1080/01621459.1979.10482531) · doi.org - [Marcos López de Prado: Advances in Financial Machine Learning (Wiley, 2018), chapter 5, “Fractionally Differentiated Features”](https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086) · wiley.com ### Data - [Level 2 data: synthetic pairs with their true relationship (Parquet)](https://pythonparatrading.com/curso/datos/nivel-2/pares2.parquet) · Download · pares2.parquet Previous lesson: [2.1.1 · Hypothesis tests, p-values, bootstrap and regression with robust standard errors](https://pythonparatrading.com/en/python-trading-course/level-2/hypothesis-testing-bootstrap-robust-errors/) Next lesson: [2.1.3 · ARMA and ARIMA models: autocorrelation (ACF) and partial autocorrelation (PACF)](https://pythonparatrading.com/en/python-trading-course/level-2/arma-arima-models-acf-pacf/)