Fractional Differentiation — Stationarity Without Erasing Memory

Differencing Without Throwing Everything Away

Machine-learning models want stationary inputs; prices are not stationary. The reflex is to difference them into returns, but integer differencing is a blunt instrument — it buys stationarity and destroys almost all of the series' memory in the same stroke. Returns are close to serially independent, and the level information, how far price sits from its own history, is gone.

Fractional differentiation resolves the trade-off by differencing a fractional amount $d\in(0,1)$: enough to reach stationarity and no more. The operator $(1-B)^d$ expands as a binomial series whose weights are computed here from scratch and match the closed form $(-1)^k\binom{d}{k}$ to 1.1×10⁻¹⁶. For $d=1$ the weights are just $[1,-1]$ — a first difference, no memory. For fractional $d$ they form a long, slowly decaying tail, and that tail is the memory.

The standard recipe is then to pick the smallest $d$ that clears an augmented Dickey-Fuller test. That recipe is what López de Prado prescribes, and it is the weakest link in the chapter — it rests the whole choice on one test, evaluated at a threshold, applied to precisely the process that test handles worst.

Where the Standard Recipe Breaks

Three problems, in increasing order of severity. The ADF p-value is not monotone near the threshold, so the selected $d$ depends on the grid: 0.35 on a 0.05 grid, 0.31 on a 0.01 one. The R companion, running the same recipe through CRAN's fracdiff::diffseries, picks 0.25 — an expanding window rather than a truncated fixed-width one, and a different ADF lag rule. Two correct implementations, three answers.

And the second test does not rescue the rule either. ADF and KPSS (Kwiatkowski–Phillips–Schmidt–Shin) test opposite nulls and are meant to be read together, a series counting as stationary only when both concur. They do concur at the two ends: everyone agrees the raw log price is non-stationary and that returns are. Between those ends they disagree at every $d$ from 0.35 to 0.70 — which is exactly the range in which ADF starts declaring stationarity, and exactly where the minimum-$d$ rule makes its choice. The agreement is confined to the cases nobody was in doubt about; the disagreement covers every case the rule is actually used to decide.

Simulating series whose true integration order is known settles which test to believe, and the answer is neither. ADF calls a genuinely non-stationary $I(0.8)$ process stationary; KPSS calls a genuinely stationary $I(0.2)$ process non-stationary. Both are built to detect a unit root against a short-memory alternative, and a fractionally integrated series is neither. The rule is not being applied carelessly — it is being applied with a tool that cannot answer the question.

Measuring It Instead of Testing It

Stationarity here is not a matter of testing at all. An $I(\delta)$ process is stationary exactly when $\delta<0.5$, so the right move is to estimate $\delta$ directly by log-periodogram regression — from scratch in Python, and as fdGPH in the R companion, which agree to three decimals.

The log price measures $I(0.926)$, a near-random-walk. Each application of $(1-B)^d$ reduces the order roughly one-for-one, as the theory requires, and the series crosses below 0.5 at about $d=0.50$ — not 0.35, and certainly not 0.25. At the ADF-selected $d=0.35$ the result still measures $I(0.63)$: it passes the test and is not stationary.

None of which damages the method. At the honest $d\approx0.50$ the series retains 0.58 correlation with the price level against 0.05 for returns — more than ten times the memory, genuinely stationary, a perfectly good feature. Fractional differencing does exactly what it claims. Choosing its order with a single unit-root test does not, and the correction costs about 0.2 of retained correlation to buy a series that is actually stationary.

One reading caution the R side makes visible: fracdiff()'s maximum likelihood returns exactly 0.500 on the log price, because ARFIMA (autoregressive fractionally integrated moving average, the model class that fractional differencing belongs to) is only defined for $d<0.5$, so the optimiser is pinned against its constraint. That is a boundary, not an estimate, and it should not be averaged in with fdGPH's 0.926 and fdSperio's 0.992 as a third opinion.

Does It Actually Predict Better?

Everything above concerns the properties of the transformed series. None of it establishes the claim the method rests on — that the result is a better feature. That is an empirical question, and it is the only test that can adjudicate when "more memory" and "more stationary" pull in opposite directions. The setup is deliberately plain: the same six lags of the same price series in three representations, the same model, scored out-of-sample on an expanding walk-forward. The target is next-day log realized volatility, which the rest of the collection shows to be genuinely forecastable. Alongside the score sits the diagnostic that explains it — the share of test feature values falling outside the range the model saw in training.

Raw levels do not merely underperform; they score negative, worse than predicting the training mean, with one fold at −1.69. And sweeping $d$ puts the result beyond doubt: predictive skill has an interior maximum at $d=0.50$ — the same order the integration-order analysis identified, arrived at by a completely independent route. The ADF rule's 0.35 gives up about a third of the achievable $R^2$. Both sides of the peak are explained: below it the feature is still non-stationary and the model is extrapolating; above it the memory that carried the signal has been differenced away.

Two different failures

Splitting the forecast error into a systematic part and a scatter part shows that the two rejected representations fail for opposite reasons — which is more informative than either score alone.

Levels fail through bias. The forecast tracks the shape of the series tolerably — correlation 0.55, moving about two thirds as much as the target — but sits systematically in the wrong place, and roughly a third of its squared error is that constant offset rather than scatter. The offset is unstable too, ranging from −0.75 to −0.02 across folds, because the feature drifts out of the range the model was fitted on and the intercept stops meaning what it meant. A forecast can correlate well with the truth and still be worse than useless if it is pointed at the wrong level. Returns fail through inertness. Only 4% of their error is bias — the forecast is well centred — but it barely moves: a quarter of the target's variation and a correlation of 0.07, which is a flat line with a wobble. Stationarity was bought by deleting the information that would have let it track anything. The fractionally differenced series is the only one avoiding both: lowest RMSE, highest correlation, and a forecast that moves about as much as the target does.

Two qualifications keep this honest. With a random forest — scale-invariant, and unable to extrapolate at all — returns edge out fracdiff (0.42 against 0.39), reading volatility clustering straight off lagged magnitudes. So the size of the advantage depends on the model class, while the failure of raw levels does not. And on next-day return direction, all three representations sit at 0.50: no transform rescues a target with nothing in it, which is the right negative control.

Where this sits

Fractional integration is the same long-memory mathematics as ARFIMA, which is why this connects directly to Volatility Persistence: Regimes vs Long Memory — that example asks whether apparent persistence is regimes or genuine long memory, and this one measures the memory parameter that settles it. The output is a feature for the models in Financial Returns Predictability and the capstone, and the validation it needs comes from Purged & Embargoed Cross-Validation. The pattern of a standard diagnostic failing on the class of process it is pointed at recurs in Calibration, where ECE needed a simulated floor before it could be read.

Notebooks

Downloads

References

• López de Prado, M. (2018). Advances in Financial Machine Learning, ch. 5. Wiley. — fractional differentiation and the minimum-d rule
• Hosking, J. R. M. (1981). Fractional differencing. Biometrika 68(1), 165–176. — the operator, and stationarity for d < 0.5
• Granger, C. W. J. & Joyeux, R. (1980). An introduction to long-memory time series models and fractional differencing. Journal of Time Series Analysis 1(1), 15–29.
• Geweke, J. & Porter-Hudak, S. (1983). The estimation and application of long memory time series models. Journal of Time Series Analysis 4(4), 221–238. — the log-periodogram estimator used here
• Diebold, F. X. & Rudebusch, G. D. (1991). On the power of Dickey-Fuller tests against fractional alternatives. Economics Letters 35(2), 155–160. — exactly the failure reproduced in the simulation
• Lee, D. & Schmidt, P. (1996). On the power of the KPSS test of stationarity against fractionally-integrated alternatives. Journal of Econometrics 73(1), 285–302. — the mirror-image failure