ML Arc Capstone — Forecasting Realized Volatility

One Problem, the Whole Toolkit

The capstone runs the whole machine-learning arc on one real problem — forecasting S&P 500 realized volatility — the way a desk actually would: engineered features, the full model roster, three escalating validation schemes, calibrated uncertainty, and an explanation of what drives the forecast. The target is next-day log realized volatility. Realized volatility is how much the market actually moved on a given day, computed from high-frequency returns rather than implied from option prices; the log is taken because volatility is bounded below by zero and its raw distribution is heavily right-skewed, while its logarithm is close to symmetric and roughly Gaussian, which is what squared-error loss assumes. Every tabular model sees the same engineered matrix, so the comparison is about models rather than about who got better inputs. One consequence worth carrying into the tables below: because the target is in logs, an RMSE (the root mean squared error, the typical size of a miss) of 0.33 means the forecasts are typically out by about a third of a log unit — a factor of roughly 1.4 in volatility terms, so a day whose true volatility is 1% is typically forecast somewhere between about 0.7% and 1.4%. These are not precise forecasts; the question is only which method is least imprecise.

The features encode the known structure of volatility: Corsi's HAR components (heterogeneous autoregressive: today's log realized volatility plus trailing weekly and monthly averages, on the theory that traders operate on daily, weekly and monthly horizons), short lags, return-based terms for the leverage channel, a rolling dispersion — and a fractionally-differenced log-price carried over from the Financial ML subsection, which injects a price-level signal the pure-volatility features lack.

The fractional order matters and the obvious rule picks the wrong one. Selecting the smallest $d$ that clears an ADF test (augmented Dickey–Fuller, the standard test for whether a series still wanders rather than reverting to a stable level) gives about 0.35 on this series, but that result still measures $I(0.63)$ — above the 0.5 boundary at which a fractionally integrated series becomes stationary at all. The order that genuinely crosses it is $d\approx0.50$, which is also the order maximising out-of-sample $R^2$ on this exact target. Two independent criteria, one answer.

The eleven features

Every tabular model receives the same eleven columns, so nothing in the comparison turns on feature access. Ten of them are built from the volatility series and the returns; one is not, and that one turns out to matter.

Two things about that matrix are worth noticing before the results. It is deliberately collinear — RV_d, RV_w, RV_m, the three lags and disp10 all measure recent volatility, and correlated predictors are exactly the setting in which unregularized least squares becomes unstable and ridge earns its place. And ten of the eleven are functions of the volatility path alone; fd_price is the sole carrier of information about the price level. HAR-RV, which sees only the first kind, therefore has no access to the signal the SHAP ranking puts second — which is the mechanical reason the roster beats it rather than a vague appeal to flexibility.

The Roster, Tested Rather Than Ranked

A ranking by RMSE says nothing about whether the gaps are real, so each model is tested against the leader with a Diebold–Mariano test on squared-error loss — which takes the two models' errors on the same days, forms the difference series, and asks whether its mean is distinguishable from zero once autocorrelation is allowed for. It behaves like a t-statistic, so beyond about ±1.96 the ordering is real and inside it the two models are tied however different their RMSEs look. That turns "the roster converges" from an impression into a claim that can be checked — and it does not entirely survive.

Four of the eight challengers are statistically indistinguishable from a plain ridge regression: the other two regularized linear models (elastic net, lasso), the Gaussian process at 0.3327, and the random forest. The other four are significantly worse, and one of them is the classical benchmark.

HAR-RV is not "matched but barely beaten" here — it loses by 0.0163 RMSE at $p<0.001$. What beats it is not a neural network but three lines of regularized regression on better inputs. Complexity bought nothing: XGBoost, the SVR (support-vector regression with an RBF kernel) and the MLP (multilayer perceptron, a small feedforward neural network) all lose to that ridge significantly. The features did the work, and the simplest model in the roster was enough to collect it.

The LSTM and Transformer are reported outside the ranking, deliberately. They are trained on the raw 22-day sequence in the deep-learning examples, on those notebooks' own splits, so placing them in the same bar chart would imply a like-for-like comparison that was never run. HAR is fitted in both setups and bridges them: 0.3473 there against 0.3462 here, a difference of 0.0011. That gap is an order of magnitude smaller than the effects being discussed, which is what makes the comparison worth making at all. On those notebooks’ 688 test days the LSTM scores 0.3465 against HAR’s 0.3473 — a Diebold–Mariano statistic of −0.35, so statistically tied, and reached with 4,513 parameters against HAR’s four. The Transformer scores 0.3531, behind HAR. Neither, therefore, comes close to the ridge’s 0.3299: the sequence models tie or lose to the classical benchmark that this page’s ridge beats by 0.0163 at p < 0.001. The raw 22-day sequence does not contain what the engineered matrix contains.

Where the Validation Slogan Needs Narrowing

The validation section produces a corrective to its own slogan. Shuffled cross-validation is measurably optimistic — about 0.0085 RMSE, some 3% — but on this problem that is the small effect.

The three honest schemes disagree with each other by roughly six times as much. They are honest in different ways and, crucially, they score different periods: a temporal split trains on the first 80% of the timeline and tests on the last 20%; walk-forward refits repeatedly on an expanding window and averages six successive out-of-sample stretches; purged and embargoed cross-validation tests on every period including the calm early years, deleting training rows whose label windows touch the test fold. Because volatility is far harder to forecast in some years than others, that choice of test period moves the number more than the choice of scheme does. With a one-day-ahead label there is almost nothing for purging to remove; purging earns its keep on multi-day labels, which is what the purged cross-validation example demonstrates in a setting where the truth is known by construction. Comparing RMSEs across validation schemes is meaningless unless they score the same period — a caveat worth more here than the leakage warning itself.

Uncertainty and Explanation

On uncertainty, three lessons converge. The Gaussian process returns a predictive distribution for free, but a fixed-noise GP is overconfident and covers 0.81 against a 90% target. Split-conformal drops the model-correctness assumption and does better at 0.84 — yet a volatility regime shift between the calm calibration window and the crisis test period still pushes it below nominal, because exchangeability is exactly what a non-stationary market breaks. On a genuinely exchangeable split it recovers to 0.88 — on 652 test points the standard error of an empirical coverage is 0.012, so that sits 1.9 standard errors below nominal — inside the noise, but not comfortably on target. The guarantee is sound; the assumption is what markets violate.

SHAP (SHapley Additive exPlanations, which splits each prediction into signed per-feature contributions that sum back to it) confirms the HAR persistence terms carry about half the total attribution — the model does rediscover what the classical model hard-codes. But it also puts the fractionally-differenced price second of eleven features, ahead of two of the three HAR components. That is the clearest evidence in the collection for the López de Prado feature transform, and it explains the ranking: HAR has no access to a price-level feature, and volatility history does not substitute for it.

Where this sits

The roster comes from Regularized & Kernel Learning, Trees and Ensembles and Neural Networks; the feature transform from Fractional Differentiation; the validation from Purged & Embargoed Cross-Validation; the interval from Conformal Prediction; and the attribution from Interpretability. The same volatility series is modelled classically in Realized Volatility and Recurrent Networks & LSTMs — both on different splits and losses, which is why their figures are not interchangeable with these.

Notebook

Downloads

References

• Corsi, F. (2009). A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174–196. — HAR-RV, the benchmark
• Diebold, F. X. & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics 13(3), 253–263. — the test that decides which gaps are real
• Andersen, T. G. & Bollerslev, T. (1998). Answering the skeptics: yes, standard volatility models do provide accurate forecasts. International Economic Review 39(4), 885–905. — realized volatility as the target
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. — fractional differentiation, purging and embargo
• Lei, J. et al. (2018). Distribution-free predictive inference for regression. JASA 113(523), 1094–1111. — split conformal
• Lundberg, S. M. et al. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence 2, 56–67. — TreeSHAP