Recurrent Networks & LSTMs — Forecasting Volatility

Sharing Weights Across Time

A convolutional network shares one filter across space; a recurrent network shares one set of weights across time. It reads a sequence step by step, carrying a hidden state that summarises everything seen so far, using the same $W_x, W_h, b$ at every step. That is the temporal analogue of spatial weight sharing, and it lets one network handle sequences of any length at a fixed parameter count. The recurrence is a few lines; built from scratch here, it reproduces PyTorch's nn.RNN hidden states to 8×10⁻⁸ once the weights are copied across.

Plain RNNs forget. Unrolled over $T$ steps the gradient is multiplied by $W_h$ $T$ times on the way back — eigenvalues below one and the signal vanishes, above one and it explodes. The LSTM adds a gated cell state: a memory line that information travels along with only gentle, gated modification rather than being squashed through a $\tanh$ at every step, so gradients survive long horizons.

A Real Forecasting Problem

The task is deliberately a real one: forecasting S&P 500 realized volatility, 2000–2013, about 3,460 trading days spanning the 2008 crisis. Volatility is the right target because it is genuinely predictable — log realized volatility has a one-day autocorrelation of 0.78, while daily returns sit near zero. Forecasting returns with an LSTM would be a fool's errand; forecasting volatility is a well-posed problem with serious classical competition.

Making the comparison fair

Setting up that competition fairly turned out to be the hard part, and it is worth stating because it is an easy mistake to make invisibly. A sequence model is trivially handicapped by an off-by-one in its windowing: if the input window ends on day $t-1$ while the label is day $t+1$, the network never sees day $t$ — the single most informative predictor, correlated 0.78 with the target — and is quietly solving a two-step-ahead problem while every benchmark solves a one-step-ahead one. That costs about 0.02 RMSE here, comfortably more than the gap being reported, and would have manufactured a decisive classical victory out of nothing. Corrected, all five models forecast the same 688 test days from the same information set.

The race

On that footing the race is close at the top and the result is a tie, not a win. HAR-RV scores 0.3473 and the LSTM 0.3465 — separated by 0.0009, which is not a result until it is tested. Because both forecast the identical test days, a Diebold–Mariano test on squared-error loss applies directly: DM = −0.35 against a critical value of 1.96. They are statistically indistinguishable.

The tie is the finding, because of what each pays for it. HAR-RV uses four coefficients; the LSTM uses 4,513 — about a thousandfold difference — to reach the same accuracy. Three hand-designed features (yesterday, last week, last month, a linear approximation to volatility's long memory) capture essentially everything a recurrent network can extract from this series on its own history. That is the tabular lesson from the first example transposed to time: deep learning wins where there is structure hand-crafted features miss, and on a single well-understood volatility series there is very little left over.

Why GARCH Trails, Precisely

GARCH trails at 0.4568, but the usual gloss — that returns-based models are simply worse for this — overstates what the numbers show. Its forecasts sit +0.259 above realized log-volatility on average, and that single constant accounts for 32% of its squared error. Two checks turn that from a curve-fit into a finding.

Is the offset real, or fitted? A bias estimated on the test set is an oracle quantity — it uses the answers. Re-estimated on the training period alone it comes to +0.221, close to the +0.259 the test set implies, and applying that correction — one an analyst could have made in advance — gives RMSE 0.3783 against the oracle version's 0.3764. The level shift is stable, so essentially all of the improvement survives without peeking.

Where does it come from? Partly the overnight gap, though only partly — worth being precise rather than hand-waving. Realized volatility is built from intraday returns while GARCH models close-to-close ones, and overnight moves are 23% of total daily variance here, which on its own implies a log offset of +0.128 — about half the observed +0.259. The remainder is the plain unconditional level gap between the series: sd(returns) = 1.32 against mean realized volatility 0.97, a ratio of 1.36. GARCH is not forecasting badly so much as forecasting a different quantity.

Corrected, GARCH scores 0.3783 — it beats the random walk (0.4206) and still trails HAR (0.3473), with correlation 0.66 against realized volatility. That ordering, GARCH ahead of the random walk, is exactly what Realized Volatility reports on its own scale. The raw ranking here was the odd one out across the collection, and the level offset is why. None of this touches the LSTM's own numbers or its tie with HAR — it changes where GARCH sits, and what the comparison means.

Where this sits

This is the deep-learning entry in a lineup the collection already covers from the Bayesian side: Bayesian GARCH, Bayesian Asymmetric GARCH (GJR), Volatility Persistence: Regimes vs Long Memory and Copula-GARCH all model this same quantity. The recurrence's weakness — strictly sequential processing, and fading memory of distant steps — is what the next example removes, replacing recurrence with attention so every position can look directly at every other.

Notebook

Downloads

References

• Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural Computation 9(8), 1735–1780. — the gated cell state
• Corsi, F. (2009). A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174–196. — HAR-RV, the four coefficients that tie the network
• Andersen, T. G., Bollerslev, T., Diebold, F. X. & Labys, P. (2003). Modeling and forecasting realized volatility. Econometrica 71(2), 579–625. — the realized measure being forecast
• Diebold, F. X. & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics 13(3), 253–263. — the test that turns a 0.0009 gap into a verdict
• Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics 31(3), 307–327. — the returns-based benchmark
• Bengio, Y., Simard, P. & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166. — the vanishing gradient the LSTM answers