Transformers & Self-Attention

Attention Instead of Recurrence

The LSTM carried memory through a sequence one step at a time. A transformer throws recurrence out: every position looks directly at every other in a single parallel operation. There is no distance penalty — position 1 and position 1,000 are one step apart — and no sequential bottleneck. That combination, unlimited range plus parallelism, is why transformers scaled to the models behind modern AI.

The mechanism is scaled dot-product attention: each position emits a query, a key and a value; queries are compared to all keys by dot product; the scores are softmax-normalised into weights; the output is the weighted sum of values. The softmax weights are the attention — a learned, content-based decision about what each position should read from the rest. Built from scratch here in three matrix multiplies and a softmax, matching PyTorch's optimised kernel to 9×10⁻⁸.

Three additions make it the transformer used in practice. Multi-head attention runs several attention operations in parallel with separate projections, so different heads specialise. Positional encoding is not optional: attention is permutation-invariant — shuffle the inputs and the outputs shuffle identically — so a position signal has to be added or the model has no notion of order at all. And residual connections with layer normalisation keep deep stacks trainable.

The Long-Range Advantage, Measured

The advantage over recurrence shows cleanly in a controlled test. Each sequence contains random values with one position marked; the task is to output the marked value, which requires reaching back an arbitrary distance. Averaged over three seeds — single runs are noisy enough to mislead at short lengths — the transformer's error is flat in sequence length while the LSTM's grows by more than an order of magnitude. That is the property that lets transformers model long documents, genomes and long price histories.

And the Limit

Then the honest counterweight. Added to the volatility race from the previous example — same series, same 22-day window, same 688 test days, benchmarks refitted rather than copied across — the transformer reaches RMSE 0.3531 against HAR-RV's 0.3473. A Diebold–Mariano test puts that gap right on the 1.96 boundary, and it moves to either side depending on the training seed. The honest reading: at best level with a four-coefficient regression, quite possibly a little behind, and weaker than the LSTM's clean tie (DM −0.35). It does not win.

How it loses, not just that it loses

The proportions-vs-predictions view says something the RMSE column cannot, and it is the more useful result. Binned into deciles of the forecast, the transformer's gaps run −0.185 to +0.066 against HAR's −0.100 to +0.060, and the slope of actual on predicted is 0.859 where HAR manages 0.929 and the LSTM 0.954.

That is a Mincer–Zarnowitz regression, and the benchmark is 1.0: an optimal forecast satisfies $\operatorname{cov}(y,\hat y)=\operatorname{var}(\hat y)$, which fixes the slope at 1 however much the forecast shrinks. Worth stating because the obvious shortcut is wrong — a good forecast should have smaller spread than the outcome, since it is a conditional mean, so comparing standard deviations proves nothing. Here HAR's predictions have sd 0.384 against the outcome's 0.497 and are nonetheless close to unbiased.

A slope below 1 therefore means the forecast varies more than its own information content justifies — it over-reacts, running too low where it predicts low and too high where it predicts high. All three models sit below 1, so all three over-react slightly, but the transformer at 0.859 is furthest from the benchmark. It does not lose to HAR by a small margin of random error; it loses in a systematic direction, exaggerating swings the data does not support. RMSE says the forecast is slightly worse; the slope says how it is worse, and only the second tells you whether the errors arrive as false alarms or missed ones.

That is not a failure of attention but attention being asked for something the problem does not contain. Section 3 showed its advantage is long-range recall. A 22-day window of one strongly autocorrelated series has no long-range structure left to recover — HAR's daily, weekly and monthly terms already span it — so all-to-all mixing buys nothing and costs parameters. The recurring lesson of this section holds: match the architecture to the structure of the data. Attention is transformative for language and long sequences, and unnecessary here.

Reading these against the volatility notebooks

One caveat about reading these numbers alongside the rest of the collection, which the notebook now states explicitly. Realized Volatility runs the same family — HAR, GARCH-t, Realized GARCH, stochastic volatility — on this identical series, and its figures are not interchangeable with these. It evaluates on the variance scale under QLIKE rather than RMSE on log-volatility; it splits 60/40 so its test period includes the 2008 crisis, where the 80/20 split here tests the calmer 2011–2013 stretch; and it fits GARCH from its own maximum-likelihood code rather than the arch package. It also carries two competitors absent from this race, Realized GARCH and stochastic volatility, with Realized GARCH edging HAR on QLIKE. So "the network beats GARCH" here means beating the weakest member of that family, on an easier period, under a different loss.

Notebook

References

• Vaswani, A. et al. (2017). Attention is all you need. NIPS 30. — the architecture
• Bahdanau, D., Cho, K. & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR. — attention before transformers
• Ba, J. L., Kiros, J. R. & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450. — what keeps the stack trainable
• Corsi, F. (2009). A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174–196. — the four coefficients attention could not beat
• Diebold, F. X. & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics 13(3), 253–263. — the test behind the verdict
• Patton, A. J. (2011). Volatility forecast comparison using imperfect volatility proxies. Journal of Econometrics 160(1), 246–256. — QLIKE, and why the volatility notebooks use it instead