Vaswani et al. (2017) Attention Is All You Need

Summary

Vaswani et al. propose the Transformer, a sequence-transduction architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Attention lets every output position attend directly to every input position, so the model is highly parallelizable and learns long-range dependencies easily. On two WMT-2014 machine-translation tasks the Transformer is both higher quality and much faster to train than the best recurrent/convolutional models, achieving 28.4 BLEU (English→German) and a new single-model state of the art of 41.0 BLEU (English→French) after 3.5 days on eight GPUs.

Key Claims

• Attention-only architecture. The Transformer replaces recurrent and convolutional layers with stacked self-attention and position-wise feed-forward layers in an encoder–decoder design.
• Scaled dot-product attention. $\text{Attention}(Q,K,V)=\mathrm{softmax}(QK^\top/\sqrt{d_k})V$; the $1/\sqrt{d_k}$ scaling prevents large dot products from saturating the softmax.
• Multi-head attention. Running $h$ attention functions in parallel over projected subspaces lets the model jointly attend to information from different representation subspaces.
• Self-attention advantages. Constant ($O(1)$) maximum path length between any two positions (vs. $O(n)$ for recurrence) improves long-range dependency learning, and computation parallelizes across positions.
• Positional encoding. Order information is supplied by parameter-free sinusoidal position encodings added to token embeddings.
• Empirical results. New state-of-the-art BLEU on WMT-2014 En→De (28.4, +2 over prior best incl. ensembles) and En→Fr (41.0), at a fraction of prior training cost; also generalizes to English constituency parsing.

Concepts Introduced or Extended

• Attention Mechanism

Entities Mentioned

• Ashish Vaswani
• Noam Shazeer
• Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (equal-contribution coauthors)

Quotes

"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely."

My Take

One of the most consequential papers in modern machine learning: it showed that the recurrence everyone assumed necessary for sequence modeling was optional, and that pure attention — parallelizable and $O(1)$ in path length — was not only sufficient but better. The engineering choices that made it work (the $\sqrt{d_k}$ scaling, multiple heads, sinusoidal positions, residual+layernorm blocks) are all still standard. Its relevance to this wiki is oblique but growing: the Transformer is the backbone of the LLMs reshaping every field, and self-attention has become a serious tool for time-series forecasting, where it competes with the classical and machine-learning methods catalogued elsewhere here. The honest caveat is the quadratic cost and data hunger — attention buys flexibility by discarding inductive bias, which is exactly why it needs scale to shine.