An attention mechanism lets a neural network compute each output element as a data-dependent weighted average of a set of value vectors, where the weights measure the relevance ("compatibility") of a query to each key. The Transformer (Vaswani et al. 2017) builds an entire sequence model out of attention alone — dispensing with recurrence and convolution — and is the architecture underlying modern large language models (Vaswani et al. 2017).
Key Ideas
Query–Key–Value retrieval. Each position emits a query q, and every position offers a key k and a value v. The output for q is ∑iαivi with weights αi=softmaxi(score(q,ki)) — a soft, differentiable lookup into an associative memory.
Scaled dot-product attention. With queries/keys of dimension dk packed into matrices, Attention(Q,K,V)=softmax(dkQK⊤)V. The 1/dk scaling counteracts the growth of dot products in high dimensions, which would otherwise push the softmax into vanishing-gradient regions.
Multi-head attention. Project Q,K,V into h lower-dimensional subspaces, run attention in parallel in each "head," and concatenate — letting the model attend to information from different representation subspaces (e.g. syntactic vs. semantic relations) at once.
Self-attention. When queries, keys, and values all come from the same sequence, every position can directly attend to every other, giving an O(1) path length between any two positions (versus O(n) for RNNs) — far better at learning long-range dependencies, and fully parallelizable across positions.
Positional encoding. Because attention is order-agnostic, position information is injected via (parameter-free) sinusoidal encodings added to the token embeddings, so the model can use sequence order.
The Transformer block. Stacked encoder/decoder layers, each = multi-head (self-)attention + a position-wise feed-forward network, wrapped in residual connections and layer normalization; the decoder adds masked self-attention (causal) and encoder–decoder (cross) attention.
Why It Matters
Replaced recurrence in sequence modeling. On WMT-2014 machine translation the Transformer beat prior recurrent/convolutional models (28.4 BLEU En→De) while training far faster because attention parallelizes where RNNs are inherently sequential.
The foundation of modern deep learning. Transformers are the backbone of BERT, GPT, and essentially all large language models, and have spread to vision, speech, and — increasingly — time-series forecasting, where self-attention captures long-range temporal structure.
Interpretability handle. Attention weights offer a (partial, contested) window into which inputs the model relies on.
Open Questions
Quadratic cost. Full self-attention is O(n2) in sequence length; efficient/sparse/linear-attention variants trade exactness for scalability.
Do attention weights explain? Whether attention maps constitute faithful explanations of model behavior is debated.
Inductive bias. Attention imposes weak structural priors, so Transformers are data- and compute-hungry relative to architectures with built-in locality.