Attention is how a model decides which parts of the input matter for the piece it is currently processing. It weighs every position against every other and lets the relevant ones dominate.
This is what lets a model resolve which noun a pronoun refers to, twenty words later. Rather than passing information along a chain, every position can reach every other directly.
Think of it like this. Think of reading a contract with a highlighter. For each clause you are checking, different earlier lines matter. Attention is choosing what to highlight, freshly, for every single word.
How it works
- Three vectors per token. A Query (what am I looking for), a Key (what do I offer), and a Value (what I actually contribute).
- Score every pair. Each Query is compared against every Key, producing a relevance score.
- Normalise with softmax. Scores become weights that add up to one.
- Blend the Values. The output is a weighted mix, dominated by whatever scored highest.
- Multiple heads run in parallel. Each learns to look for a different kind of relationship.
Trade-offs
- Quadratic cost. Comparing every pair means work scales with the square of sequence length, which is the central constraint on context size.
- Global reach, at a price. Any two positions can connect directly, but nothing is free: every connection is computed.
- Heads have diminishing returns. More heads capture more relationship types up to a point, then add cost without accuracy.
Typical values
- Complexity: O(n squared times d) in time, O(n squared) in memory, where n is sequence length.
- 2,048 tokens: over 4 million attention pairs to compute.
- 100,000 tokens: over 10 billion pairs. This is what the word quadratic actually costs.
- Flash Attention cuts memory to O(n) by tiling and recomputing, while compute stays quadratic. It is why long context became affordable.
- KV caching stores past keys and values during generation, so each new token does not recompute the whole history.
Common mistakes
- "Attention is memory." It is dynamic weighting, recomputed from scratch every time. Nothing is stored.
- "Attention solved long-range dependencies." It improved them enormously. Very long inputs still lose focus.
- "More heads is better." Past a point extra heads cost compute and add nothing.
