A transformer is a neural network design that reads an entire sequence at once, using attention to decide which parts matter to each other. It is the architecture behind nearly every modern language model.
Before it, models read text one word at a time, in order. That was slow to train and it made long-range connections weak. The transformer dropped the sequence entirely and looked at everything together.
Think of it like this. Think of reading a paragraph one word at a time through a slit, versus seeing the whole paragraph at once. With the whole thing in view you can connect the last sentence to the first immediately, instead of trying to hold it in mind.
How it works
- No recurrence. Tokens are processed together rather than in sequence, so training parallelises across hardware.
- Attention connects everything. Every token can look at every other token directly, however far apart.
- Multiple heads look for different things. One might track grammar, another subject and object.
- Positional encoding restores order. Attention alone has no notion of sequence, so position is added explicitly.
- Encoder and decoder. The encoder builds understanding, the decoder generates. Many models use only one half.
Trade-offs
- Cost grows quadratically with length. Every token attending to every other means doubling the input roughly quadruples the work. This is why context windows are limited and expensive.
- Data hungry. Transformers need far more training data than the architectures they replaced.
- Parallel training, sequential generation. Training parallelises beautifully. Producing output is still one token at a time.
Common mistakes
- "Transformers are only for text." They handle images, audio, video, and protein structures.
- "A transformer is a kind of RNN." They are fundamentally different. Removing recurrence was the whole point.
- "They are always enormous." Small transformers run on phones and in browsers.
