Dropout randomly switches off a fraction of units during training. Each batch sees a different partial network, so no single unit can be relied on.
That unreliability is the point. If any unit might vanish, the network has to spread what it learns across many, and redundant representations generalise better than fragile ones.
Think of it like this. Think of a team where a random few people are out sick each day. Nobody can be the only person who knows something, because knowledge that lives in one head is knowledge the team keeps losing.
How it works
- A fraction of units is zeroed each training step, chosen randomly.
- A different subset every batch, so the network is effectively an ensemble of many thinner networks.
- Remaining activations are scaled up so the total signal stays comparable.
- It is switched off at inference. Every unit participates when you actually use the model.
Typical values
- Fully connected layers: around 0.5, meaning half the units drop.
- Convolutional layers: much lower, roughly 0.1 to 0.2, because convolutions already share weights.
- Input layers: rarely above 0.2 if used at all.
- Transformers: commonly 0.1.
Common mistakes
- "Leave it on at inference." Predictions become random. Frameworks handle this with train and eval modes, and forgetting to switch is a classic bug.
- "More dropout is safer." Too much and the network cannot learn at all, which is underfitting.
- "Use it everywhere." With batch normalization present, heavy dropout often hurts. Pick one primary regulariser.
