Regularization is any technique that deliberately holds a model back so it generalises instead of memorising. It trades a little accuracy on training data for better accuracy on new data.
That trade is the point. A model free to fit its training data perfectly usually will, including the noise, and then fails on anything new. Regularization makes that harder on purpose.
Think of it like this. Think of a student who is allowed to bring one index card into an exam rather than the whole textbook. The limit forces them to capture what generalises instead of copying everything down.
Types
- L2, or weight decay. Penalises large weights so no single input dominates. Shrinks weights toward zero without reaching it.
- L1. Penalises absolute size and drives some weights exactly to zero, which doubles as feature selection.
- Dropout. Randomly switches off units during training, so the network cannot rely on any one path.
- Early stopping. Halt when validation loss stops improving. Free, and often the most effective single measure.
- Data augmentation. Generate varied versions of your training data. More variety beats more copies.
How it works
- A penalty is added to the loss. Total loss becomes prediction error plus lambda times a penalty on the weights.
- Lambda sets the strength. Typical L2 values run 1e-5 to 1e-2.
- Dropout rates. Around 0.5 for fully connected layers, 0.1 to 0.2 for convolutional ones.
- Biases are normally left alone. Penalising them shifts predictions without limiting capacity.
Common mistakes
- "Regularization means L2." Dropout, early stopping and augmentation are all regularization. Early stopping is often the cheapest win.
- "More is better." Push it too far and you get underfitting, which is the opposite failure and just as bad.
- "Regularization and normalization are the same." Regularization constrains capacity. Normalization rescales values. Batch normalization does have a mild regularizing side effect, which muddies the naming.
- "Regularize the biases too." Standard practice is not to.
