Batch Normalization

Batch normalization rescales the values passing between layers, using the mean and variance of the current batch. It makes deep networks train faster and tolerate higher learning rates.

Without it, the distribution of values reaching each layer shifts constantly as earlier layers update, so every layer is chasing a moving target. Normalizing steadies that.

Think of it like this. Think of a relay where each runner hands over at a wildly different speed each time. Standardising the handover means the next runner can practise one thing rather than adapting to chaos.

How it works

• Compute mean and variance across the batch for each feature.
• Normalize so values have roughly zero mean and unit variance.
• Scale and shift by two learned parameters, so the network can undo the normalization where it needs to.
• Inference uses running averages collected during training, since a single prediction has no batch to compute statistics from.

Trade-offs

• Faster training and higher learning rates, which is the main reason it is used.
• Batch size matters. Small batches give noisy statistics and degrade it badly. Below about 8 it becomes unreliable.
• Train and inference behave differently, which is a common source of confusing bugs.
• Awkward for sequences, which is exactly why layer normalization exists.

Common mistakes

• "It prevents overfitting." It regularises mildly as a side effect. It is a training stabiliser, not a regulariser.
• "Batch size does not matter." With very small batches the statistics are noise and results suffer.
• "It fixes internal covariate shift." That was the original explanation and it is now disputed. It works; why is less settled than the name suggests.