The learning rate controls how big a step the model takes each time it updates. Gradient descent decides the direction; the learning rate decides the distance.
It is the setting most likely to break training outright. Too high and the loss explodes. Too low and it barely moves. Most other hyperparameters are far more forgiving.
Think of it like this. Think of finding the bottom of a valley in the dark. Huge strides carry you straight across and up the far side. Tiny shuffles get you there eventually, some time next week.
How it works
- It scales every update. New value equals old value minus learning rate times gradient.
- Typical starting points. 0.001 for Adam, 0.01 or 0.1 for plain SGD.
- Fine-tuning needs far less. Usually 1e-5 to 1e-4, ten to a hundred times smaller, because the model is already close.
- Schedules reduce it over time. Big steps early to cover ground, small steps later to settle.
How to spot it
- Loss becomes NaN or spikes. Learning rate far too high. This is the most common training failure there is.
- Loss bounces without improving. Still too high.
- Loss falls painfully slowly. Too low.
- Loss falls then flattens early. Try a schedule that decays, or a warm restart.
Common mistakes
- "Smaller is safer." Very small rates stall on plateaus and waste enormous amounts of training time.
- "One rate fits every layer." When fine-tuning, earlier layers usually want smaller rates than later ones.
- "It is the same as batch size." They interact, and they are separate settings. A larger batch gives a cleaner gradient, which supports a larger rate.
- "Adam means I can ignore it." Adam adapts per parameter around a base rate you still choose.
