Gradient descent is how most models learn. It repeatedly adjusts every parameter a small step in whichever direction reduces the loss.
The gradient says which way is downhill for each parameter. The learning rate says how big a step to take. Repeat that a few million times and the model has learned something.
Think of it like this. Think of walking down a hill in thick fog. You cannot see the bottom, so you feel which way the ground slopes and take a step. Repeat. You will reach a low point, though not necessarily the lowest one.
How it works
- Compute the loss on some training examples.
- Compute the gradient, which is the slope of the loss with respect to each parameter.
- Step downhill. New value equals old value minus learning rate times gradient.
- Repeat until the loss stops improving.
Types
- Batch. Uses the whole dataset for each step. Stable, and far too slow for large data.
- Stochastic (SGD). One example per step. Fast and noisy, and the noise sometimes helps it escape bad spots.
- Mini-batch. A small group per step, typically 32 to 256. What almost everyone actually uses.
- Adam and friends. Adaptive methods that keep a per-parameter learning rate. Usually the sensible default.
Common mistakes
- "It finds the best possible answer." It finds a local minimum. For non-convex problems there is no guarantee it is the global one, and in practice a good local minimum is usually fine.
- "Gradient descent and backpropagation are the same." Backpropagation computes the gradients. Gradient descent uses them. They are two halves of a step.
- "Smaller steps are safer." Too small and training stalls or takes forever.
- "Adam is always better than SGD." Adam converges faster with less tuning. Well-tuned SGD with momentum often generalises slightly better.
