Hyperparameter tuning is the search for the settings you choose rather than the ones the model learns. Learning rate, number of layers, regularization strength and batch size are all hyperparameters.
The distinction matters. Weights are learned from data during training. Hyperparameters are fixed before it starts, and getting them wrong can make a good architecture look useless.
Think of it like this. Think of baking. The recipe learns nothing; you set the oven temperature and the time. The same ingredients at the wrong temperature give you a very different result.
Types
- Grid search. Try every combination. Exhaustive, and the cost explodes with each parameter added.
- Random search. Sample combinations at random. Usually beats grid search for the same budget, because most parameters barely matter and random sampling covers the ones that do.
- Bayesian optimization. Model which settings look promising and try those. Efficient when each run is expensive.
- Early stopping based methods. Kill unpromising runs quickly and give the budget to the survivors.
Common mistakes
- "Tune everything at once." Learning rate usually dominates. Get it right first, and the rest often matters much less than expected.
- "Grid search is thorough, so it is better." For the same compute, random search typically finds better settings, because grid search wastes most of its budget on parameters that make no difference.
- "Tune on the test set." Then your final number is optimistic and you have no honest estimate left. Tune on validation.
- "The best settings transfer." They are tied to your data, your model size and your batch size.
