Loss Function

A loss function measures how wrong a model's predictions are and turns that into one number. Training is the process of making that number smaller.

It is the only thing the model is actually optimising. Whatever you choose to measure is what it will get good at, so picking the wrong loss quietly teaches the wrong behaviour.

Think of it like this. Think of a score in golf. Lower is better, and every shot changes it. The player never sees the whole course at once, only whether the last decision moved the number in the right direction.

How it works

• It takes predictions and truth, and returns a single error value.
• Training minimises the average of that value across all examples.
• It has to be differentiable, because gradients are what tell each weight which way to move.
• The task decides the loss. Squared error for numbers, cross-entropy for categories.

Types

• Mean squared error. Regression default. Squares the error, so large mistakes dominate.
• Mean absolute error. Treats every unit of error the same, which makes it far less sensitive to outliers.
• Cross-entropy. The classification standard. Punishes confident wrong answers hard.
• Huber. Squared for small errors, absolute for large ones, giving you both behaviours.

Common mistakes

• "Loss and metric are the same." Loss guides training and must be differentiable. A metric measures success for humans and has no such constraint. Accuracy makes a fine metric and a useless loss.
• "Lower training loss means a better model." Loss can approach zero while the model memorises. Watch validation loss.
• "Squared error is the safe default." It lets a handful of outliers dominate the fit.
• "Use MSE for classification." Paired with a sigmoid it produces a landscape that is hard to optimise. Cross-entropy exists for this.