Inference

Inference is using a trained model to make a prediction on new data. The model does not change during inference. It only applies what it already learned.

Training and inference are two different jobs with different economics. Training happens rarely and costs a lot. Inference happens on every request forever, which is where the real bill accumulates.

Think of it like this. Think of the difference between studying for a licence and driving a car. The studying was expensive and finished. The driving happens every day and is what the licence was for.

How it works

• Weights are frozen. Nothing is learned. The parameters are fixed.
• New input arrives and gets the same preprocessing the training data received.
• A forward pass runs through the network and produces an output.
• Batch or real time. Batch processes many inputs on a schedule. Real time answers one request now, and is much harder to make fast.

Trade-offs

• Latency against cost. Faster responses mean better hardware or a smaller model.
• Accuracy against speed. Quantization and distillation trade a little accuracy for large speed gains.
• Training is once, inference is forever. Over a model's life, serving usually costs more than training it did.

Common mistakes

• "Inference is the fast part." A large model can be slow to run and that cost repeats on every single request.
• "Preprocessing does not matter." Input must be transformed exactly as it was during training. Any mismatch degrades results silently.
• "A deployed model stays good." The world moves and the data shifts underneath it. Accuracy decays without anyone changing a line of code.