Batch size is the number of training examples processed before the model updates its weights, trading noise and speed against gradient stability.
Batch size controls how many examples contribute to each gradient update. Small batches add noise that can help generalization but make training less stable. Large batches give cleaner gradients and faster throughput but can reduce the beneficial noise that prevents narrow minima.
That single setting changes the shape of the optimization path. Tiny batches of one example, or true stochastic gradient descent, bounce around the loss surface and can escape shallow local minima. Large batches smooth the gradient and reach minima faster per step, but they may converge to sharper points that generalize worse. In practice, practitioners tune batch size alongside learning rate because the two interact.
Think of it like this. Think of walking downhill in the dark. Small batches are short impulsive steps that may zigzag but can avoid small holes. Large batches are long smooth strides that travel faster but may step into a narrow pit.
The dataset is divided into chunks of batch size. For each chunk, the model computes a loss, backpropagates gradients, and updates weights. A full pass through the entire dataset is one epoch, containing dataset size divided by batch size steps. Very large batches may use gradient accumulation to simulate even larger effective batches without extra memory.
"Larger batches always train faster." Throughput improves, but convergence behavior changes and may require learning rate retuning. "Batch size does not affect accuracy." It affects optimization dynamics and can shift generalization. "One global batch size fits every GPU." Memory limits, pipeline efficiency, and mixed precision often force different sizes per hardware.
Smaller batches add regularization through noise and fit on limited memory; larger batches improve hardware utilization and gradient stability but may need learning rate scaling and regularization adjustments.