The KV cache stores key and value tensors from previous transformer attention layers so the model does not recompute them on every generated token, dramatically speeding up autoregressive inference.
The KV cache is a memory structure that holds the key and value projections for every token already processed. During autoregressive generation, each new token only computes its own key and value; all previous pairs are read from the cache.
That reuse is why decoding is cheaper than prefill. Without the cache, generating one thousand tokens would require computing attention over the full sequence one thousand times. With the cache, the model performs one large initial prefill and then cheap per-token lookups. The cost is memory: the cache grows linearly with sequence length and batch size.
Think of it like this. Think of a researcher taking notes while reading a long report. On the first read they write down every key finding. For later questions they only read the new section and cross-reference their notes, rather than rereading the entire report from the start.
During the initial prompt, every token attends to every other token and the resulting key-value projections are stored. For each subsequent token, only its key and value are computed and appended. Attention then attends over the cached history plus the current token. This reduces per-token compute from quadratic to linear.
"KV cache is free." It costs GPU memory proportional to batch size, sequence length, layer count, and hidden dimension. "KV cache eliminates attention cost." It removes the key-value computation, but attention still scales with sequence length. "Larger batches do not affect cache size." Cache memory grows directly with batch size and sequence length.
Dramatically reduces decode latency, but consumes significant GPU memory. Long sequences or large batches can exceed cache capacity and force recomputation or eviction strategies.