Perplexity measures how surprised a language model is by a piece of text. Lower is better, and it roughly means how many options the model was effectively choosing between at each token.
A perplexity of 10 means the model was about as uncertain as if picking uniformly among 10 possibilities. It is the standard number for comparing language models on the same data.
Think of it like this. Think of guessing the next word in a sentence someone is reading aloud. Perplexity is roughly how many words you were torn between. Being torn between two is confident. Being torn between fifty is not.
How it works
- Derived from the model's probabilities for each actual next token.
- It is the exponential of average cross-entropy loss, which is why training loss and perplexity move together.
- Lower means less surprised, and therefore better prediction.
- Only comparable on identical data and tokenization. Different test sets or tokenizers make the numbers meaningless against each other.
Common mistakes
- "Lower perplexity means a better assistant." It measures prediction, not helpfulness. RLHF often raises perplexity while making a model far more useful.
- "You can compare perplexity across models." Only with the same test data and the same tokenizer. Different tokenization changes the number without changing the model.
- "It measures factual accuracy." It measures how well the model predicted this specific text. Confidently predicting something false scores well.
