Reinforcement learning from human feedback trains a model to produce responses people prefer. Humans rank outputs, a reward model learns those preferences, and the language model is tuned against it.
It exists because helpfulness is easy to recognise and hard to define. Nobody can write the rule for a good answer, and most people can pick the better of two.
Think of it like this. Think of coaching a writer without a style guide. You cannot list the rules, and you can reliably say which of two drafts is better. Enough of those judgements and the writer learns your taste.
How it works
- Start from a pretrained model that can already produce fluent text.
- Collect human comparisons. People rank several responses to the same prompt.
- Train a reward model to predict which response a human would prefer.
- Tune the language model to score well against that reward model, usually with PPO or a simpler direct method.
Trade-offs
- Turns capability into usefulness. A base model completes text. RLHF is a large part of what makes it answer questions.
- Inherits the raters' biases. The model learns the preferences of whoever did the ranking.
- Reward hacking. The model optimises the reward model, which is only a proxy. Confident, agreeable, verbose answers often score well.
- Expensive. Human comparison data is slow and costly to collect.
Common mistakes
- "RLHF makes models truthful." It makes them produce answers people rate highly. A confident wrong answer often outranks an honest uncertain one.
- "It teaches new knowledge." It shapes behaviour and style. Facts come from pretraining or retrieval.
- "It is the only alignment method." Direct preference optimisation and constitutional approaches achieve similar ends with less machinery.
