RLHF and alignment
How a model learns what is wanted: preference data, reward models, DPO as a shortcut, and the limits of the method.
The idea
After pre-training a model can continue text. After fine-tuning it can follow instructions. What is still missing is a sense of which of two possible answers is better.
For that, people are shown two answers and pick the better. From many such comparisons the model learns what is preferred.
The flow
- 01
Generate answer pairs
The model produces two or more answers to a request.
- 02
Have them rated
People pick the better one, following a written guideline.
- 03
Learn the preference
Either via a reward model or directly from the pairs.
- 04
Adapt the model
The preferred direction is reinforced without drifting too far from the base model.
The rating guideline
This is where the desired behaviour is actually specified, and it is often written casually.
- Rank the criteria: correctness before completeness before form.
- State explicitly how to handle uncertainty: is "I do not know" better than a guess?
- Do not allow length as a quality signal, or the model learns verbosity.
- Measure inter-rater agreement. Below 70 percent the guideline is too vague.
What measurably improves
| Property | Effect of alignment |
|---|---|
| Following instructions | clearly better |
| Keeping to a format | clearly better |
| Tone and politeness | clearly better |
| Refusing impermissible requests | clearly better |
| Factual correctness | small, sometimes worse |
| Admitting ignorance | only if explicitly rewarded |
The reward model
DPO
The decisive step is the observation that the optimal policy can be written in closed form through the reward model. Substituting that, the reward model drops out:
The advantage is considerable: no separate reward model, no reinforcement-learning loop, no sampling during training. What remains is supervised training on pairs, with correspondingly stable optimisation.
Reward hacking, concretely
| Observed shortcut | Cause in the rating |
|---|---|
| Answers keep getting longer | Length correlated with approval |
| Excessive bullet lists | Structure was read as quality |
| Agreeing with the asker's position | Politeness was rewarded |
| Avoiding clear statements | Errors were punished harder than evasion |
| Formulaic openings | They occurred often in preferred answers |
All five have been documented in released models. They are not optimisation failures but correct optimisation of an imprecise specification.
The regulatory link
For general-purpose models the AI Act requires, among other things, a description of the training and alignment procedures and an assessment of systemic risks. The rating guideline, the selection of raters and the measured agreement are therefore documentation artefacts, not merely internal working papers.
Related courses and sources
Hugging Face deep reinforcement learning course
Reward, policy and exploration in playable environments. Useful for understanding what actually happens when a language model is aligned.
For anyone wanting to understand what actually happens when a language model is aligned.
Training Language Models to Follow Instructions
How a text continuation engine becomes an assistant. The paper behind alignment from human feedback.
For anyone asking how a text continuation engine becomes an assistant.