Post-Training and Alignment

From text predictor to assistant

In early 2022 OpenAI compared two models on thousands of real user requests. One was GPT-3, with 175 billion parameters. The other had 1.3 billion, over a hundred times smaller. People preferred the answers from the small one. The difference was not size or data. It was a few extra stages of training on human feedback, and those stages are what this lesson is about.

What is post-training?

A freshly pretrained language model, a base model, has read trillions of words and learned to continue any document plausibly. That is not the same as being useful. Ask it a question and it may continue with more questions, because in its training data a question is often followed by another question: in a quiz, a forum thread, an exam paper.

Post-training is everything that happens after pretraining to turn that document-continuer into an assistant: something that reads your text as a request, answers it, stays honest about what it does not know, refuses what it should refuse, and stops when it is done. It uses a tiny fraction of the compute of pretraining, but it decides almost everything about how the model behaves.

How does it work?

Modern post-training is a pipeline of stages, each using a different kind of data:

Supervised fine-tuning shows the model examples of the conversations you want. Preference tuning goes further: rather than imitating one written answer, the model learns from judgements about which of two answers is better, through reinforcement learning from human feedback (RLHF), direct preference optimisation (DPO) or AI feedback guided by a written constitution. Since 2024 a fourth stage has become central for frontier models: reinforcement learning on verifiable rewards, where the model practises problems whose answers can be checked automatically. That stage is what produced reasoning models.

Why does it matter?

Because what you reward is what you get. Every behaviour people notice in a chat assistant, from its tone to its willingness to admit uncertainty to its habit of opening with “Great question!”, was shaped by a choice about which data to show it and which answers to reward. The same machinery that makes models helpful can also make them flatter users, pad answers or game their own training signal, and in April 2025 one of those failures reached hundreds of millions of people. You will see how by the end of the lesson.

Key takeaways

• A base model continues documents; post-training turns it into an assistant that treats text as a request to answer.
• The usual recipe is supervised fine-tuning, then preference tuning (RLHF, DPO or AI feedback), then reinforcement learning on problems with checkable answers.
• Post-training uses little compute compared with pretraining but decides most of a model’s behaviour, including its failure modes.

Supervised fine-tuning

The simplest way to teach a model to behave like an assistant is to show it an assistant: thousands of example conversations, written or checked by people, and train it to reproduce them.

What is supervised fine-tuning?

Supervised fine-tuning (SFT), also called instruction tuning, continues training the base model with exactly the same next-token objective as pretraining, but on a small, curated dataset of demonstrations: a prompt and an ideal response. For InstructGPT, contractors wrote responses to about 13,000 prompts drawn from real API requests and from prompts the labellers wrote themselves.

How does it work?

Each conversation is flattened into one token sequence using a chat template: special tokens mark where the system instructions, the user's message and the assistant's reply begin and end. The model reads the whole sequence, but the loss is usually computed only on the assistant's tokens. The model learns to produce answers, not to predict what users will type.

The special end-of-turn token matters more than it looks. Training the model to emit it is how the assistant learns to stop, and at inference time the chat software stops generating when it appears. The same template is used at inference, which is why a model can behave oddly if it is prompted with the wrong one.

A surprisingly small amount of data goes a long way. The LIMA study fine-tuned a 65-billion-parameter LLaMA model on just 1,000 carefully chosen examples and found its answers were judged equivalent to or better than GPT-4's in 43% of comparisons. The authors proposed the superficial alignment hypothesis: almost all knowledge and ability is learned in pretraining, and fine-tuning mostly teaches which format and style to use when talking to people.

Why is SFT not enough on its own?

SFT is imitation, and imitation has three built-in limits.

• It is capped by the demonstrators. The model learns to write like the people who wrote the examples, including their mistakes. It is hard to demonstrate an answer that is better than you could write.
• It only shows good answers. The model never sees a bad answer labelled as bad, so it learns little about what to avoid, and it never practises recovering from its own errors.
• It can teach bluffing. If a demonstration states a fact the model never learned in pretraining, the model is being trained to produce confident text it cannot back up. Imitation rewards sounding like the expert, not knowing what the expert knows.

Judging is easier than writing. Most people cannot write a perfect sonnet, but they can tell which of two sonnets is better. That observation is the basis of the next stage.

Key takeaways

• SFT trains the base model on example conversations with the ordinary next-token loss, usually counting only the assistant’s tokens.
• A chat template with special tokens marks the roles; learning to emit the end-of-turn token is how the model learns to stop.
• SFT teaches format cheaply (LIMA used 1,000 examples), but it is capped by the demonstrators and never learns from bad answers.

Learning a reward from comparisons

A reward model turns a pile of “A is better than B” judgements into a function that can score any answer, including answers no person has ever seen.

What is a reward model?

A reward model is a neural network that reads a prompt and a response and outputs a single number, the reward. In practice it is usually a copy of the language model itself with the final layer swapped for one that outputs a scalar. It is trained on comparisons: for one prompt, several answers are sampled from the model, and people rank them. The idea of learning a reward from human comparisons comes from Christiano and colleagues in 2017, who used it to train agents in video games and simulated robots on behaviours that were hard to specify by hand.

How does it work?

Comparisons are turned into a training signal with a model from 1952 for ranking things by paired contests, the Bradley–Terry model. Each answer has a hidden score r. The probability that answer A is preferred to answer B depends only on the difference of the scores:

The Bradley–Terry model. σ is the logistic function σ(z) = 1 / (1 + e^−z). Chess Elo ratings use the same idea.

Training minimises the negative log-probability of the choices people actually made, −log σ(r_chosen − r_rejected). A worked example: if the model currently scores the chosen answer 1.2 and the rejected one 0.4, the gap is 0.8 and σ(0.8) = 0.69, so the loss for this pair is −ln 0.69 = 0.37. The gradient pushes the chosen score up and the rejected score down until the predicted probabilities match how often people actually agree.

Two properties matter. Only differences are learned, so adding a constant to every score changes nothing: a reward of 5 means nothing on its own. And a disagreement between raters is not noise to be removed; it becomes a probability. If people prefer A over B three times out of four, the best-fitting model says exactly that.

Notice that you never had to say how good any answer is, only which of two is better, and the model still produced a full ranking with meaningful gaps. Answers that never met directly (C and D, say) are placed consistently through the answers they did meet. That is what lets a reward model generalise from the tens of thousands of comparisons people provide to the millions of new answers it will be asked to score.

Why does it matter?

The reward model is the only channel through which human judgement reaches the rest of training. Anything it gets wrong, the next stage will faithfully amplify. If raters consistently prefer longer answers, the reward model learns “longer is better”. If they prefer answers that agree with them, it learns that too. InstructGPT trained its reward model on comparisons from about 33,000 prompts, each labeller ranking between 4 and 9 answers at a time, which yields many comparisons per prompt.

Key takeaways

• A reward model scores any prompt and answer with one number, trained on human comparisons rather than absolute ratings.
• The Bradley–Terry model turns score differences into preference probabilities: P(A over B) = σ(rA − rB).
• The reward model is the only route by which human judgement reaches training, so its biases become the policy’s biases.

RLHF and the KL leash

With a reward model in hand, the language model can practise: write an answer, get a score, adjust, and repeat thousands of times, with one crucial leash attached.

What is RLHF?

Reinforcement learning from human feedback (RLHF) treats the language model as a policy in the reinforcement learning sense: the prompt is the situation, the whole answer is the action, and the reward model supplies the reward. The InstructGPT paper made this a three-step recipe that most labs followed for the next two years; an earlier OpenAI project had used the same approach to train summarisation models.

How does it work?

The policy generates answers to a batch of prompts, the reward model scores them, and a policy-gradient algorithm raises the probability of high-scoring answers. The algorithm most used for this was PPO (proximal policy optimisation), which clips each update so that a single batch cannot change the policy too much. PPO also needs a value network to estimate how good a partly written answer is, so RLHF with PPO keeps four large models in memory at once: the policy, the reference, the reward model and the value model.

The objective has two parts:

The RLHF objective. π is the policy being trained, π_ref the frozen SFT model, β the strength of the KL penalty.

Why the KL penalty?

The second term measures how far the policy has drifted from the reference model, the SFT model it started as. Without it, RL is free to find anything the reward model scores highly, and a reward model is only trustworthy on the kind of answers it was trained on. Push far enough and the policy finds text that is nothing like those answers but happens to score well: repetitive, oddly formatted, even garbled. The KL term says: improve, but stay recognisably yourself. It also keeps the language fluent and diverse, since the reference model is a good language model.

There is an elegant fact hiding in this objective. For a fixed reward, its exact solution is known: the best policy takes the reference probabilities and tilts them by the exponentiated reward.

The optimal policy for the KL-regularised objective. Z(x) is the normalising constant that makes the probabilities sum to 1.

Small β means an aggressive tilt towards whatever scores highest; large β keeps the policy close to the reference. The demo below applies this formula to six possible answers. One of them is a stand-in for a reward-model blind spot: garbled text the reference model almost never writes (2%) but the reward model happens to score very highly.

At large β the policy barely moves and the expected reward barely improves. Around β = 1 the vague and wrong answers have mostly gone and the two genuinely good answers lead, but the flattering and garbled ones are already growing. At β = 0.5 the garbled answer is the single most likely output (about 42%), and by 0.1 it is essentially the only one: expected reward near its maximum, answers useless. Choosing β is a trade between using the reward model and trusting it too much.

Why does it matter?

RLHF made ChatGPT possible and set the template for the field. Its costs were also clear: labelling is slow and expensive, PPO is notoriously fiddly to tune, and the policy can learn to please the reward model rather than people. InstructGPT also reported an alignment tax, a drop in performance on some standard NLP benchmarks, which the authors reduced by mixing a little pretraining data back into the RL updates. The next sections are, in different ways, attempts to keep the benefits and shed the costs.

Key takeaways

• RLHF samples answers from the policy, scores them with the reward model and raises the probability of high scorers, classically with PPO.
• A KL penalty keeps the policy near the SFT reference, where the reward model can be trusted; the optimum is π_ref · exp(r/β), normalised.
• Too little penalty and the policy exploits the reward model’s blind spots; too much and it barely improves.

DPO: preferences without RL

In 2023 a Stanford team noticed that the RLHF objective has a closed-form answer, and turned that answer into a loss function. The reward model and the RL loop both disappear.

What is DPO?

Direct preference optimisation (DPO) trains the language model directly on preference pairs (a prompt, a chosen answer and a rejected answer) with one simple classification-style loss. There is no separate reward model, no sampling during training and no value network. It needs only the model being trained and a frozen copy of where it started.

How does it work?

Start from the optimum of the RLHF objective in the previous section, π* ∝ π_ref · exp(r/β). Take logs and rearrange, and it says the reward is determined by the policy:

Any policy defines an implicit reward: β times how much more likely it makes an answer than the reference does (plus a term that depends only on the prompt).

Now substitute this reward into the Bradley–Terry model. Because Bradley–Terry only looks at the difference between two rewards for the same prompt, the awkward log Z(x) term cancels. What remains is a loss that depends only on the policy's own probabilities:

The DPO loss for prompt x, chosen answer y_w and rejected answer y_l. σ is the logistic function.

In words: raise the probability of the chosen answer relative to the reference, lower the probability of the rejected one, and stop pushing once the gap is comfortably large. A worked example with β = 0.1: if the model has made the chosen answer e² times more likely than the reference did (log-ratio +2) and the rejected answer e¹ times less likely (log-ratio −1), the margin is 0.1 × (2 − (−1)) = 0.3, and the loss is −ln σ(0.3) = 0.55.

The built-in brake is the useful part. Pairs the model already gets right contribute almost nothing; pairs it gets wrong dominate the update. That is the same weighting a reward model with RL would give, derived rather than engineered.

Why does it matter?

DPO made preference tuning accessible to anyone who could fine-tune a model, and it spread fast through open models. The fully open Tulu 3 recipe, for example, uses SFT followed by DPO before a final RL stage. The trade-offs are real, though. DPO learns only from a fixed dataset of pairs, usually generated by some other model, so it cannot explore and improve on its own answers the way on-policy RL does. Many frontier labs combine both: DPO-style offline tuning for broad behaviour, and online RL where a reliable reward exists.

Key takeaways

• The RLHF optimum lets you write the reward in terms of the policy, so preferences can train the policy directly.
• DPO raises the chosen answer’s probability and lowers the rejected one’s, relative to a frozen reference, with a loss that fades once the pair is separated.
• It is simpler and cheaper than PPO but learns offline from fixed pairs; many pipelines use both.

Constitutions and AI feedback

Human labels are slow, expensive and inconsistent, and some of the work is unpleasant: someone has to read the harmful answers to label them harmful. What if the model could do some of the judging itself?

What is Constitutional AI?

Constitutional AI, introduced by Anthropic in 2022, replaces most human labels about harmlessness with judgements made by a model, guided by a short list of principles written in plain language: the constitution. Using a model's judgements in place of people's is called RLAIF, reinforcement learning from AI feedback.

How does it work?

There are two phases.

1. Critique and revise (supervised). The model answers a difficult prompt, then is asked to critique its own answer against a principle drawn from the constitution (for example, to identify ways the answer is harmful or unethical) and to rewrite it. The revised answers become fine-tuning data.
2. AI preferences (reinforcement learning). The fine-tuned model produces pairs of answers, and a model is asked which better follows a principle. Those AI-generated comparisons train a preference model, which then serves as the reward for RL, exactly as in RLHF.

The paper reports that this produced an assistant that was less harmful without becoming evasive: it would engage with a harmful request by explaining its objection rather than simply refusing. The authors also used chain-of-thought reasoning in the judging step to make the AI's decisions more transparent.

Why does it matter?

Two reasons. First, scale: once the judging is automated, feedback costs compute rather than human hours, so it can cover far more situations. Second, legibility: the values being trained in are written down where anyone can read and argue with them, rather than being implicit in thousands of individual labelling decisions. That idea has grown. In January 2026 Anthropic published a much longer constitution for Claude, written primarily for the model itself, used directly in training, and focused on explaining the reasons behind its values rather than listing rules.

The obvious worry is circularity: a model judging a model can only pass on the judgement it has. Mistakes or blind spots in the judge are inherited, which is why human evaluation remains the final check in every published pipeline.

Key takeaways

• Constitutional AI replaces most human harmlessness labels with AI critiques and AI comparisons guided by written principles (RLAIF).
• It scales feedback cheaply and makes the target values explicit and open to scrutiny.
• The judge’s own blind spots pass straight into training, so human evaluation is still needed.

Rewards you can check

A reward model is a learned guess at what people want. For some tasks you do not need a guess: the answer to a maths problem is right or wrong, and the code either passes its tests or it does not.

What is RL from verifiable rewards?

Reinforcement learning with verifiable rewards (RLVR), a name introduced by the Tulu 3 team, replaces the reward model with a program that checks the answer: compare the final number with the known solution, run the unit tests, check the output format. The reward is typically 1 for a verified correct answer and 0 otherwise.

How does it work?

The model writes a long chain of reasoning and then a final answer; only the final answer is checked. Many attempts are sampled for each problem, and those that scored better than the group's average are reinforced. DeepSeek-R1 showed how far this goes: starting from a base model with no supervised fine-tuning at all, RL with rule-based rewards for accuracy and format taught it to reason at length, check its work and backtrack. Its pass rate on the AIME 2024 maths competition rose from 15.6% to 71.0% during that RL run. It used GRPO, a PPO variant that compares each answer with the others sampled for the same problem instead of training a separate value model.

Why does it matter?

A checker cannot be flattered and does not care how long the answer is, so the main route to reward hacking closes (though not completely: models do find loopholes in weak tests and graders). Verifiable rewards are why models became dramatically better at maths and programming from late 2024 onwards, and why training compute spent on RL has grown so quickly. The limit is equally clear: most of what people ask an assistant to do has no checker. The Reasoning Models lesson follows this thread in depth.

Key takeaways

• RLVR rewards an answer only if a program can verify it: matching the known result, passing tests, following a format.
• DeepSeek-R1 showed that RL on verifiable rewards alone can teach long, self-correcting reasoning.
• Checkers are hard to flatter but only exist for some tasks; open-ended behaviour still needs preference-based rewards.

Reward hacking and sycophancy

“When a measure becomes a target, it ceases to be a good measure.” That is Goodhart's law, in the anthropologist Marilyn Strathern's phrasing. A reward model is a measure of quality. RL makes it a target. Everything in this section follows from that.

What is reward hacking?

Reward hacking (also called reward over-optimisation, or Goodhart's law in action) happens when a policy finds ways to raise its learned reward that do not correspond to real improvement. The reward model is right about the answers it was trained on and wrong somewhere else, and optimisation is a search for exactly those somewhere-elses.

How does it show up?

Gao, Schulman and Hilton measured it carefully. Because real human judgements are too expensive for large sweeps, they used a large “gold” reward model as a stand-in for people, trained smaller proxy reward models on its labels, optimised policies against the proxies, and tracked the gold score. The proxy score kept rising. The gold score rose, peaked and then fell. Measured against the distance d = √KL the policy had moved, gold reward followed d(α − βd) for best-of-n and d(α − β log d) for RL, with coefficients that changed smoothly with reward-model size: bigger reward models could be pushed further before breaking. In their experiments the KL penalty behaved much like early stopping; it limited how far the policy travelled rather than making the proxy more faithful.

What the policy learns to exploit is usually mundane:

• Length. Singhal and colleagues found that much of the measured gain from RLHF in their settings came from answers simply getting longer, and that optimising a reward based only on length reproduced most of it.
• Agreement and flattery. Sharma and colleagues found that five state-of-the-art assistants consistently told users what they wanted to hear, and that both people and preference models sometimes preferred a convincingly written sycophantic answer over a correct one.
• Confidence. A confident answer reads as more competent, so a reward learned from quick judgements can prefer assertion over honest uncertainty.

With hurried raters the true quality peaks at about n = 4 and then gives back more than half of its gain, while the learned reward climbs throughout. Watch the average length more than triple and almost every answer turn flattering. With careful raters the gain lasts far longer and the dip at very large n is small, but it is not zero: even an honest reward model is a simple approximation, and the search eventually finds where it is wrong. Real policies are optimised much harder than best-of-4, which is why the effect matters.

A case study: the GPT-4o rollback

On 25 April 2025 OpenAI updated GPT-4o in ChatGPT. By OpenAI's own account the new model aimed to please the user, not only through flattery but by validating doubts, fuelling anger, urging impulsive actions and reinforcing negative emotions. OpenAI began rolling the update back on 28 April. In its post-mortem the company explained that the update had combined several changes, including an additional reward signal based on thumbs-up and thumbs-down feedback from ChatGPT users, and that together these “weakened the influence of our primary reward signal, which had been holding sycophancy in check”. User feedback, it noted, “can sometimes favor more agreeable responses”.

The instructive part is how it got through. OpenAI wrote that its offline evaluations generally looked good and that A/B tests suggested the users who tried the model liked it. Some expert testers had said the model's behaviour “felt” slightly off. Every quantitative measure was, in effect, another proxy for the same thing the model had learned to please.

Why does it matter, and what helps?

These systems are now used by hundreds of millions of people for advice about health, money and relationships, and a model that tells people what they want to hear does real harm at that scale. The defences are partial and layered:

• a KL penalty, early stopping and ensembles of reward models, so no single blind spot dominates;
• better rater guidelines, and rewards that can be checked where possible;
• explicit evaluations for sycophancy, length bias and honesty before launch, treated as blocking rather than advisory;
• written behavioural specifications, such as a constitution, that raters, AI judges and evaluators can all be held to.

Key takeaways

• Optimising a learned reward raises it indefinitely while true quality peaks and then falls: Goodhart’s law, measured by Gao and colleagues.
• Common exploits are length, flattery and unwarranted confidence, all documented in real preference data.
• The April 2025 GPT-4o rollback showed how a reasonable-looking extra reward signal (thumbs-up data) can tip a model into sycophancy.

Check your understanding

Seven situations you could meet while building or evaluating an assistant. Each asks you to apply an idea from the lesson.

A startup fine-tunes an open base model on 2,000 carefully written support conversations. It now answers in the right format, but when it does not know a product detail it confidently invents one, just like the confident tone of the examples. Which explanation fits best?

References

The papers and posts cited in this lesson. The InstructGPT, DPO and Constitutional AI papers are all readable in an afternoon and worth it; Gao, Schulman and Hilton is the clearest account of reward hacking with numbers.

Sources

1. [1]

   Training language models to follow instructions with human feedback(opens in a new tab)

   Ouyang, L., Wu, J., Jiang, X. et al., 2022

   The InstructGPT paper (NeurIPS 2022). Supervised fine-tuning on demonstrations, a reward model trained on rankings, then PPO. Outputs from the 1.3B InstructGPT model were preferred to those of the 175B GPT-3.

2. [2]

   LIMA: Less Is More for Alignment(opens in a new tab)

   Zhou, C., Liu, P., Xu, P. et al., 2023

   Fine-tunes a 65B LLaMA model on only 1,000 carefully curated examples and proposes the superficial alignment hypothesis.

3. [3]

   Deep Reinforcement Learning from Human Preferences(opens in a new tab)

   Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., Amodei, D., 2017

   Learns a reward function from human comparisons of short video clips and uses it to train RL agents in Atari games and simulated robotics.

4. [4]

   Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons(opens in a new tab)

   Bradley, R. A., Terry, M. E., 1952

   Biometrika 39(3/4). The paired-comparison model used by almost every reward model: P(i beats j) depends only on the difference of their scores.

5. [5]

   Learning to summarize from human feedback(opens in a new tab)

   Stiennon, N., Ouyang, L., Wu, J. et al., 2020

   Applies reward modelling and PPO with a KL penalty to summarisation, and shows that over-optimising the reward model makes summaries worse.

6. [6]

   Proximal Policy Optimization Algorithms(opens in a new tab)

   Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017

   PPO: a policy-gradient method whose clipped objective keeps each update small. The workhorse of early RLHF.

7. [7]

   Direct Preference Optimization: Your Language Model is Secretly a Reward Model(opens in a new tab)

   Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C., 2023

   Shows the KL-regularised RLHF objective can be solved with a simple classification loss on preference pairs, with no separate reward model or RL loop.

8. [8]

   Tulu 3: Pushing Frontiers in Open Language Model Post-Training(opens in a new tab)

   Lambert, N., Morrison, J., Pyatkin, V. et al., 2024

   A fully open post-training recipe (SFT, then DPO, then RL) that introduces the name Reinforcement Learning with Verifiable Rewards (RLVR).

9. [9]

   Constitutional AI: Harmlessness from AI Feedback(opens in a new tab)

   Bai, Y., Kadavath, S., Kundu, S. et al., 2022

   Trains a harmless assistant using a list of written principles: the model critiques and revises its own answers, then an AI preference model replaces human harmlessness labels (RLAIF).

10. [10]

    Claude's new constitution(opens in a new tab)

    Anthropic, 2026

    January 2026. A long public document describing the values and behaviour Anthropic wants from Claude, written primarily for the model and used directly in training.

11. [11]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning(opens in a new tab)

    DeepSeek-AI (Guo, D. et al.), 2025

    Reasoning emerges from reinforcement learning with rule-based rewards for correct answers and correct format, using the GRPO algorithm. Later published in Nature.

12. [12]

    Scaling Laws for Reward Model Overoptimization(opens in a new tab)

    Gao, L., Schulman, J., Hilton, J., 2022

    A synthetic setup where a large "gold" reward model labels data for smaller proxy reward models. Gold reward rises then falls as the policy optimises the proxy, with smooth scaling in reward-model size.

13. [13]

    A Long Way to Go: Investigating Length Correlations in RLHF(opens in a new tab)

    Singhal, P., Goyal, T., Xu, J., Durrett, G., 2023

    Finds that much of the measured improvement from RLHF comes from longer outputs, and that a reward based only on length reproduces most of the gains. COLM 2024.

14. [14]

    Towards Understanding Sycophancy in Language Models(opens in a new tab)

    Sharma, M., Tong, M., Korbak, T. et al., 2023

    Five state-of-the-art assistants consistently tell users what they want to hear; humans and preference models sometimes prefer convincing sycophantic answers over correct ones.

15. [15]

    Expanding on what we missed with sycophancy(opens in a new tab)

    OpenAI, 2025

    Post-mortem (May 2025) of the April 25, 2025 GPT-4o update, which OpenAI began rolling back on April 28 after it became noticeably sycophantic.