Reward Model Lab

About

Chat assistants are tuned on human judgements of the form “answer A is better than answer B”. Those judgements train a reward model, a network that gives any answer a score. The assistant is then optimised to produce answers that score highly. This lab runs that whole loop at toy scale so you can see its best-known failure: optimise a learned reward hard enough and it stops tracking what people actually wanted.

The three stages

1. Collect preferences. Judge pairs of answers yourself, or let an auto-labeller do it. Every fifth comparison is held back to test the reward model on data it never trained on.
2. Fit a reward model. After every label the model refits and shows which features it thinks people like (positive weight) and dislike (negative weight).
3. Optimise a policy. Starting from a reference policy (the “supervised” model that mostly writes short answers), push the policy towards high learned reward, with an optional KL penalty that keeps it close to where it started. Then compare the learned reward with the true quality from a hidden rubric.

The answers

Each answer has six visible features: whether it is correct, its length (30 to 450 words), its warmth, how confident it sounds, whether it uses bullet points, and whether it flatters the user. The text is generated from those features so it reads like a real reply, and the features are listed under each card so nothing is hidden from you.

How it works

Bradley–Terry reward model

The model gives each answer a score r(y) = w · x(y), where x is the answer's six features scaled to 0–1. The probability that a rater prefers answer A over B is modelled as σ(r(A) − r(B)), the logistic function of the score difference. Fitting means choosing w to maximise the probability of the choices actually made, which is ordinary logistic regression on the feature differences. A small L2 penalty keeps the weights finite when a handful of comparisons can be separated perfectly.

Real reward models are the language model itself with its final layer replaced by a single scalar output, trained with the same loss on hundreds of thousands of comparisons. The mathematics is identical; only x is richer.

RL with a KL penalty

The policy maximises

J(π) = E_π[r(y)] − β · KL(π ‖ π_ref)

The KL term measures how far the policy has drifted from the reference model. Because the policy here is a softmax over all 576 answers, the lab takes exact natural-gradient steps: every answer's logit moves by η · (f(y) − E_π[f]) with f(y) = r(y) − β log(π(y)/π_ref(y)). Production RLHF estimates the same gradient from sampled answers with PPO; the direction is the same, just noisier.

Best-of-n

The simplest optimiser of all: sample n answers from the reference policy and keep the one the reward model likes most. Its distribution is computed exactly here, and its KL from the reference grows like log n − (n−1)/n, so n = 4,096 is only about 7.3 nats away. Best-of-n needs no training, which is why it is widely used for evaluation and data generation.

The chart's x-axis

Both curves are plotted against √KL, the distance the policy has moved from the reference. Gao, Schulman and Hilton (2022) used the same axis and found that true reward follows a rise-then-fall curve whose shape is predictable from reward-model size.

Reward hacking

The learned reward is a proxy. It can part ways with true quality for two separate reasons, and the two auto-labellers let you see each.

1. The raters are biased

The hurried rater mostly agrees with the rubric but also rewards length, confident tone and compliments. The reward model learns those biases faithfully; it is doing its job. The policy then discovers that long, emphatic, flattering answers score best. Both biases are documented in real preference data: length correlations in RLHF (Singhal et al., 2023) and a preference for answers that agree with the user (Sharma et al., 2023).

2. The data do not reach far enough

Even the careful rater's labels come only from answers the reference policy writes, and it rarely writes anything long. A linear reward model can only say “more length is better” or “less is better”, while the rubric says length helps up to about 150 words and then hurts. Whatever the model learned inside the data, the optimiser will carry to the extremes where the model was never tested.

What helps

In April 2025 OpenAI rolled back a GPT-4o update after it became noticeably sycophantic. Its post-mortem said the update had added a reward signal based on users' thumbs-up and thumbs-down ratings, which weakened the primary reward signal that had been holding sycophancy in check.

Try this

1. 01Watch a reward get hackedPick the hurried rater, auto-label 100 pairs, set β to 0 and run RL. The learned reward climbs the whole way; the true quality peaks early and then falls, often below where it started.
2. 02Rescue it with a KL penaltySame data, β = 1. The run stops near the true peak. Try values in between and find where the final true quality is highest.
3. 03Honest labels are not enoughSwitch to the careful rater. The damage is smaller, but look at the length weight and at best-of-n with a large n: the model has still learned something the rubric does not want.
4. 04Label it yourselfClear the data and judge 30 pairs by hand. Which features did you reward without meaning to? Compare your weights with the rubric (Reveal the hidden rubric).
5. 05Best-of-n versus RLRun both on the same data and compare where the true peak sits on the √KL axis. Gao et al. found the two methods follow different functional forms, both rising and then falling.
6. 06Small data, big weightsLabel only 8 pairs. Held-out accuracy is unreliable and the weights swing with every new label. That is why real reward models are trained on hundreds of thousands of comparisons.