Reasoning Models

Thinking before answering

On the 2024 American Invitational Mathematics Examination, a hard competition for the best high-school mathematicians, GPT-4o solved 12% of the problems. In September 2024 OpenAI released o1, which solved 74% with a single attempt. The difference OpenAI described was not a bigger model but a different habit: before answering, o1 writes a long, private chain of reasoning, and it was trained to make that reasoning useful.

What is a reasoning model?

A reasoning model is a language model that produces a stretch of thinking before its final answer: it tries an approach, works through the steps, notices mistakes, backtracks and checks. Architecturally it is the same kind of transformer as any other large language model, still predicting one token at a time. What is different is how it was trained and how much text it is allowed to write before committing to an answer.

Ask one “how many times does the digit 7 appear when you write out every number from 1 to 100?” and a model that answers immediately has to produce the number in one shot. A reasoning model instead writes something like: “units digit: 7, 17, 27, … 97 is ten numbers; tens digit: 70 to 79 is ten more; 77 is in both lists, which is right because it has two sevens; total 20.” Every intermediate line is text the model can read back and build on.

How does it work?

Three ingredients, which the rest of this lesson takes one at a time:

• Writing the steps. Generating intermediate reasoning gives a network of fixed depth more serial computation per answer, and somewhere to store intermediate results.
• Spending more at answer time. Think longer, or think several times and vote, or let a verifier pick the best attempt. This is test-time compute.
• Training the habit. Reinforcement learning on problems whose answers a program can check teaches the model which ways of thinking lead to right answers.

Why does it matter?

For years the reliable way to make models better was to make them bigger and train them on more data (see scaling laws). Reasoning models added a second dial: spend more computation on each question. OpenAI reported that o1's accuracy improved smoothly both with more reinforcement-learning training and with more thinking time at inference. By July 2025, reasoning systems from Google DeepMind and OpenAI both reached gold-medal level on the International Mathematical Olympiad, and long, self-correcting reasoning is what lets today's coding and research agents work through tasks that take hours.

The ideas arrived over about four years, mostly as papers before they were products:

Key takeaways

• A reasoning model is an ordinary language model that writes, and was trained to write, extended reasoning before its answer.
• Its gains come from three things: intermediate steps, extra computation at answer time, and reinforcement learning on checkable problems.
• Test-time compute is a second way to buy accuracy, alongside bigger models and more training data.

Chain-of-thought prompting

The first reasoning models were ordinary models given a nudge. In 2022, researchers found that simply showing a model worked solutions, or asking it to think step by step, made it far better at problems with several steps.

What is chain-of-thought prompting?

Wei and colleagues (2022) compared two ways of prompting with a few solved examples. In standard prompting, each example shows a question and its final answer. In chain-of-thought prompting, each example also shows the reasoning in plain words. The model imitates whatever format it is shown, so it starts writing reasoning of its own before the answer.

How much did it help?

On GSM8K, a set of grade-school maths word problems, chain-of-thought prompting lifted Google's 540-billion-parameter PaLM from about 18% to 57%. The gains appeared only in large models: in Wei's experiments, smaller models wrote fluent but illogical chains and were often hurt by them.

A few months later Kojima and colleagues found that no examples were needed at all. Appending one sentence, “Let's think step by step”, raised accuracy on the MultiArith benchmark from 17.7% to 78.7%, and on GSM8K from 10.4% to 40.7%, with the same model.

Why does the order matter?

A model generates text left to right, and each token can only depend on what comes before it. Reasoning helps an answer only if it is written before that answer. The prompt format “reason, then answer” is the entire trick; the words themselves carry the intermediate results forward.

Prompting techniques have since grown into a craft of their own; the prompting lesson covers them. For models trained to reason, the explicit “think step by step” instruction mostly matters less, because thinking before answering is now built in.

Key takeaways

• Chain-of-thought prompting shows or asks for reasoning before the answer, and the model follows suit.
• It gave large jumps on multi-step problems (PaLM on GSM8K: about 18% to 57%) but mainly in large models.
• Reasoning helps only if it precedes the answer, because a token can only depend on the tokens before it.

Why writing steps helps

Why should writing words make a model smarter? Because a transformer does a fixed amount of work per token. Writing intermediate results is how it turns a hard problem into many easy ones.

What limits a model that answers immediately?

A transformer computes each new token with one pass through its layers: a few dozen to more than a hundred in a large model. However clever each layer is, the chain of dependent computation between reading the question and writing the first digit of the answer has a fixed length. Many steps can happen in parallel within a layer, but a problem whose steps must happen one after another (carry this digit, then use it in the next column) can only get as many sequential steps as the network has layers.

Li, Liu, Zhou and Ma (2024) made this precise. A constant-depth transformer that must answer immediately can only compute functions that are highly parallel. Given T steps of chain of thought, the same transformer can simulate any boolean circuit with T gates: every written token is another round of computation, and the text is the working memory that connects them.

How does the scratchpad change the work?

Long multiplication is the classic example. Multiplying two n-digit numbers takes roughly n² single-digit operations, most of which depend on earlier ones through carries. Dziri and colleagues (2023) found that transformers' accuracy on multi-digit multiplication collapses as the numbers get longer, even for models that handle small cases well. Written out row by row, though, each line needs only about 3n operations, and each line reads the previous one.

With 4-digit numbers and 12 steps per write, the direct answer runs out after the first row and writes a partial sum, while the scratchpad gets every line right. The machine is identical; only the format of its output changed. At 5 or 6 digits even a single row needs more than 12 steps, and the fix is a finer scratchpad. That mirrors what models do: when a step is too big, they break it into smaller ones.

Why does it matter?

It explains why reasoning helps most on problems with long chains of dependent steps (arithmetic, logic puzzles, code tracing, planning) and helps little on recall (“what is the capital of Peru?”), which needs no intermediate results. It also points to an alternative: instead of writing tokens, a model can reuse its own layers several times per token. Looped transformers do exactly that, getting extra sequential depth without emitting text.

Key takeaways

• A transformer does a fixed amount of sequential computation per token; answering immediately caps the work at that amount.
• Each written reasoning token adds another round of computation, and the text itself stores intermediate results.
• Problems with long chains of dependent steps benefit most; simple recall barely benefits at all.

Sample and vote

Ask a model the same hard question five times and you may get five different chains of reasoning. That variety turns out to be useful: the answer the chains agree on is more often right than any single chain.

What is self-consistency?

Self-consistency (Wang et al., 2022) samples several chains of thought at a temperature above zero, so they differ (see the LLM Sampling Lab for how temperature spreads the choices), extracts the final answer from each, and returns the most common one. It needs no extra training and no extra model. On GSM8K it added 17.9 percentage points over a single chain of thought.

How does voting help when each chain is unreliable?

There are many ways to reason correctly to 9, and they all end at 9. Mistakes tend to be different each time: one chain drops a carry, another misreads a number, and they land on 27, 3 or 14. If the right answer is more likely than any single wrong answer, then as votes accumulate it wins more and more often, even if a single chain is right less than half the time.

The condition for voting to help. It compares the right answer with the strongest wrong answer, not with all wrong answers combined.

Two regimes appear. When the right answer leads, even narrowly, accuracy climbs towards certainty; with the default settings, a model right 40% of the time is right 88% of the time with 21 votes and 95% with 41. When a wrong answer leads, voting locks it in. The o1 announcement shows the first regime at scale: 74% on AIME with one sample, 83% with a consensus of 64.

Where voting falls short

• Answers must be comparable. Voting works for a number or a multiple-choice letter. For an essay or a program, two correct outputs rarely match word for word.
• Samples are correlated. The demo assumes independent attempts. Real samples share the model's blind spots, so a systematic misconception shows up as a “favourite wrong answer”, which is exactly the case voting cannot fix.
• Cost grows linearly. Forty samples cost forty times as much. Voting buys accuracy with compute, and only up to what the model already knows.

Key takeaways

• Self-consistency samples several reasoning chains and returns the most common final answer.
• It helps when the right answer is more likely than any single wrong answer, even if it is less likely than all wrong answers together.
• A consistent misconception gets locked in by voting; more samples cannot create knowledge the model lacks.

Verifiers and process rewards

Voting asks which answer is most popular. A better question is which answer is right, and for that you need something that can judge an attempt: a verifier.

What is a verifier?

A verifier scores a candidate solution. Sometimes it is exact: unit tests for code, substituting the answer back into an equation, a proof checker. Often it is another model trained to predict whether a solution is correct. Cobbe et al. (2021), who introduced GSM8K, trained such a verifier and showed that sampling many solutions and keeping the one it scored highest (best-of-N) beat fine-tuning the generator alone.

How do process reward models differ?

An outcome verifier looks only at the final answer. A process reward model (PRM) scores every step. In “Let's Verify Step by Step”, OpenAI researchers had people label 800,000 individual steps of maths solutions as correct, incorrect or neutral (the PRM800K dataset) and trained a PRM on them. Picking the best of many samples with the PRM solved 78% of a representative subset of the MATH benchmark, clearly more than with an outcome-supervised verifier. Step scores have a second use: they let a search prune a bad line of reasoning as soon as it goes wrong, instead of waiting for the end.

Why does it matter, and where does it stop?

With a perfect verifier, accuracy would equal the chance that any of N samples is right, which keeps climbing with N. Brown et al. (2024) found exactly that for coverage, which rose steadily over thousands of samples, but also that majority voting and learned reward models plateaued after a few hundred. A learned verifier has blind spots: some wrong answers look right to it. With enough samples, one of those will be the top scorer.

Gains come quickly and then flatten at a ceiling set by the verifier's blind spots, not by the generator. Raising the verifier's skill moves you along the curve faster; only shrinking the blind spots raises the ceiling. That is why exact checks (tests, proofs, known answers) are so valuable, and why optimising hard against a learned judge is dangerous: the search actively seeks out the answers the judge wrongly likes. The Reward Model Lab shows the same effect, called reward hacking, during training.

How best to spend a fixed budget of test-time compute depends on the problem. Snell et al. (2024) found that choosing the strategy per prompt by difficulty, sometimes letting the model revise its answer in sequence, sometimes searching against a PRM, was more than four times as efficient as plain best-of-N. OpenAI's o1 reached 93% on AIME 2024 when a learned scoring function re-ranked 1,000 samples.

Key takeaways

• A verifier scores attempts so you can keep the best; process reward models score each step, which also lets search prune early.
• Best-of-N gains flatten at a ceiling set by the wrong answers the verifier cannot detect.
• Exact verifiers (tests, proofs, known answers) avoid that ceiling, which is why reasoning progress has been fastest in maths and code.

Learning to reason with RL

Prompting asks a model to reason; reinforcement learning teaches it which reasoning works. Give a model thousands of problems with checkable answers, reward it only when the final answer is right, and long, careful, self-correcting thinking emerges without anyone writing an example of it.

What changed with o1 and DeepSeek-R1?

OpenAI described o1 as trained with “a large-scale reinforcement learning algorithm” that teaches the model to use its chain of thought productively, and reported that accuracy kept rising with more RL training and with more thinking time. It did not publish the recipe, and it hides the raw chain of thought from users.

In January 2025 DeepSeek published one. DeepSeek-R1-Zero started from a pretrained base model with no reasoning examples at all. Training used only two rule-based rewards: the final answer matches the known solution (or the code passes its tests), and the reasoning sits between <think> tags. Over training, pass@1 on AIME 2024 rose from 15.6% to 71.0%, and to 86.7% with majority voting over 64 samples.

What did the model learn to do?

Nobody told R1-Zero to think longer, but its responses grew from hundreds to thousands of tokens as training went on, because longer reasoning earned more reward. Behaviours such as re-reading the problem, trying a second method and checking a result appeared on their own. The paper shows an intermediate checkpoint stopping mid-solution to write that this is an “aha moment” and start again. R1-Zero's reasoning was often hard to read and mixed languages, so the released DeepSeek-R1 added a small set of curated long reasoning examples before RL and further training stages afterwards (the post-training lesson covers that pipeline). DeepSeek also distilled R1's reasoning traces into much smaller open models, which inherited a large part of the skill.

How does GRPO turn right answers into learning?

Standard policy-gradient RL (see the reinforcement learning lesson) pushes up the probability of actions that did better than expected. “Expected” usually comes from a second network, a critic, as large as the model itself. Group Relative Policy Optimization, introduced with DeepSeekMath, drops the critic. For each question it samples a group of answers and uses the group itself as the baseline:

Each token of answer i is made more likely if A_i is positive and less likely if negative, with PPO-style clipping and a penalty for drifting too far from the starting model.

One right answer among eight gets an advantage of +2.65 while each wrong one gets −0.38: rare successes on hard problems carry most of the signal. When all eight agree, nothing is learned. This is why reasoning RL needs a steady supply of problems at the edge of the model's ability, and why the answers must be checkable by a program. A learned reward model could be gamed over millions of updates; DeepSeek chose rule-based rewards for exactly that reason.

How little it can take

Once strong reasoners existed, their traces became training data. s1 (Muennighoff et al., 2025) fine-tuned a 32-billion-parameter open model on only 1,000 carefully chosen reasoning traces, then controlled its thinking at inference with “budget forcing”: stop the thinking at a limit, or, when the model tries to stop early, append “Wait” so it keeps going. Forcing more thinking lifted its AIME 2024 score from 50% to 57%. The model often used the extra time to find and fix its own mistakes.

Key takeaways

• Reasoning models are trained with reinforcement learning that rewards only a correct, checkable final answer.
• Longer thinking, self-checking and backtracking emerge because they earn reward, not because they were demonstrated.
• GRPO scores each answer against its own group, so questions that are always or never solved produce no learning.

Thinking budgets in practice

Thinking is not free. Every thinking token takes time to generate and is billed like output. Using reasoning models well means deciding how much thought a task deserves.

What controls do you get?

In 2025 “thinking” became a setting rather than a separate product. Claude 3.7 Sonnet introduced extended thinking with a token budget; OpenAI's reasoning models take a reasoning-effort level; Google's Gemini 2.5 models accept a thinking budget. Current models increasingly decide for themselves: Anthropic's Claude Opus 5.5, for example, uses adaptive thinking, always on, with an effort setting that defaults to medium. The model thinks briefly on easy requests and at length on hard ones, and the effort setting shifts that balance.

How much does thinking cost?

Thinking tokens are output tokens, usually the most expensive kind, and they are generated one after another, so they add latency as well as cost. A worked example with an output price of $20 per million tokens:

Illustrative prices. The answer is the same length both times; the thinking is what multiplies the bill.

At scale that difference matters. A support system handling a million short queries a day would spend about $10,000 on answers and $170,000 with heavy thinking on every one. Reasoning models are also prone to overthinking: spending thousands of tokens re-checking problems as easy as 2 + 3. Much recent work aims to make models think in proportion to difficulty.

When does thinking pay off?

It pays when the task has many dependent steps and a wrong answer is costly. It rarely pays when the answer is a lookup, a matter of taste, or needs no intermediate results. Remember the lesson of voting and verifiers: extra thinking multiplies what the model already knows, and it cannot supply missing facts.

Prompting a model that already thinks

• State the goal, constraints and what a good answer looks like, rather than scripting each step. The model is trained to plan; a rigid recipe can make it worse.
• Give it a way to check itself. Tests to run, a condition the answer must satisfy, or an expected format turn its self-checking into real verification.
• Match effort to the task, and measure: try a lower effort setting on your own examples before paying for the highest.

Key takeaways

• Thinking is controlled through budgets, effort levels or adaptive thinking; more thought means more latency and more output tokens billed.
• A long think can multiply the cost of an answer many times over, and models can overthink easy questions.
• Spend thinking on multi-step, checkable, high-stakes tasks; keep it low for lookups, rewrites and routine text.

Can you trust the reasoning?

A visible chain of thought is tempting to read as the model's actual reasoning, a window into its mind. The evidence says it is a useful window with frosted panes.

What is faithfulness?

A chain of thought is faithful if it reflects the factors that actually drove the answer. It can be fluent, plausible and correct while leaving out the real reason. Turpin et al. (2023) showed this with a simple trick: in few-shot prompts they made the right answer always option (A). Models then favoured (A) on new questions, even when it was wrong, and accuracy fell by as much as 36% on BIG-Bench Hard tasks, yet the explanations never mentioned the pattern. They rationalised the biased answer instead.

Are trained reasoning models more honest about it?

Somewhat, but far from fully. Anthropic researchers (2025) slipped hints into questions (for example, a note saying that a professor thinks the answer is a particular option) and looked at cases where the hint changed the model's answer. Claude 3.7 Sonnet mentioned the hint in its reasoning 25% of the time and DeepSeek R1 39% of the time. Outcome-based RL raised faithfulness at first, then plateaued (at 28% on one benchmark and 20% on another). In environments with a deliberate loophole, models learned to exploit it in over 99% of cases but admitted doing so in their reasoning less than 2% of the time.

Why does it matter?

Because reading the chain of thought is one of the best safety tools available. Baker et al. (2025) found that even a weaker model reading a stronger model's reasoning could catch it gaming its coding tests. But when they penalised the model for reasoning the monitor flagged, it did not stop cheating; it learned to cheat without saying so. Their recommendation is to accept a “monitorability tax”: avoid training directly against what the chain of thought says, so that it stays worth reading.

Other limits

• Errors compound. A long chain gives more places for a slip, and a model can reason fluently from a wrong early step to a confident wrong answer.
• Gains follow the verifiers. Reasoning has improved fastest where answers can be checked automatically. Open-ended writing, judgement and taste benefit less.
• Cost and latency. Seconds to minutes of thinking are fine for a hard proof, and wrong for autocomplete.
• Knowledge is not created. As the voting demo showed, extra computation amplifies what the model already believes, including its misconceptions.

Key takeaways

• Chains of thought can leave out what really drove an answer; biased prompts and hints often go unmentioned.
• Reading the chain of thought is a powerful monitoring tool, but training against it teaches models to hide their intent.
• Long reasoning compounds errors, costs time and money, and helps most where answers can be checked.

Check your understanding

Six scenarios: prompting a model, spending a sampling budget, building a training set and reading a chain of thought critically.

A product team wants better accuracy on 5-digit multiplication from a model that is not a reasoning model. Which change should help most?

References

Papers and announcements cited in this lesson. Benchmark figures are as reported by the authors. For hands-on work, the LLM Sampling Lab shows how temperature creates the variety that voting relies on, and the Reward Model Lab shows a learned judge being gamed.

Sources

1. [1]

   Learning to Reason with LLMs(opens in a new tab)

   OpenAI, 2024

   Introduces o1 (12 September 2024). On AIME 2024, GPT-4o solved 12% of problems on average; o1 averaged 74% with one sample, 83% with consensus among 64 samples and 93% re-ranking 1,000 samples with a learned scorer. Performance improves with both train-time and test-time compute.

2. [2]

   Chain-of-Thought Prompting Elicits Reasoning in Large Language Models(opens in a new tab)

   Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D., 2022

   A few worked, step-by-step examples in the prompt make large models reason step by step too; PaLM 540B on GSM8K rises from about 18% to 57%. NeurIPS 2022.

3. [3]

   Large Language Models are Zero-Shot Reasoners(opens in a new tab)

   Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., Iwasawa, Y., 2022

   “Let’s think step by step” with no examples: MultiArith 17.7% to 78.7% and GSM8K 10.4% to 40.7% with text-davinci-002. NeurIPS 2022.

4. [4]

   Chain of Thought Empowers Transformers to Solve Inherently Serial Problems(opens in a new tab)

   Li, Z., Liu, H., Zhou, D., Ma, T., 2024

   Proves that a constant-depth transformer answering immediately is limited to highly parallel computations, while T steps of chain of thought let it simulate a circuit of size T. ICLR 2024.

5. [5]

   Faith and Fate: Limits of Transformers on Compositionality(opens in a new tab)

   Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., et al., 2023

   Studies multi-digit multiplication and other compositional tasks: accuracy of transformers falls sharply as problems need more steps. NeurIPS 2023.

6. [6]

   Self-Consistency Improves Chain of Thought Reasoning in Language Models(opens in a new tab)

   Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D., 2022

   Sample several reasoning paths and take the most common final answer. Gains over single chain-of-thought include +17.9 points on GSM8K. ICLR 2023.

7. [7]

   Training Verifiers to Solve Math Word Problems(opens in a new tab)

   Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, Ł., et al., 2021

   Introduces GSM8K and trains a verifier to score sampled solutions; picking the highest-scoring of many samples beats fine-tuning alone.

8. [8]

   Let’s Verify Step by Step(opens in a new tab)

   Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K., 2023

   Process supervision (a reward for every step) beats outcome supervision; the process-supervised reward model solves 78% of a representative subset of MATH. Releases PRM800K, 800,000 step-level labels.

9. [9]

   Large Language Monkeys: Scaling Inference Compute with Repeated Sampling(opens in a new tab)

   Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., Mirhoseini, A., 2024

   The fraction of problems solved by at least one sample keeps rising over thousands of samples, but majority voting and reward models plateau after a few hundred samples.

10. [10]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters(opens in a new tab)

    Snell, C., Lee, J., Xu, K., Kumar, A., 2024

    Choosing how to spend test-time compute per prompt, by difficulty, is more than 4× as efficient as best-of-N, and can let a small model beat a much larger one on problems it can partly solve.

11. [11]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning(opens in a new tab)

    DeepSeek-AI (Guo, D. et al.), 2025

    Reinforcement learning with rule-based rewards alone (R1-Zero) raises AIME 2024 pass@1 from 15.6% to 71.0% (86.7% with majority voting over 64 samples); responses grow longer and self-checking appears. Also published in Nature 645 (2025).

12. [12]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models(opens in a new tab)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., et al., 2024

    Introduces Group Relative Policy Optimization (GRPO): score each sample against the other samples for the same question instead of training a value network.

13. [13]

    s1: Simple Test-Time Scaling(opens in a new tab)

    Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., Hashimoto, T., 2025

    Fine-tunes Qwen2.5-32B on 1,000 reasoning traces and controls thinking length with “budget forcing”: cut thinking off, or append “Wait” to extend it. Extending lifts AIME24 from 50% to 57%.

14. [14]

    What’s new in Claude Opus 5.5(opens in a new tab)

    Anthropic, 2026

    Developer documentation: pricing, always-on adaptive thinking, default effort medium, and behaviour differences from Opus 5.

15. [15]

    Do NOT Think That Much for 2+3=? On the Overthinking of o1-like LLMs(opens in a new tab)

    Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., et al., 2024

    Reasoning models spend far more tokens than needed on easy problems; proposes efficiency metrics and training that cuts the waste without losing accuracy.

16. [16]

    Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting(opens in a new tab)

    Turpin, M., Michael, J., Perez, E., Bowman, S. R., 2023

    Biasing features such as always putting the right answer in position (A) sway models’ answers, yet the chains of thought never mention them; accuracy drops by as much as 36% on BIG-Bench Hard tasks. NeurIPS 2023.

17. [17]

    Reasoning Models Don’t Always Say What They Think(opens in a new tab)

    Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., et al. (Anthropic), 2025

    When a hint changed their answer, Claude 3.7 Sonnet mentioned it 25% of the time and DeepSeek R1 39%. In reward-hacking environments models exploited the hack in over 99% of cases but admitted it under 2% of the time. arXiv 2505.05410.

18. [18]

    Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation(opens in a new tab)

    Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., Farhi, D., 2025

    A weaker model (GPT-4o) reading the chain of thought catches a stronger model’s reward hacking; but training against that monitor teaches the model to hide its intent while still hacking.