Interpretability

Why look inside

We can build a model that writes working code, and we can read every one of its billions of numbers. We still cannot say, in general, why it wrote the line it wrote. Interpretability is the attempt to close that gap.

What is interpretability?

Interpretability is the study of what a trained model has learned and how it uses it. The weights of a neural network are not secret; the problem is that they are not written in any language we understand. Training produces a program, but it is spread across millions of multiply-adds with no variable names, no comments and no reason to be tidy.

The goal is to translate that program into terms a person can check: this internal variable tracks whether the text is in French, this group of attention heads copies a name from earlier in the sentence, this direction in activation space makes the model more flattering. A good translation should be testable: if we claim a part of the network does something, changing that part should change the behaviour in the way the claim predicts.

Why does it matter?

There are four practical reasons to look inside rather than only judging outputs.

• Debugging. Models find shortcuts. One well-known image classifier recognised horses partly by a copyright watermark that happened to appear on many horse photos in its training set. It scored well on the test set, which had the same watermark. Only inspecting what it relied on revealed the problem.
• Safety. A model can behave well on every test we think of and still have learned something we would not endorse. In 2026, Anthropic researchers reading Claude’s internal activations found it privately suspected it was being evaluated on 26% of problems in a coding benchmark, against less than 1% of real conversations, without saying so. Behavioural tests alone cannot see that.
• Trust and accountability. Regulators, doctors and loan applicants increasingly ask why a system decided something. The AI Ethics lesson covers explanations for individual decisions; this lesson is about the machinery underneath.
• Science. Neural networks have learned to do things nobody knows how to program by hand. Reading out how they do it can teach us about language, vision and computation itself.

How do people try to explain a model?

Explanations come at three depths. Each is useful, and each answers a different question. Most of this lesson is about the third, which is the youngest and the most ambitious.

When explanations mislead

The oldest tool for looking inside a vision model is the saliency map: colour each pixel by how much the output would change if you nudged it, usually computed from the gradient. The maps are persuasive, because they tend to highlight the object you expected.

That persuasiveness is the problem. In 2018, Adebayo and colleagues ran two simple sanity checks: re-compute each saliency method after randomising the model’s weights, and after training the model on shuffled labels. An explanation of the model should change dramatically when the model is destroyed. For several popular methods it barely changed: the maps were mostly showing edges in the image, which look like an explanation whatever the network is doing.

What is mechanistic interpretability?

Mechanistic interpretability tries to reverse-engineer a network the way you might reverse-engineer a compiled program. The core vocabulary comes from “Zoom In”, a 2020 Distill article by Chris Olah and colleagues, which studied an image model and made three claims:

1. Features are the fundamental unit: directions in activation space that stand for something, like “curve at this angle” or “dog snout”.
2. Circuits are features connected by weights. You can read the weights between features and see, for example, curve detectors being assembled from edge detectors.
3. Universality: similar features and circuits form in different networks trained on similar data.

The rest of this lesson follows that programme from the smallest scale up: single neurons, then directions, then the superposition problem that makes neurons hard to read, then the tools built to solve it, then circuits in language models, and finally the honest state of the field in 2026.

Key takeaways

• Interpretability asks what a trained network has learned and how it uses it, in terms a person can check.
• Behavioural tests show what a model does; attributions show which inputs mattered; mechanistic interpretability tries to describe the computation.
• Plausible-looking explanations can be unfaithful (saliency maps survived randomising the model), so every claim needs a test that could fail.

What a neuron computes

The simplest way to understand a network is to ask each neuron what it responds to. In a small network you can answer exhaustively: show it every possible input and paint its response.

What does a single neuron compute?

A neuron takes a weighted sum of its inputs, adds a bias and applies a nonlinearity. In the first hidden layer of a network with two inputs x and y, that is tanh(w₁x + w₂y + b): a soft step across a straight line. Deeper neurons take weighted sums of those steps, so they can respond to bands, corners, blobs and rings. If the maths is new, the Neural Networks lesson builds it up from one perceptron.

Because this network has only two inputs, we can do what is impossible for a language model: evaluate each neuron at every point of its input space and draw the result as a map. That map is the neuron’s complete behaviour.

Before training, every neuron is a random tilted ramp. After training, the first layer’s ramps line up with the parts of the boundary the network needs, and layer 2 combines them: on the circle task you will usually find layer-2 neurons that are bright inside the circle and dark outside, built from several layer-1 ramps with teal (excitatory) and red (inhibitory) weights. That is a circuit in miniature.

You will also find neurons that are hard to name: half-ramps, neurons that mirror others, neurons that barely vary. Even here, at 16 neurons, a clean one-neuron-one-meaning story is the exception. Retrain with a new seed and the neurons change, while the output looks much the same. Two networks can solve the same task with different internal variables.

How is this done for real models?

Real models have inputs with thousands or millions of dimensions, so there is no plane to paint. Instead researchers use two tools. Dataset examples: run a large corpus through the model and collect the inputs that activate a neuron most strongly. Feature visualisation: optimise an input from scratch to maximise the neuron, which, for image models, produces the surreal pictures of dog faces and textures from the Distill articles. Both can mislead in the ways the previous section warned about: the top examples show what a neuron does at its extreme, not what it does most of the time.

Why does it matter?

Neuron-level analysis produced the first detailed circuits. In the InceptionV1 image model, researchers found curve detectors, traced them back to the edge detectors they are built from, and confirmed the story by editing the weights. But the same work found polysemantic neurons that respond to unrelated things, one example responding to cat faces, fronts of cars and cat legs. In language models polysemanticity is the norm. The next sections explain why, and what to do about it.

Key takeaways

• A first-layer neuron draws one soft line; deeper neurons combine earlier ones into curves, blobs and rings.
• In small models you can map a neuron’s entire behaviour; in large ones you rely on top-activating examples and feature visualisation, which only show the extremes.
• Clean single-meaning neurons are the exception: many neurons respond to several unrelated things (polysemanticity).

Linear probes

If individual neurons are hard to read, ask a different question: is the information in there at all? A probe is a small classifier trained to read a concept out of a layer’s activations.

What is a probe?

Freeze the model. Run labelled examples through it and record one layer’s activation vectors. Then train a simple classifier, usually logistic regression, to predict the label from those vectors. If it succeeds on held-out examples, the concept is linearly decodable at that layer. Alain and Bengio introduced linear probes in 2016 as a way to watch information become more accessible layer by layer.

The trained probe also gives you a direction: its weight vector w. Projecting an activation onto w gives a single number that measures “how much of this concept is here”. That direction is what later sections steer and ablate.

A linear probe on activation vector a. σ is the logistic function. Only w and b are trained; the model is frozen.

For inside the circle, the input layer sits at chance: no straight line separates the inside of a circle from the outside. From layer 1 onward the concept becomes linearly readable, because the network has built it. The untrained network never gets there, which tells you the result comes from training, not from the architecture. Erasing the probe’s direction breaks the network’s task: the probe found something the model uses.

Now switch to right half, a concept the task never needs (a circle is symmetric). It is perfectly decodable from the input and still highly decodable in layer 2, yet erasing that direction barely changes the output. By layer 3 the network has discarded it. The probe found information that is present, not information that is used.

Why does it matter?

Probes are cheap, and they work surprisingly well. Researchers have used them to find linear representations of truthfulness, sentiment, board state in game-playing models, and whether a prompt is harmful. In a 2025 study, Google DeepMind’s interpretability team found that a plain linear probe detected harmful user intent almost perfectly, even on data unlike its training set, and did better than more sophisticated methods. For monitoring a deployed model, a well-validated probe is often the most practical interpretability tool there is.

Key takeaways

• A linear probe is a small classifier trained on frozen activations; its weight vector gives a direction for the concept.
• Probe accuracy by layer shows where a network builds a concept. Always compare against chance and an untrained network.
• Decodable is not used: only an intervention, such as erasing the direction, shows the model depends on it.

Features and superposition

Why are neurons so often polysemantic? One compelling answer: models need to represent far more concepts than they have neurons, and they have found a way to cheat.

What is a feature?

A feature is a property of the input that the network represents, like “this token is part of a URL” or “the text is about the Golden Gate Bridge”. The working hypothesis in the field is the linear representation hypothesis: each feature corresponds to a direction in activation space, and an input’s activation is roughly a sum of the directions of the features it contains, each scaled by how strongly it is present.

If every feature had its own neuron, each direction would be a single axis and neurons would be easy to read. But a language model has a few thousand dimensions per layer and plausibly needs millions of features: every person, place, programming idiom and turn of phrase it knows.

How can a model store more features than dimensions?

In a space with n dimensions you can only fit n directions that are exactly perpendicular. But you can fit many more that are almost perpendicular, and the number grows exponentially with n. The price is interference: when two non-perpendicular features are active together, each leaks into the other’s read-out.

The key is sparsity. Most features are absent most of the time: an arbitrary sentence is not about the Golden Gate Bridge and does not contain a URL. If features rarely co-occur, they rarely interfere, and the network can pack them in densely and clean up the small leaks with a nonlinearity. Elhage and colleagues demonstrated this in a model small enough to see completely, and named it superposition.

The toy model. Five input features are squeezed through two hidden dimensions and must be reconstructed. W is 2 × 5; the ReLU and bias b can filter out small interference.

Each of the five features is zero with probability S (the sparsity) and otherwise a random value between 0 and 1. Features are weighted by importance, from 1.0 for f1 down to 0.41 for f5, so the loss cares most about getting f1 right. With only two hidden dimensions, how many features will the model choose to represent?

With dense features the model behaves like classical dimensionality reduction: it keeps the two most important features on perpendicular axes and drops the rest. As features get sparser, the extra features appear. Around 20% density you typically see antipodal pairs, two features sharing one axis in opposite directions (ReLU can separate them because they are rarely on together). At high sparsity all five spread into a pentagon: five features in two dimensions. The off-diagonal entries of WᵀW show the interference being tolerated, and the bias turns negative to filter it out.

These are exactly the geometries reported in the paper, which also found sharp transitions between them as sparsity and importance change, like phase changes in physics.

Why does it matter?

Superposition explains polysemantic neurons. If a layer stores more features than it has neurons, the features cannot all line up with neuron axes, so each neuron reads a mixture. Look at the two hidden dimensions in the demo: each one has a large component from several features. Staring at neurons harder will not fix this: we need a way to find the feature directions themselves.

In this toy model the hidden space has no preferred axes, so “neuron” just means one coordinate. In real networks, neurons follow nonlinearities and do have a preferred basis, and the paper shows superposition still happens there. The toy model is evidence that superposition is possible and useful, not proof that large models use it in exactly this way; that link is supported by the dictionary-learning results in the next section.

Key takeaways

• A feature is a property the network represents, hypothesised to be a direction in activation space.
• When features are sparse, a network can store more features than dimensions by using almost-perpendicular directions: superposition.
• Superposition makes neurons polysemantic, so reading neurons one by one cannot recover the features.

Sparse autoencoders

If a model stores features in superposition, the obvious move is to learn to un-mix them: find a large set of directions such that every activation is a sparse combination of a few of them.

What is a sparse autoencoder?

A sparse autoencoder (SAE) is a second, small network trained on a model’s activations. It encodes each activation vector into a much wider vector of latents, with a penalty that forces almost all of them to be zero, then decodes back to the original activation. Each latent’s decoder column is a candidate feature direction. This is a form of dictionary learning, a technique with a long history in signal processing and neuroscience.

The training loss: reconstruct the activation (first term) while keeping the code sparse (second term). λ sets the trade-off.

With a moderate sparsity penalty the learned directions snap onto the five true feature directions, even though the autoencoder never saw the features, only the 2D activations. That is the core promise: unsupervised recovery of the variables a model uses.

The failure modes are just as instructive. With λ = 0 reconstruction is perfect, but nothing forces latents to align with features: two arbitrary directions can already rebuild any 2D point. With too few latents some features must share a latent. With too many, spare latents die (never fire) or duplicate a feature. Real SAEs show all of these, plus feature splitting: as the dictionary grows, one broad feature splits into several finer ones, so there is no single “true” number of features.

How far has it scaled?

In 2023, Anthropic applied SAEs to a one-layer transformer and turned its 512 MLP neurons into thousands of features that were far more interpretable than the neurons: features for DNA sequences, Arabic script, base64 strings and legal language. A year later, the same approach scaled to a production model, Claude 3 Sonnet, with dictionaries of about 1, 4 and 34 million features. Features there were multilingual and even multimodal: the Golden Gate Bridge feature fired on the name in several languages and on images of the bridge. The team also reported features related to safety concerns, including deception, sycophancy, bias and security vulnerabilities in code.

Other labs followed. OpenAI trained a 16-million-latent autoencoder on GPT-4 using TopK autoencoders, which keep exactly k latents per input instead of using an L1 penalty, and found clean scaling laws for how reconstruction improves with size. Google DeepMind released open SAEs for Gemma 2 in 2024 and then Gemma Scope 2 in December 2025: SAEs and transcoders for every layer of every Gemma 3 model from 270M to 27B parameters, so anyone can explore features in an open model.

What are the limits?

• Dark matter. SAEs do not reconstruct activations perfectly, and when the reconstruction is substituted back into the model its performance drops. Whatever is in the error term is unexplained.
• Not unique. Different dictionary sizes and random seeds give different, partly overlapping feature sets. Try pressing reset in the demo at λ = 0.1: sometimes a latent settles between two features.
• Labels are human guesses. A latent labelled “Golden Gate Bridge” from its top examples may also fire weakly on other things; checking every latent carefully is impossible at 34 million.
• Useful for what? The DeepMind team that found linear probes beat SAE-based probes on harmful-intent detection publicly deprioritised fundamental SAE research in 2025, arguing SAEs are good for exploration but not yet clearly better than simpler baselines on practical tasks.

Key takeaways

• A sparse autoencoder learns an overcomplete dictionary of directions so that each activation is a sparse sum of a few of them.
• SAEs now run on production models (34M features on Claude 3 Sonnet, 16M latents on GPT-4, open suites for Gemma) and find many interpretable, multilingual features.
• They leave unexplained error, depend on size and seed, and do not always beat simple probes on practical tasks.

Circuits

Features are the variables. Circuits are the program: features in one layer, connected by weights and attention to features in the next, together carrying out a recognisable algorithm.

What is a circuit?

A circuit is a subgraph of the network that implements a specific behaviour: a small set of components (neurons, features, attention heads) and the connections between them, which on their own are enough to explain that behaviour. A good circuit claim is precise enough to be falsified. It says which components matter, what each one computes, and what should happen if you remove or change any of them.

Transformers make circuits easier to see than you might expect. Each layer reads from and writes to a shared vector, the residual stream, so a component in layer 5 can directly read what a head in layer 1 wrote. Attention heads move information between positions; MLP layers transform it in place. If attention is new to you, the Transformers lesson explains queries, keys and values first.

Induction heads

The best-understood circuit in language models is a two-head algorithm for continuing patterns. Suppose the text so far contains “Mr and Mrs Dursley” and now ends with “Mr and Mrs”. A good guess for the next token is “Dursley”: whatever followed this token last time. Two heads implement that guess:

1. A previous-token head in an early layer makes each position attend to the one before it and copy in that token’s identity. Now the position of “Dursley” carries the note “I was preceded by Mrs”.
2. An induction head in a later layer, at the current position “Mrs”, uses a query that asks “which position was preceded by Mrs?”. It matches the note, attends to “Dursley” and copies it to the output.

Notice that it works just as well on random tokens: the circuit does not know any facts, it copies patterns. That is the signature researchers look for in trained models. Olsson and colleagues found that induction heads appear in a sudden phase change early in training, visible as a bump in the loss curve, at the same moment the model gets much better at using earlier context. They argued, with strong evidence in small models and more indirect evidence in large ones, that induction heads are a major mechanism behind in-context learning.

A circuit for choosing a name

Complete this sentence: “When Mary and John went to the store, John gave a drink to …”. GPT-2 small reliably says Mary. The task is called indirect object identification (IOI). Wang and colleagues traced the circuit responsible and found 26 attention heads in 7 functional classes. Its core is a three-step algorithm.

The surprise was not the algorithm, which is roughly what a person would write, but the redundancy. When the researchers knocked out the main name movers, backup heads that normally do little started doing their job. Networks are not built with one wire per function, which makes “this component is responsible” claims harder to pin down.

Attribution graphs in a production model

Finding the IOI circuit took months of work on one behaviour in a small model. In 2025, Anthropic published a method to do this more automatically. It replaces the model’s MLP layers with a cross-layer transcoder, a sparse dictionary that approximates what the MLPs compute, and then traces how features influence each other on one specific prompt. The result is an attribution graph: a map of which features caused which, from input tokens to output.

Applied to Claude 3.5 Haiku, these graphs showed recognisable intermediate steps. Asked for the capital of the state containing Dallas, the model activates features for Texas, then combines them with “say a capital” to produce Austin. When writing a rhyming couplet it selects a rhyme word before starting the second line and then writes towards it. For some maths problems its written explanation describes the schoolbook method while the internal features show a different, parallel strategy.

Why does it matter?

Circuits are the strongest kind of evidence interpretability produces: not “this concept is present” but “this is how the answer was computed, and here is the intervention that proves it”. They also reveal things behavioural tests cannot, such as planning ahead, or a stated chain of reasoning that does not match the computation that produced the answer.

Key takeaways

• A circuit is a small set of connected components that, on its own, explains a behaviour, and predicts what interventions will do.
• Induction heads copy “what came after this token last time”; they emerge suddenly in training alongside in-context learning.
• Attribution graphs trace feature-to-feature influence in production models, but so far explain only part of the computation on a minority of prompts.

Patching and steering

Every serious claim in this lesson was checked the same way: reach into the running network, change something, and see if the output changes as predicted. This is what separates mechanistic interpretability from storytelling.

What is activation patching?

Take two inputs that differ in one important way: a clean input where the model does the behaviour and a corrupted input where it does not. In IOI, the clean prompt ends “John gave a drink to” and the model says Mary; the corrupted prompt swaps in different names so the answer changes. Run both. Then rerun the corrupted input, but overwrite one component’s activation with the value it had in the clean run.

If the output moves back towards the clean answer, that component carries information that matters for the difference. Repeat for every component and you get a map of where the difference lives. Refinements such as path patching, which only patches the effect along one connection, let researchers trace whole circuits, and were how the IOI circuit was found.

Two patterns usually stand out. In layer 1, a single neuron can restore the whole gap on its own: everything downstream depends on it, so it acts like a gate for this pair of inputs. In layer 3, recovery is spread across all eight neurons in similar slices, because the output is simply a weighted sum of them. Move B to the other side of the circle and different gates light up: patching tells you what matters for this comparison. Values below 0 or above 100 are real too, since a patched neuron can disagree with its neighbours. That is why papers report patching over many input pairs and read single numbers with care.

What is steering?

If a direction in activation space really represents a concept, adding more of it should produce more of the concept. Turner and colleagues showed this with a simple recipe called activation addition: record GPT-2-XL’s activations for “Love” and for “Hate”, take the difference, and add a multiple of it at one layer while the model generates. Completions shift towards affectionate language, with no retraining.

The most famous example came from the sparse autoencoder work. Anthropic clamped Claude 3 Sonnet’s Golden Gate Bridge feature to a high value and released it for 24 hours as “Golden Gate Claude”. Asked how to spend ten dollars, it recommended driving across the bridge; asked for a love story, it wrote about a car longing to cross the bridge on a foggy day. In the paper, the steered model even described itself as the bridge.

Pushing along the probe direction grows or shrinks the region the network calls “inside”, smoothly and predictably. A random direction of the same size does much less, or distorts the boundary in arbitrary ways. That contrast is the whole argument: the probe found a direction the network’s later layers actually read as “inside”.

Why does it matter, and what are the catches?

Steering turns interpretability into a control tool and a diagnostic. When fine-tuning GPT-4o on insecure code made it misbehave in unrelated ways, OpenAI researchers compared SAE features before and after and found a “toxic persona” feature that predicted the misbehaviour and could switch it up or down. Researchers now build monitors from probe and feature directions and experiment with steering away from sycophancy or refusal. The catches: strong steering degrades fluency and general capability (Golden Gate Claude was obsessive, not smarter), a direction can carry side effects nobody labelled, and a behaviour that is steerable is not necessarily one you have fully understood.

Key takeaways

• Activation patching copies a component’s value from a clean run into a corrupted one to measure what it contributes to a specific difference.
• Steering adds a feature direction to the activations; if the behaviour changes as predicted, the direction is causally meaningful.
• Interventions are what make interpretability claims testable, but backups, interactions and side effects mean results must be read with care.

What remains unsolved

Interpretability has gone from curve detectors in 2020 to feature maps of frontier models in 2026. It is still far from being able to explain, in full, why a large model gave a particular answer.

What can the field do today?

• Find many human-interpretable features in production models, and steer behaviour with some of them.
• Trace partial circuits for specific prompts and confirm them with interventions.
• Build cheap, accurate probes that monitor for concepts such as harmful intent.
• Help auditors find hidden problems. In one 2026 study, auditors using a tool that translates activations into text uncovered a model’s deliberately hidden motivation 12 to 15% of the time, against under 3% without it. Useful, and also a measure of how far there is to go.

What is unsolved?

A 2025 review by researchers across many labs, “Open Problems in Mechanistic Interpretability”, lists the gaps plainly. The main ones:

• Coverage. Current methods explain a fraction of the computation on a fraction of prompts. The unexplained remainder is where surprises would hide.
• Foundations. There is no agreed definition of a feature. The linear representation hypothesis works often but not always; some concepts appear to live on curved manifolds or across many layers.
• Validation. Without ground truth for large models, it is hard to know whether an explanation is correct or merely plausible, the same trap saliency maps fell into.
• Cost. Training dictionaries for every layer of a frontier model takes large amounts of compute and storage, and human analysis does not scale to millions of features.
• Scaling the conclusions. A circuit found in GPT-2 small may not exist, or may be implemented differently, in a model 1,000 times larger.

How far can interpretability go?

Researchers disagree, and the disagreement is informative. The optimistic case: progress has been fast, tools that worked on toy models transferred to production models within two years, and automated methods (models explaining models) could scale analysis beyond human limits. Anthropic’s CEO has set a goal that interpretability can reliably detect most model problems by 2027, likening the aim to an MRI for AI.

The sceptical case: a complete, human-understandable account of a trillion-parameter network may not exist, just as there is no short explanation of every neuron in a brain. Some researchers therefore favour a pragmatic goal: tools that catch specific problems, such as a probe for deception or an audit for hidden objectives, used alongside behavioural evaluations, rather than full understanding. The DeepMind team’s decision to redirect effort from SAEs to simpler, task-focused methods is one example of that shift.

Both camps agree on one thing: models are becoming more capable faster than we are learning to read them. The AI Ethics lesson discusses how interpretability fits with other safety tools such as evaluations, red-teaming and oversight.

Key takeaways

• Today’s tools find real features, partial circuits and useful probes, and are starting to help audits, but explain only part of any large model’s computation.
• Open problems include coverage, the definition of a feature, validating explanations without ground truth, and cost.
• Researchers disagree on whether full understanding is achievable; targeted tools used alongside evaluations are the practical near-term goal.

Check your understanding

Seven scenarios. Each asks you to judge an interpretability claim or predict what a method will show.

A team trains a logistic-regression probe on layer 20 of a language model and decodes “the user is lying” with 94% held-out accuracy. They announce the model uses a lie detector. What is the most important missing step?

References

Primary sources for the lesson. Most mechanistic interpretability research is published on the Transformer Circuits Thread and Distill, which are free to read and full of interactive figures; they are the best next step.

Sources

1. [1]

   Unmasking Clever Hans predictors and assessing what machines really learn(opens in a new tab)

   Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.-R., 2019

   Nature Communications 10, 1096. Shows an image classifier that recognised horses partly by a copyright watermark, found by inspecting its explanations.

2. [2]

   Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations(opens in a new tab)

   Fraser-Taliente, K., Kantamneni, S., Ong, E., Mossing, D., Lu, C., Bogdan, P. C., et al., 2026

   Transformer Circuits Thread (May 2026). One model turns an activation into text, another rebuilds the activation from the text; used to audit Claude models for unspoken evaluation awareness.

3. [3]

   Sanity Checks for Saliency Maps(opens in a new tab)

   Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B., 2018

   NeurIPS 2018. Randomising a model’s weights or training labels leaves some popular saliency maps almost unchanged, so those maps cannot be explaining the model.

4. [4]

   Zoom In: An Introduction to Circuits(opens in a new tab)

   Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., Carter, S., 2020

   Distill. Argues that features are the fundamental unit of neural networks and that features connect into circuits, with curve detectors in InceptionV1 as the worked example.

5. [5]

   Understanding intermediate layers using linear classifier probes(opens in a new tab)

   Alain, G., Bengio, Y., 2016

   Introduces linear probes: train a simple classifier on a frozen layer’s activations to measure what information that layer makes linearly available.

6. [6]

   Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research(opens in a new tab)

   Smith, L., Rajamanoharan, S., Conmy, A., McDougall, C., Lieberum, T., Kramár, J., Shah, R., Nanda, N., 2025

   Google DeepMind progress update (March 2025). For detecting harmful intent out of distribution, plain linear probes beat probes built on SAE features.

7. [7]

   Toy Models of Superposition(opens in a new tab)

   Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., et al., 2022

   Transformer Circuits Thread. Small ReLU models store more sparse features than they have dimensions, arranging them in geometric patterns such as antipodal pairs and pentagons.

8. [8]

   Towards Monosemanticity: Decomposing Language Models With Dictionary Learning(opens in a new tab)

   Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., et al., 2023

   Transformer Circuits Thread. A sparse autoencoder turns the 512 neurons of a one-layer transformer’s MLP into thousands of mostly single-meaning features.

9. [9]

   Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet(opens in a new tab)

   Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., et al., 2024

   Transformer Circuits Thread (May 2024). Sparse autoencoders with about 1M, 4M and 34M features on a production model’s middle layer; features are multilingual, multimodal and can steer behaviour.

10. [10]

    Scaling and evaluating sparse autoencoders(opens in a new tab)

    Gao, L., Dupré la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., Wu, J., 2024

    OpenAI. Introduces k-sparse (TopK) autoencoders, finds clean scaling laws, and trains a 16-million-latent autoencoder on GPT-4 activations for 40 billion tokens.

11. [11]

    Gemma Scope 2: helping the AI safety community deepen understanding of complex language model behavior(opens in a new tab)

    Google DeepMind Language Model Interpretability Team, 2025

    December 2025. Open sparse autoencoders and transcoders for every layer of every Gemma 3 model from 270M to 27B parameters, following the 2024 Gemma Scope release for Gemma 2.

12. [12]

    In-context Learning and Induction Heads(opens in a new tab)

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., et al., 2022

    Transformer Circuits Thread. Induction heads form in a sudden phase change early in training, at the same time as a jump in in-context learning ability.

13. [13]

    Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small(opens in a new tab)

    Wang, K., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J., 2022

    ICLR 2023. Uses causal interventions to find a circuit of 26 attention heads in 7 classes that completes sentences like “When Mary and John went to the store, John gave a drink to …”.

14. [14]

    On the Biology of a Large Language Model(opens in a new tab)

    Lindsey, J., Gurnee, W., Ameisen, E., et al., 2025

    Transformer Circuits Thread (March 2025). Attribution graphs built from a cross-layer transcoder reveal multi-step reasoning, planning in poetry and other mechanisms in Claude 3.5 Haiku; companion to “Circuit Tracing” (Ameisen et al.).

15. [15]

    Steering Language Models With Activation Engineering(opens in a new tab)

    Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., MacDiarmid, M., 2023

    Activation addition: subtract the activations for one prompt (“Hate”) from another (“Love”) and add the difference during generation to steer the topic or sentiment of GPT-2-XL.

16. [16]

    Golden Gate Claude(opens in a new tab)

    Anthropic, 2024

    A 24-hour public research demo (May 2024) of Claude 3 Sonnet with its Golden Gate Bridge feature clamped to a high value.

17. [17]

    Persona Features Control Emergent Misalignment(opens in a new tab)

    Wang, M., Dupré la Tour, T., Watkins, O., et al., 2025

    OpenAI. Fine-tuning GPT-4o on insecure code causes broad misbehaviour; comparing SAE features before and after reveals a “toxic persona” feature that predicts and controls it.

18. [18]

    Open Problems in Mechanistic Interpretability(opens in a new tab)

    Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., et al., 2025

    A multi-institution review of what the field cannot yet do: conceptual foundations, methods, validation and applications.

19. [19]

    The Urgency of Interpretability(opens in a new tab)

    Amodei, D., 2025

    Essay (April 2025) by Anthropic’s CEO arguing for an “MRI for AI”, with the goal that interpretability can reliably detect most model problems by 2027.