Watching a circuit switch on, and why the loudest head is the wrong way to find it
An is the best-understood circuit in interpretability: it notices that the current token appeared earlier and predicts whatever followed it last time. Models are not born with one. It appears at a particular moment in training — a — and you can watch it happen if you have from during the run.
How you score a head, rather than listen for it
The is a sequence of random tokens repeated twice. Nothing in it can be predicted from language: the only usable information is the repeat, so a head that scores well is doing induction and not remembering a common phrase. For every position in the second copy, the answer sits exactly 48 tokens back.
A head doing induction makes the blue hop: from where it is now, back to where that same token appeared before. What it reads there is the token that followed last time, and that is its prediction. The score this page plots is how much of the head’s attention lands on that one position.
Three numbers come out of each checkpoint. How much attention the best head puts on that target. How much attention it puts on the first token, the , which is where a head parks when it is doing nothing. And the number this page was built for: which head the old loudest-hop rule would have picked, and what that head scores on induction.
The scorer is imported from the same file the site's guided tour uses, rather than reimplemented here, so the checkpoint-level and corpus-level definitions cannot drift apart. Nineteen published revisions of Pythia-160M, eight prompts, one seed.
Nothing, nothing, nothing, then everything
For 512 steps there is no induction in this model. The best head scores 0.016 to 0.019 against a 0.014 mean-head baseline, which is the difference between weak and absent.
One bracket later a head is attending to the induction target at 98%, and it stays there for the remaining 142,000 steps. That is a 23× jump between two adjacent checkpoints, and it reproduces a known phase change on a model nobody here trained, using this site's own scorer.
| checkpoint | best induction | mean head | best head | its sink |
|---|---|---|---|---|
| step 0 to 8 | 0.016 | 0.0142 | L9·H9 | 0.049 |
| step 256 | 0.0186 | 0.0125 | L5·H9 | 0.058 |
| step 512 | 0.0432 | 0.0137 | L4·H10 | 0.063 |
| step 1000 | 0.9803 | 0.0448 | L4·H10 | 0.102 |
| step 2000 | 0.9823 | 0.0484 | L4·H6 | 0.137 |
| step 32000 | 0.9676 | 0.071 | L4·H6 | 0.745 |
| step 143000 | 0.9563 | 0.0972 | L4·H6 | 0.925 |
Two details worth keeping. The head is identifiable before it works: L4·H10 is already the best induction head at step 512, while still scoring 0.043, first among nothing.
The baseline rises too, 0.014 to 0.097 across training, so the best-versus-average ratio falls from about 70× at onset to about 10× at the end even though the best head barely moves. A ratio quoted without its step would be misleading in either direction, which is why the grey line is on the chart.
Meanwhile the loud thing gets louder
The red line is the sink: attention parked on the first token, doing nothing. It climbs all through training, 0.049 to 0.925, and most of that climb happens after induction has already formed. So the thing a salience ranking finds grows steadily over exactly the period when the mechanism it is missing stays flat.
Tracking what the loudest long-range hop actually pointed at makes the failure legible. The distance column is the tell: 47 is the induction distance for these prompts.
| step | loudest head | distance of its hop | its induction | a sink? |
|---|---|---|---|---|
| 0 to 64 | L1·H2 | 25 | 0.0142 | no |
| 128 to 512 | drifts | 28 to 36 | 0.017 | no |
| 1000 to 16000 | L4·H6, L4·H10 | 47 | 0.885 | no |
| 32000 | L11·H3 | 75, onto position 0 | 0.0021 | yes |
| 64000 to 143000 | L9·H5, L4·H10 | 47 | 0.616 | no |
Post-onset the loudest hop sits at distance 47 at every checkpoint but one. At step 32000 it jumps to a hop of 75 onto position 0, and the picked head scores 0.0021 on induction while a head scoring 0.9676 is sitting in the same model.
This is the failure mode the page is meant to isolate: the miss is a sink, and it happens when the sink out-shouts a real head.
So the proxy fails in two different ways depending on where you are in training. Before onset it is uninformative, not imprecise: the head it picks scores 0.0142, exactly the mean-head baseline. It is not finding a weak induction head, it is finding a head that does none, because none exists yet.
After onset it is unreliable: usually an induction head, often not the best one, and at least once a sink that does nothing. Nothing in the salience score distinguishes those cases.
The practical rule, and the reason this page sits beside the GPT-2 trace that exposed the same failure mode: salience is not evidence. Rank by the behaviour you mean to find, not by how loud something is.
What this does not show
One model, one size, one prompt family. Repeated random tokens isolate induction, so 0.98 is the mechanism's ceiling under an ideal probe rather than what it does in the wild: measured on natural text, our own 0.1M models score 0.022 to 0.027. Whether onset timing moves with model size or data order is untested here.
The onset is bracketed, not located, and the step-32000 result rests on one configuration with no seed spread, so it should be replicated before it is leaned on. Why that checkpoint specifically is not explained: we know the miss is a sink, not why the sink wins there.
One curiosity from the audit that came free: step 0 and step 1 are byte-identical, 0 of 148 tensors different, so the run has 19 revisions and 18 distinct models.