Glossary

A positional scheme that skips embeddings entirely: it subtracts a fixed, distance-proportional penalty straight from each attention score, so farther-back tokens are down-weighted directly rather than tagged with a position. An alternative to RoPE, tested here as a check on whether a length limit comes from the position scheme rather than the model.

See alsoRoPEPositional encoding

tested against RoPE on the same limit

Width (n_embd) against depth (n_layer) at a fixed parameter count. Same size, different shape - and the shape matters less than the scaling-law fits suggest.

See alsoHead dimensionScaling law

width vs depth, measured

The mechanism letting each token pull information from earlier tokens, weighted by relevance. The weights are what the trace views draw: a row per token, showing where it looked.

See alsoAttention headSoftmax attentionDelta rule (fast-weight) attention

The context length: how many past tokens the model can attend to at once. Text longer than this has to be windowed, which is a real failure source in the restorer.

Byte-pair encoding: a learned vocabulary of subword pieces, so common words are one token. Shorter sequences than character-level, at the cost of a fixed lexicon.

See alsoCharacter-levelToken

One token per character. A tiny vocabulary and long sequences, but nothing is ever out-of-vocabulary - which is why the restorer, a spelling task, is character-level.

See alsoBPERestorer

char vs BPE, measured

A learned forget control on a fast-weight memory: before each write the existing memory is multiplied by a value between 0 and 1, so old contents fade unless something refreshes them. Without it the memory only accumulates. Two published granularities exist - one value per head (Gated DeltaNet) and one per channel (Kimi Delta Attention) - and here they tie, with the per-head form matching the finer one at 1/32 the gate parameters while both beat an ungated memory by 0.21 nats. The gate is the mechanism; the extra parameters it came wrapped in are not.

See alsoFast weightsDelta rule (fast-weight) attentionWrite gate (β)Param-matched control

what happens when you also tell it the order

A linear-attention variant that carries a running state matrix and edits it per token instead of re-reading the whole past. Constant memory in sequence length; a different, more overwriting kind of memory.

See alsoFast weightsDecay gate (α)Softmax attentionAttention

softmax vs delta, same input

The alternative to re-reading the past: instead of scoring the current token against every earlier one, the layer keeps a single memory matrix and writes each token into it as it goes, then reads that memory to produce its output. Schmidhuber proposed it in 1992 and it returned as linear attention. Attention re-reads the whole history at quadratic cost; fast weights carry a running summary of it at constant cost, and the interesting questions are what gets written (the write gate) and what gets forgotten (the decay gate).

See alsoDelta rule (fast-weight) attentionDecay gate (α)Write gate (β)Softmax attention

Attention runs several times in parallel with different learned projections; each copy is a head. Heads specialise, which is why the viewers let you pick one at a time.

See alsoAttentionHead dimension

Width per head (n_embd / n_head). It sets how much room each head has to work in, and at fixed parameters it trades off against how many heads you get.

See alsoAttention headAspect ratio

A learned gate on the residual: the block decides how much of its output to write into the stream rather than always adding all of it. One of the few selectivity mechanisms here that actually won.

See alsoResidual stream

Generation reuses each token’s attention keys and values instead of recomputing the whole prefix per step. Same output, roughly linear rather than quadratic decode.

the live decoder

The raw per-token scores the model outputs before softmax turns them into probabilities. The top few, and how far apart they are, is what the probability strips show.

See alsoSoftmax attentionLogit lens

Supplying no positional encoding at all and letting a causal model infer order from the shape of the computation itself (Kazemnejad et al. 2023). Not an absence of a decision: on this repo's 3.4M arms softmax gets measurably worse without position, plain fast weights do not care, and a gated fast-weight arm gets better, so what NoPE costs or buys depends on what else in the architecture already encodes recency.

See alsoPositional encodingDecay gate (α)The positional confound

three architectures, three answers

The model’s adjustable internal numbers - what training changes and what a checkpoint stores. The models trained here run from 0.2M to 91M of them; the largest thing on the site is the off-the-shelf GPT-2 at 124M. All tiny by current standards.

See alsoParam-matched control

A transformer has no built-in sense of order: shuffle the words and the machinery is unchanged. Positional encoding is the extra signal that supplies it, either as one learned vector per slot or as a rotation whose angle grows with position (RoPE).

See alsoRoPEAttention

when it stops helping

Storing weights in fewer bits — here one byte instead of four, which shrinks the model about 4× so it can ship to a browser. The question is always what accuracy it costs; on our 14M model, plain int8 costs +0.0018 bpc, so the extra “quantization-aware training” stage was skipped and the null recorded.

See alsoParametersbpc (bits per character)WebAssembly (WASM)

run the int8 model

The running vector each layer adds into rather than replaces. It is the model’s working memory, and reading it at intermediate depths is how the logit lens works.

See alsoLogit lens

A cheaper LayerNorm: rescale a vector by its root-mean-square without subtracting the mean. Standard in modern stacks, and one of the toggles the ablation harness can flip back.

See alsoAblation

Rotary position embedding: position is encoded by rotating each query and key by an angle proportional to where the token sits, so attention sees relative distance. It needs an even head size.

See alsoHead dimensionAttention

Scaled dot-product attention: the fused, fast implementation of standard attention that every framework ships. What "softmax attention" runs as in practice.

See alsoSoftmax attentionAttention

Standard attention: score every earlier token, normalise the scores to a distribution, mix. Quadratic in sequence length, and the baseline every other attention variant here is measured against.

See alsoDelta rule (fast-weight) attentionAttention

softmax vs delta, same input

The gated feed-forward variant used here: one branch decides how much of the other to let through. The modern default, and a measurable win over the plain version.

See alsoAblation

The atomic unit a model reads and predicts - here either a single character or a subword piece. Everything the model knows, it knows as relationships between tokens.

See alsoCharacter-levelBPETokenizer

The rulebook that splits text into the pieces a model actually sees. One tokenizer may treat a word as one piece while another splits it into several, which is why some scores on this site are only compared within the same model.

See alsoTokenBPEbpc (bits per character)

Sharing one matrix between the token embedding and the output head, since both map between tokens and vectors. Fewer parameters, and usually a slightly better model.

See alsoParametersToken

In a fast-weight memory, a number the model computes for every token deciding how hard to write that token into its memory. Unlike a training trick imposed from outside, nobody tells it what to write hard — it is learned, so auditing it asks what selectivity SGD chooses when left to itself.

See alsoDelta rule (fast-weight) attentionAttentionLearning-progress weighting

what it turned out to track

Loss converted to bits needed per character of raw text. Tokenizer-independent, so it is the only fair way to compare a character model against a BPE one.

See alsoValidation lossCharacter-level

A saved copy of a model's weights partway through training. Almost every released model is one checkpoint - the last one - which is why anything that happens *during* training is normally invisible: you only ever see the finished state. EleutherAI's Pythia publishes 19 of them for a single run, and that is the only reason the moment an induction head appears can be located here at all.

See alsoPhase changeInduction head

19 checkpoints of one run

The compute-efficient tokens-per-parameter ratio (about 20:1) from Hoffmann et al. 2022. Small models here run far past it, which is the regime the scaling page is about.

See alsoScaling law

past the optimum

Upweighting the tokens whose loss is currently falling fastest, on the theory that effort should go where the model is actually learning. Measured here across several forms: a null.

See alsoNull resultSpectral bias

How wrong the predictions are, as one number training tries to make smaller. For next-token prediction it is cross-entropy: how surprised the model was by the token that actually came next.

See alsoValidation lossnats

The unit loss is reported in here: natural-log cross-entropy. Divide by ln 2 for bits. A 0.01 nat difference is small; whether it is real depends on the seed spread.

See alsoLossSeed

Cross-entropy exponentiated, read as “how many equally-likely options was the model choosing between here?” Lower means the text looked more natural to it. This is what score.py reports and what makes the fluency-judge use work — comparable only within one model.

See alsoLossValidation lossbpc (bits per character)

score a sentence

Instead of copying training text, the model generates, gets scored, and is pushed toward higher-scoring output. How the molecule generator was tuned toward drug-likeness.

See alsoQED (drug-likeness)Genetic algorithm (Graph GA)

An empirical curve for how loss falls as you add parameters, data, or compute. Fitted on large models, then extrapolated - the question this site asks is whether it still holds far below the fitted range.

See alsoChinchilla-optimalAspect ratio

sub-Chinchilla scaling

SGD learns coarse, simple structure before fine detail - an implicit curriculum you get for free. Part of why curricula imposed by hand tend to add nothing on top.

See alsoLearning-progress weightingTraining

The loop: the model guesses the next token millions of times, and each time it is wrong its internal numbers get nudged. Nothing else happens - all the architecture choices only change what gets nudged and how.

See alsoLossParametersAdamW

Cross-entropy measured on held-out text the model never trained on. The honest score - training loss can be lowered by memorising - and what nearly every table on this site ranks by.

See alsoLossbpc (bits per character)Seed

Initialising from an already-trained model instead of from scratch, hoping its knowledge transfers. A hypothesis to be tested, not a free win - here the warm-started BPE restorer lost to a smaller model trained cold.

where it lost

How big each training step is. Too high and training diverges, too low and it crawls; the single most important knob. Here it is ramped up over a short warmup and down again late, and the Muon margin on this site changed twice when a second learning rate was tuned.

See alsoTrainingMuonAdamW

The direction and size of the nudge to each weight that would most reduce the loss, computed by backpropagation. Training is repeatedly nudging weights along it; several arms on the selectivity page keep only its largest entries.

See alsoTrainingLossLearning rate

The default optimizer nearly everywhere: per-parameter adaptive step sizes plus decoupled weight decay. Every optimizer result here is stated relative to it.

See alsoMuon

Minimal-effort backprop: keep only the top few percent of each gradient and zero the rest, so the backward pass touches less of the network. One more arm in the selectivity campaign, and one more loss.

See alsoNull result

A recent method for doing the nudging - an alternative to the long-standing default. It orthogonalises the momentum matrix (via Newton-Schulz) before stepping, so no direction dominates the update. It beats AdamW at equal parameters here - one of the few wins that replicated cleanly.

See alsoAdamWParam-matched control

a published optimizer, replicated here

Muon with the orthogonalisation applied per attention head instead of to the whole fused weight matrix. A genuine null below about 14M parameters and a real win above it, tracking head dimension rather than parameter count.

See alsoMuonHead dimensionScale trend

where it switches on

A training guardrail that caps attention logits by rescaling the query and key weights. At a tuned learning rate it is bit-identical to plain Muon and only rescues a deliberately mistuned run - a dormant guardrail, not a failed mechanism.

See alsoMuonNull result

the dormant-guardrail result

Total counts every expert’s weights; active counts only the ones a given token actually uses. Quoting the total makes a sparse model look bigger than it computes.

See alsoMoE (mixture of experts)Param-matched control

Replace one feed-forward layer with several smaller expert copies and route each token to a few of them. More total parameters, same compute per token.

See alsoRouterActive vs total parameters

The small learned layer that scores each token against every expert and picks the top-k. What it learns to specialise on is the interesting part; imposing that specialisation by hand did not work here.

See alsoMoE (mixture of experts)

Swap or remove exactly one component, hold everything else fixed, and measure. The core move of this repo: one change, several seeds, reported with its spread.

See alsoSeedNull result

When a result could just be "it had more parameters." The reason almost every claim here comes with a param-matched arm next to it.

See alsoParam-matched control

What random guessing scores — 25% on a four-choice question. A benchmark number only shows knowledge if it beats chance by more than its own error bar, which for a thousand questions is about ±1.2 points. It is why 27% is not a result.

See alsoZero-shot evaluationSeedNull result

the benchmark it flags as empty

Letting the model choose only among answers the input actually allows. For the restorer, every Latin letter licenses a short list of Cyrillic letters, so the decoder is held to that list and stops exactly when the input runs out — it can still pick the wrong letter, but it can no longer drop, repeat or invent one. No retraining: the same weights, given fewer ways to go wrong.

See alsoRestorerShlyokavitsaCharacter-level

the restorer demo, constrained

Nvidia's GPU compute platform. Every comparison across models or configurations on this site was measured on cloud CUDA hardware, because Apple's own GPU (MPS) gives numbers that can differ from CUDA by more than seed noise - mixing the two would compare hardware, not models.

See alsoSeedParam-matched control

Vary the strength of an intervention and check the effect scales with it. A stronger form of evidence than a single on/off comparison, and a good way to catch a fluke.

Evidence that two functions are genuinely separate: one intervention breaks A but not B, another breaks B but not A. About the strongest claim available short of a mechanism.

See alsoFeature ablationDose-response

A measured "no effect": the change lands inside the baseline’s seed range. Documented here rather than deleted, because a null that took real compute is information.

See alsoSeedAblation

why the nulls stay

A comparison arm with the same parameter count, so a win can be attributed to the mechanism rather than to the extra capacity that came with it.

See alsoCapacity confoundAblation

Comparing two architectures that cannot share a setting, and letting that setting move too. Fast-weight models here use learned positions because RoPE was assumed not to apply to them, so for a long time every fast-weight-versus-softmax comparison in this project changed two things at once. It biased the answer in both directions before it was priced and corrected. The lesson generalizes: when two architectures cannot share a setting, that setting is part of the experiment, not part of the background.

See alsoNoPEParam-matched controlCapacity confound

Writing down what you expect to see, with the number that would count as wrong, before the analysis runs. It is what stops a result being reverse-engineered from the data — and it only does its job when you publish the ones that come back inverted.

See alsoNull resultAblationSeed

Reproducing a known result validates the harness; it is not a finding. Which of the two a number is gets stated here rather than blurred.

See alsoMuonAblation

Measuring the same effect at several model sizes to see whether it grows, shrinks, or flips. It is what separates a real null from an effect that has not switched on yet.

See alsoNull resultPer-Head Muon

a null that switches on

The random starting point for weight init and batch order. Re-running with different seeds gives the noise floor, and a difference smaller than that floor is not a result.

See alsoNull resultValidation loss

Scoring a model on a task it was never trained for and shown no examples of. A base model can still answer multiple choice: put each option after the question and pick whichever it finds most likely. acc_norm is the same thing divided by answer length, so short answers cannot win by default.

See alsoChance baselineNull resultValidation loss

the measurement itself

Running one sentence, keeping a single piece of what the model computed part way through, and replaying a second sentence with that piece substituted in. If the second sentence then gives the first one's answer, the decision was already carried in the piece that moved - which is a claim about where in the network something happens, and one that watching or removing a component cannot make.

See alsoResidual streamAblationTotal effect

patch a minimal pair on the restorer

A head that parks almost all of its attention on the first token, usually because softmax forces every row to sum to 1 and the head has nothing it wants to look at. It is a way of doing nothing, not a computation - but it is the loudest thing in most traces, so anything that ranks heads by attention weight finds sinks first. On the GPT-2 trace here the top eight long-range hops are all sinks.

See alsoInduction headAttention headAttention

see it in GPT-2

Splitting a measured improvement across the features that carry it, to ask whether the gain sits in a few places or is smeared over many. In the molecule model the top 20 features carry only about a third. Not the same as direct logit attribution, which splits a single prediction rather than an improvement.

See alsoFeatureFeature ablationDirect logit attribution

How much of the model’s predictive performance survives when a layer’s activations are replaced by the sparse autoencoder’s reconstruction. The headline health check: a dictionary you cannot substitute back in is not describing the model.

See alsoSAE (sparse autoencoder)FVU

this model’s number

How aligned two vectors are, from -1 (opposite) to 1 (identical direction), ignoring their length. Used here to match the same feature across independently trained SAEs: a masculine-article feature found at cosine 0.84 in one seed and 0.82 in another is the same feature reappearing, not a coincidence.

See alsoFeatureUniversalityd′ (d-prime)

matching a feature across seeds

How cleanly two distributions separate, in units of their own spread. Used here to score whether a feature really distinguishes the thing it appears to.

See alsoFeature

Splitting the score of one thing the model just wrote among the parts that produced it - each attention head, each layer's feed-forward block - so you can see which pushed for it and which pushed against. On this site the split is exact rather than approximate: RMSNorm has no mean-centering, so the pieces add back up to the real score with nothing left over.

See alsoLogit lensAttributionAttention head

see it on the restorer

Splitting a written token's logit into the exact contribution of every attention head, every layer's feed-forward block, and the embedding itself. RMSNorm has no mean-centering, so this decomposition is exact here, not the usual freeze-the-norm approximation.

See alsoLogit lensAttentionAttention head

the restorer's own credit panel

One direction an SAE learns - the hoped-for atom of "what the model knows." You read a feature by looking at the text that fires it hardest, and test it by steering with it.

See alsoSAE (sparse autoencoder)Steering

SAE browser

What the model loses when one of its parts is switched off completely - not only what that part wrote into the answer itself, but everything the later layers went on to do with its output. Direct logit attribution counts the first of those alone, so the gap between the two is the share of a head's job that runs through something else.

See alsoDirect logit attributionAblationAttention head

sweep every head on the restorer

Switching a feature off inside the running model and measuring what breaks, against a control of switching off random ones. The difference between a feature that correlates with a behaviour and one the behaviour needs.

See alsoSteeringAttributionParam-matched control

Fraction of variance unexplained: how much of the original activation the sparse autoencoder failed to reconstruct. Low FVU at high sparsity is what makes the features worth reading.

See alsoSAE (sparse autoencoder)

An attention head implementing "this pattern occurred before, so predict what followed it last time". The best-studied circuit in interpretability; at this scale it is weak, but it sits in the layer theory predicts. Finding one means scoring that specific behaviour: the head with the largest attention weight is usually just an attention sink.

See alsoAttention sinkAttention headAttention

found by scoring, not the loudest head

Reading out what the model would predict from an intermediate layer, to watch a prediction form with depth. Cheap, approximate, and often enough to see where an answer was decided.

See alsoResidual streamLogitsDirect logit attribution (DLA)

Opening a model and asking how a specific answer was produced, using measured internal signals rather than a story after the fact. On this site that means attention, early read-outs, attribution, and interventions on the live model.

See alsoAttentionLogit lensDirect logit attributionAblation

watch a model under glass

A capability that arrives abruptly during training instead of improving smoothly - flat for a long stretch, then a jump, then a plateau. Named after water freezing: the input changes steadily, the output does not. Induction heads are the known example, and on Pythia-160M the jump here is 0.043 to 0.980 in a single bracket, then flat for another 142,000 steps. It matters because the loss curve stays smooth through it, so averaging over training hides the event and comparing final checkpoints cannot see it at all.

See alsoInduction headAttention sink

0.043 to 0.980 in one bracket

A single neuron that responds to many unrelated things at once, so you cannot read meaning off it directly. The problem sparse autoencoders exist to solve.

See alsoSAE (sparse autoencoder)Feature

the untangled version

An input built so that only one thing can explain doing well on it. The induction probe here is random tokens repeated twice: nothing in it is guessable from language, so a head that scores highly is doing the repeat-lookup rather than recalling a common phrase. A probe is an instrument, not a sample of ordinary use - which is why the 0.98 it reports is the mechanism's ceiling, and the same models score 0.022 to 0.027 on natural text.

See alsoInduction headAttention sinkRegistered prediction

the probe, drawn

A probe trained to rewrite a layer’s activations as a sparse combination of many learned directions, in the hope those directions are human-readable. Trained on the model’s activations, not on labels.

See alsoFeatureFVUSteering

browse the features

Clamping or amplifying a feature during generation to see whether it causes the behaviour it correlates with. The difference between "this fires on X" and "this controls X".

See alsoFeatureSAE (sparse autoencoder)

A recorded snapshot of the model’s internals on one input - attention weights, residual stream, logits. Here it comes from the same forward pass that produced the output, so it cannot disagree with it.

See alsoAttentionLogits

a trace, opened up

Whether two models trained identically from different seeds learn the same features. Here it is partial and grows with depth - which bounds how much any single run’s feature numbering means.

See alsoFeatureSeed

measured across seeds

Three DNA bases spelling one amino acid. In the genome model the codon structure has to be learned, while single-base features come free with the architecture - the cross-substrate result in one line.

See alsoCross-substrateFeatureReading frame

language vs DNA

Pointing the same interpretability instrument at models trained on different kinds of data - Bulgarian text, bacterial DNA, GPT-2 - to see which findings are about language and which are about transformers.

See alsoSAE (sparse autoencoder)Trace

language vs DNA

The discriminating metric for molecule generators: how far the set you generated sits from real chemistry, measured in the feature space of a network trained on molecules. Validity and diversity are easy to score well on; FCD is the one that separates a good generator from a plausible-looking one. Lower is better, and 0 means indistinguishable.

See alsoSMILESGuacaMolSELFIES

The fraction of a DNA sequence made of G and C bases rather than A and T. It is a linear function of the raw sequence, so even a randomly-wired, untrained model can decode it - a trained model scoring well on it is not evidence of learning unless it clears that untrained baseline too.

See alsoCodonCross-substrate

the trap it sets without a control

A zero-parameter evolutionary search that mutates and crosses molecular graphs directly. It beats the trained generator on goal-directed GuacaMol - the task is search-bound, not scale-bound - and it is the second optimizer in the interpretability study.

See alsoGuacaMolReinforcement learning (RL)QED (drug-likeness)

The standard benchmark suite for molecule generation, including goal-directed tasks scored against target profiles. Its strong baselines are why the honest answer here is that a genetic algorithm wins.

See alsoGenetic algorithm (Graph GA)

A 0-1 score for how drug-like a molecule is, computed by a chemistry toolkit rather than another neural network. An exact target, which is why molecules can settle questions language cannot.

See alsoReinforcement learning (RL)Genetic algorithm (Graph GA)

Which of the three ways to group DNA into codons you start counting from - shift by one base and every codon downstream regroups differently. A property of where you start reading the text, not something the model computes.

See alsoCodon

shift the frame live

The 4.73M-parameter character model that turns Latin-typed Bulgarian back into Cyrillic. It runs in your browser as WebAssembly, and the page traces the same forward pass that answers you.

See alsoShlyokavitsaWebAssembly (WASM)Character-level

the live lab

An alternative molecule string encoding in which every syntactically valid string is a valid molecule. The correctness guarantee comes from the representation rather than the model.

See alsoSMILES

Bulgarian typed in Latin letters (“shlyokavitsa” itself is an example). Restoring it to Cyrillic is a real task with free ground truth, which is what makes it a good small-model target.

See alsoRestorer

try the restorer

A molecule written as a text string, so a language model can generate one. Nothing stops it writing a chemically invalid string, which is the problem SELFIES solves.

See alsoSELFIES

A portable binary format browsers run at near-native speed. The restorer was ported PyTorch to Go to WASM and checked byte-identical, so the in-page model is the measured one.

See alsoRestorer

running here, now