The same question, in language and in DNA
Does a neural network organize its internal concepts around the job it does, rather than the raw material it is fed (its “substrate”)? We test this across two totally different models — one that reads Bulgarian, one that reads DNA — using the same concept-finding method (a sparse autoencoder, or SAE), and comparing each model against an untrained (randomly-wired) copy of itself as a control.
Explore the genome model itself: an instrumented forward pass over DNA → (the same viewer as the language models). This page is an exploratory study, not a settled result — read the honest-limits section at the end.
The idea: built-up things are learned, single things are free
The models genuinely learn to represent composed units - things you have to build by combining several inputs: a codon (a 3-letter DNA “word” that codes for one amino acid), or grammatical agreement (which needs earlier words in the sentence).
Atomic units - a single letter, a single DNA base - are not really learned at all: even a randomly-wired model can tell them apart, because that information is already sitting right there in the input.
In DNA — the model builds codon detectors
A 4.72M-parameter model trained letter-by-letter on 54 million bases of bacterial DNA, then opened up with the same concept-finder (SAE) we used on the language models. Its whole vocabulary is four letters — A, C, G and T. Detector strength is measured in standard deviations (σ) - bigger means a cleaner, sharper detector.
The method finds clean, single-meaning (monosemantic) codon detectors. The trained model reaches 10.5σ; a randomly-wired copy of the same model only 2.3σ. So the model genuinely learned to organize DNA into codons - a random network does not, even though it sees the exact same DNA.
Honest limit: these detectors reliably read the codon, but we could not drive it - two attempts to nudge the model's internals (called steering) failed to change its output. Detecting something is not the same as controlling it.
Why the control is the experiment
Before the concept-finder, the same DNA model was measured three cruder ways. Each is shown against the randomly-wired control, and the shape of each pair is the result:
Real, yet only 3σ: the signal is smeared across dimensions. Disentangling it needed the SAE, which lifts the same structure to 10.5σ.
A near-flat pair, and a warning: GC content is a linear function of the input bases, so a randomly-wired model already decodes it. Read without the control, this looks like a finding.
A real enhancement but a modest one; in the trained model lags 3/6/9 are all positive, in the untrained control the lag-3 term flips sign.
Two of the three would have been reported as findings without the control beside them. is the clearest trap: the trained model scores 0.71, which looks like learning until the untrained copy scores 0.58 for free, because GC is a linear function of the input.
The gap, not the value, is the measurement - and it is why the flat pair in the middle is as much a part of this study as the tall one above.
In language — the further back, the more is learned
The 29M Bulgarian model, tested on how well it represents a word at increasing distance back in the sentence. Distance 0 is the current word (atomic - already in the input); further back means the model has to actively carry it forward (composed). The “learned gap” is how much the trained model beats a randomly-wired copy.
The learned gap grows as you go further back (0 → 1 → 2 words): the composed cases show 3–4× more learning than the atomic one, so composition is exactly where the learning happens. It fades again at 3 words back - but both models fade there together, so the model is not “unlearning”; the raw signal itself just gets weaker with distance.
The cross-substrate pattern
The same pattern shows up in two completely different kinds of data, each with its control - one exploratory result at small scale, not a claimed law.
| kind of data | built-up unit (composed) | single unit (atomic) |
|---|---|---|
| DNA | codon — 10σ vs 2.3σ (learned) | base (already in the input) |
| Bulgarian | context — 29σ vs 19σ (learned) | letter — 35σ vs 31σ (free) |
In both, the model learns to represent the built-up (composed) units - codons in DNA, earlier-word context in language - well above what a random model manages. The single (atomic) units it gets essentially for free.