Glassbox research lab

Small models under glass

Independent experiments on narrow neural networks small enough to run, inspect, and break in the browser. Three are below, the same shipped weights as the study pages: 3.16M to 13.77M parameters. A fifth, the 91.26M flagship, runs live too, at /live-lm-91m.

The point is not a transformer explainer: every arc, bar, and read-out here is measured from one model at one moment, with its controls and corrections kept next to the claim.

Open live labView model cards

one instrument

read straight out of the forward pass that writes the answer — nothing recomputed, nothing precomputed

restorer3.16M parameterslatin letters → кирилицаputs Bulgarian back into its own alphabetruns in your tablanguage model13.77M parametersbulgarian text → more of itwrites Bulgarian, and judges which sentence sounds rightruns in your tabgenome4.72M parametersA C G T → more genomethe same machinery, reading biology instead of languageruns in your tab
Four models trained here from scratch, and the one instrument that reads all of them. Hover a label for more; click a model to watch it work. The parameter counts are arithmetic from the weight files this site actually ships.
run it liveLive labType Bulgarian in Latin letters and watch a 3.16M-parameter model restore Cyrillic in your browser — attention, alternatives, attribution, and ablations, all exposed.Open lab →read the thesisResearch frameThe project asks what 1M–100M-parameter models can do when the task is narrow enough to measure and understand. The house rules cover seeds, controls, hardware, and what each claim does not show.Read the frame →audit the runModel recordEvery shipped model has a card with data, method, and losses. The study pages keep scaling curves, optimizer comparisons, feature traces, and benchmark failures next to the claims they support.See model cards →

What actually moved the needle

The front page keeps one ranking: mechanisms that changed validation loss, each measured against its own baseline. Solid bars isolate the mechanism with a parameter-matched control; the hatched bar includes extra capacity.

This is the project’s research layer: what changed the numbers, how large the effect was, and whether the comparison isolates the mechanism or still mixes in model size.

a decay gate on the memoryFinding 19, 3.4M0.210against the same memory without onethe Muon optimizerFindings 9 and 13, 3.4M, 6 seeds, at the shared 3e-4 AdamW LR (0.035 once retuned)0.187against AdamW, same modelrotations instead of learned positionsFinding 17, 3.4M0.180against one vector per slotthe modern stackFinding 1, 3.4M0.127against the original nanoGPTa gate on the residual streamFinding 16, 3.4M; reaching 0.247 at 32 layers0.114against an ungated residualfour experts instead of oneFinding 5, 6.9M against 3.4M0.065against the dense model it doubles00.21 nats
Validation-loss margin in nats, each against its own baseline at 3.4M parameters. Solid = a parameter-matched control isolates the mechanism; hatched = the margin includes extra capacity.

Each bar names the finding behind it, and every finding is a link: the findings log is on the site in full, corrections included.

The four read-outs, in plain words

Every live model uses the same instrument, so the vocabulary stays consistent across Bulgarian text and DNA.

This is the layer: not a claim that the model is better, but a way to inspect one decision and test which parts helped make it.

  1. attention — where it looked

    Before writing anything, the model scores every earlier symbol — how much do you matter for what comes next? Those scores are the arcs you see, and each row of them adds up to 100%.

  2. logit lens — when the answer forms

    Each layer's working state can be read out early, as if that layer were the last one. You watch a guess turn into a commitment as it moves up through the network.

  3. attribution — what wrote it

    The answer's score is split among the parts that produced it — each attention head, each layer's feed-forward block — so you can see which pushed for it and which pushed against. The pieces add back up to the whole, exactly.

  4. ablation — whether it mattered

    The only one of the four that is not just watching: switch a part off, run the model again, see whether the answer changes. Most parts turn out to be redundant; a few are load-bearing.

Corrections stay visible

Null results and withdrawn claims stay on the same site as the wins. A here means a claim was tested against its own control and did not survive it.

  • Five selective-learning variants lost to the plain baseline. The useful selectivity was in the optimizer, not the loss.

    selective learning →

  • A restorer feature detects the target concept, but switching it off has not changed the output on checked phrases.

    try it yourself in the lab →

  • One Bulgarian reading benchmark was discarded because a randomly initialized network matched every trained model.

    zero-shot benchmarks →

  • A claimed memory gap was withdrawn after a position-signal mismatch explained the margin.

    telling it the time →

  • A loudness-based rule named the wrong induction head. Causal checks replaced that ranking.

    when induction appears →

The house rules: multiple for margin claims, a when capacity could explain a win, one device for model comparisons, and traces checked against the untraced forward pass. Read the full protocol.

An independent research project by iksnerd. Every study page ends with what its result does not show. The training numbers come from multi-seed, CUDA-measured runs. About this research · plain-language glossary.