The findings log

Findings: architecture ablation on Bulgarian

A controlled comparison of the modern (Llama-style) transformer stack against the original nanoGPT baseline and several Schmidhuber-rooted mechanisms, run with ablate.py on a character/BPE GPT trained from scratch on Bulgarian book text.

Setup

• Model: decoder-only GPT, n_embd=192, n_head=6, n_layer=4, block_size=128.
• Tokenizer: byte-level BPE, vocab 8192.
• Data: 15 MB subset of the chitanka book corpus (Ultimate Bulgarian NLP dataset).
• Protocol: 2000 iterations, AdamW lr 3e-4, grad clip 1.0. 3 seeds per config.
• Metric: final validation cross-entropy (lower = better). A gap counts as real only if the seed ranges do not overlap.
• Hardware: single Colab T4.
• Size labels: "15M" and "33M" in the scale studies below are size-class shorthand for models whose real unique-parameter counts are 13.77M and 29.37M (the checkpoint cards and docs/models.md carry the exact figures). The older "33.56M" that appeared in some docs was an arithmetic error — it double-counted the tied embedding, which a state dict stores under two names — and is corrected in docs/bg_gpt.md. Scale trends below are unaffected; only the labels were ever wrong.

Each config changes exactly one axis relative to the modern default (RoPE + RMSNorm + SwiGLU + weight tying); baseline flips all four back to the original nanoGPT (learned pos + LayerNorm + ReLU + untied).

Results

Findings

1The modern stack beats the original nanoGPT baseline (decisive).

modern (4.712) vs baseline (4.838): a 0.127 gap with non-overlapping seeds (~15× the combined std). Notably baseline has more parameters (4.95M vs 3.34M, from the untied head and learned position embeddings), so this is an architecture win, not a capacity win. RoPE + RMSNorm + SwiGLU + weight tying earn their keep here.

2Highway gated residuals beat the modern stack — and the mechanism itself is the cause, not just added parameters.

This is the controlled result. highway_scalar uses a one-value-per-token gate, so it has essentially the same parameter count as modern (3.35M vs 3.34M), yet it beats modern by 0.033 with fully separated seeds (scalar's worst 4.687 < modern's best 4.700). So Highway gating (Srivastava, Greff & Schmidhuber 2015) genuinely helps independent of parameters.

The full per-channel gate (highway, +0.30M params) beats the scalar gate by a further 0.028, also separated. Decomposing highway's total 0.060 advantage over modern:

• ~0.033 (≈55%) — the gating mechanism (param-matched, free)
• ~0.028 (≈45%) — the extra capacity of the per-channel gate

3Neural History Compressor and artificial curiosity: no measurable effect.

hierarchical (4.714) and curiosity (4.717) sit within modern's seed spread (their ranges interleave with modern's), so neither helps nor hurts at this scale. The mid-stack causal history summary and the surprise-weighted auxiliary loss are noise-equivalent to the modern baseline here.

4Learning-progress loss weighting: hurts (2026-07-09 rerun).

A follow-up to the curiosity null. Hypothesis: curiosity failed because raw surprise conflates the learning frontier with unlearnable noise; weighting by the derivative of error instead (Oudeyer & Kaplan 2007; Graves et al. 2017 prediction gain) should fix the sign. --progress weights each token by z-scored learning progress — fast (0.9) minus slow (0.99) EMA of per-target-id loss — with rare ids (<16 batch sightings) held neutral and weights renormalized to mean 1 per batch so the mechanism can't act as a stealth LR change.

Rerun on the rebuilt bg_small corpus/tokenizer (2026-07-06 versions; absolute losses are therefore not comparable to the table above), T4, 2000 iters, 3 seeds, via cloud/ablate_modal.py:

Both reweighting schemes hurt decisively here (~0.06–0.07, fully separated seeds), and progress does not improve on curiosity — the derivative signal did not rescue loss reweighting. This matches the compute-parity skepticism in the literature ("No Train No Gain", Kaddour et al. 2023): SGD's implicit easy-first curriculum is not beaten by an explicit token-level one at this scale. Note that curiosity was noise-equivalent in the original run but harmful on the rebuilt corpus, so these reweighting effects are also corpus-sensitive. Untested variants the literature suggests before closing the loop entirely: hard top-k token selection instead of soft weights (RHO-1, Lin et al. 2024, found hard selection strictly better), and strengths below 0.5.

5Tiny MoE beats dense; progress-biased (competence) routing hurts it (2026-07-09).

The routing-side version of Finding 4's hypothesis. --ffn moe is a Switch-style top-1 MoE FFN (4 full-size SwiGLU experts per block, linear router, 0.01 load-balance aux at train time only); --router-progress 0.5 biases the router toward experts whose routed tokens' loss is falling (fast/slow EMA per expert) — competence routing instead of pure token affinity, at identical parameter count. Same protocol and corpus as Finding 4; modern is that run's number, not re-run (same settings, same hardware — see Conventions).

Two separated results. (a) Progress routing hurts — the param-matched mechanism test: moe_progress trails moe by 0.091 with fully separated seeds (moe's worst 5.135 < moe_progress's best 5.192), and even trails the half-size dense model. A plausible dynamics failure: the bias is token-independent, so it can only shift load between experts — fighting both the affinity signal and the load balancer — and boosting an improving expert feeds it ill-fitting tokens, which raises its loss and flips the bias: an oscillation, not a curriculum. (b) Conditional compute works here: plain moe beats modern by 0.065, separated — but this is capacity (2.1× params at the same per-token FLOPs), not a mechanism claim; a param-matched MoE (split-hidden experts) is untested. Together with Finding 4, both the loss-side and the routing-side versions of "focus learning on what is currently being learned" are falsified at this scale.

6Selective-update decomposition: specialization is real, selectivity alone is not, and dense wins at parameter parity (2026-07-09).

Finding 5's moe win is capacity-confounded, and the literature (Clark et al. 2022's effective-parameter-count law; a 2026 tiny-scale dense-vs-sparse study) predicts dense ≥ MoE at equal total params — but no published work separates the three things a param-matched MoE bundles: selective updates (each step writes ~1/4 of the FFN), smaller active capacity per token, and learned specialization. Three arms decompose it, all at hidden/4 active FFN width: moe_matched (param-matched MoE, learned router), moe_random (same but the router frozen at random init — selective updates without learned specialization; no balance aux, since with a frozen router its only gradient path is through the token representations), and ffn_small (dense at hidden/4 — the active-capacity floor). modern and moe are the Finding 4/5 numbers, not re-run (identical settings).

Three reads, two separated and one null:

• Dense beats param-matched MoE decisively (5.179 vs 5.314, separated): the param-parity question resolves as the scaling-law literature predicts, and the gap (0.135) is larger than the tiny-scale study's "slight" — consistent with its own caveats (4 experts is coarse granularity; matched-total gaps narrow with longer training than 2000 iters).
• Learned specialization is real: moe_matched beats moe_random by 0.052 with separated ranges. The router earns its keep even when experts are small.
• Selective updating alone is a null: moe_random (5.366) does not beat ffn_small (5.347) — ranges overlap, trend if anything negative — despite carrying 4× the FFN parameters in selectively-updated shards. Sparse writes without learned placement buy nothing here; Finding 5's moe win is conditional-compute capacity plus specialization, not write-sparsity per se.

With Findings 4 and 5 this closes a three-front test of "the brain updates selectively, SGD should too" at this scale: selective reading (loss weighting), selective dispatch by learning progress, and selective writing without learned routing all fail; what survives is learned write-gating that SGD itself shapes — highway gates (Finding 2) and the MoE router (here).

7Fine-grained MoE does not flip the parity result — granularity is a null here (2026-07-09).

Krajewski et al. 2024 report that expert-size-mirrors-FFN (our 4-expert Finding 6 setup) is suboptimal granularity and finer experts help at compute parity. Tested at param parity: 16 experts × hidden/16, at top-2 (active hidden/8) and top-4 (active hidden/4, matching moe_matched's active width). modern and moe_matched are the recorded Finding 4/6 numbers, not re-run.

• Granularity is a null: moe_fine_wide (16 experts) vs moe_matched (4 experts), both param-matched and both active hidden/4, are indistinguishable (5.317 vs 5.314, ranges overlap). 16 small experts ≈ 4 big experts at this scale.
• Active-width split is also a null: top-2 (hidden/8) vs top-4 (hidden/4) tie — the param-matched MoE is capacity-starved however the active budget is divided.
• Both still lose to dense by ~0.14 (separated). Finding 6 holds; the literature's granularity benefit is a scale effect that doesn't reach 3.4M / 2000 iters.

8Selective weight updating at the optimizer level (meProp): hurts, dose-dependent (2026-07-09).

The purest selective-update test — dense modern, no architecture or parameter change, just keep the top-k% of each gradient by magnitude per step (Sun et al. 2017, --grad-topk). Finding 6's random router isolated selectivity inside MoE structure; this removes even that.

Monotonic — more sparsity, worse loss, every range separated. meProp's small-task gains (the one positive the literature survey flagged) don't hold for LM pretraining here: Adam's second moment already supplies per-weight update selectivity, and forcing element-wise gradient sparsity only discards signal. This closes the "make SGD update selectively" thesis on its fourth and cleanest front — loss-weighting (4), dispatch routing (5), MoE write-sparsity (6), and now optimizer-level write-sparsity (8) all fail. Imposed selectivity loses; the selectivity SGD already does implicitly wins.

9Muon beats AdamW at parameter parity — the campaign's one positive, and it's a replication (2026-07-09).

After five negative/null selectivity arms, the thing that wins is not one we invented: Muon (Jordan et al. 2024, --optimizer muon, gpt_alpha/optim.py) — SGD-momentum whose 2D-weight update is orthogonalized by a Newton-Schulz iteration (embeddings/head/norms stay on AdamW). Same dense modern architecture, same parameters; only the update rule changes. Run as an LR bracket because Muon needs its own (much larger) learning rate. modern is the recorded Finding 4 AdamW number; the default AdamW code path is provably bit-identical, so it stays valid.

LR under-bracketed here — see Finding 13 (2026-07-19). A finer low-LR sweep found 0.005 → 4.992 (6 seeds), below the 0.01 → 5.029 above. The true 2k-iter optimum is 0.005, so Muon's win over AdamW is 0.187, not 0.150. The direction below is unchanged; the margin is larger.

And re-anchored again by Finding 25 (2026-08-12): 0.187 is the margin at adamw_lr 3e-4 on both sides. Un-starving Muon's own AdamW side (embeddings, head, norms) improves both arms and closes the gap to 0.035. Quote either number with the AdamW LR it was measured at.

• Decisive win at lr 0.01: 5.029 vs 5.179, all seeds separated (Muon's worst 5.035 ≪ AdamW's best 5.175) and unusually tight (std 0.005). At param parity, with no architecture change, orthogonalized updates buy ~0.15 nats.
• U-shaped in LR: 0.02 is exactly break-even with AdamW, 0.04 diverges. The canonical Muon LR (0.02) alone would have read as a null — the bracket is what surfaced the win, and the optimum sits below the default at this scale.
• This is a replication, not a discovery. Muon-beats-AdamW is Muon's established headline result (nanoGPT speedruns; Moonshot "Muon is Scalable", 2025) and it is already used in production training. The value here is local: a validated, architecture-agnostic recipe lever for this project — and, unlike the FFN arms, one with real reason to transfer to the 33M/91M runs. The open, genuinely-novel question is the scale trend: does the 0.15 margin grow or shrink at 15M and 33M (Muon's small-scale behavior is under-characterized). Untested here: --muon-lr needs re-tuning per scale, and Muon runs aren't yet checkpoint-resumable.

Campaign synthesis (Findings 2, 4–9): selectivity SGD does itself wins; selectivity hand-imposed on SGD loses. Learned write-gates (highway, Finding 2; the MoE router, Finding 5) and the one selectivity SGD can't express implicitly (Muon's orthogonalized update, Finding 9) help. Every schedule or mask we imposed on the loss, the routing, or the gradient — progress-weighting (4), competence routing (5), random-routed sparsity (6), fine granularity (7), meProp (8) — failed. The novel contributions are the controlled head-to-head and Finding 6's selectivity/specialization/capacity split; the one usable win (Muon) is borrowed, not built.

10Muon's advantage grows then plateaus with scale — confirmed for the 33M flagship (2026-07-09).

Track A of the flagship program: does Finding 9's Muon win (measured at 3.4M) hold at larger scale, and what is Muon's LR there? Same bg_small / 2000-iter setup at three sizes; AdamW modern measured fresh at 15M/33M (new scales, not re-runs of the recorded 3.4M point), Muon LR bracketed per scale (cloud/ablate_modal.py now forwards --n-embd/--n-head/--n-layer).

Every row above was measured with adamw_lr at its 3e-4 default, which Finding 25 (2026-08-12) shows is worth 0.591 val loss on its own. At 3.4M the re-anchored margin is 0.035, not 0.187, so this trend is quoted at a handicap that has since been removed — and if the shrink is scale-dependent, the shape changes too, not only the magnitudes. The shape is the load-bearing claim, so treat the table as pending re-measurement (backlog Phase 24, ~$5-10 on T4) rather than as a result to cite.

• Muon wins at every scale, all seed ranges separated. The margin grows 3.4M→15M (0.15→0.23) then plateaus 15M→33M (~0.22): the advantage does not wash out with scale — the load-bearing result for adopting Muon in the flagship recipe.
• LR optimum is roughly stable at 0.005–0.01 across a 10× parameter range (0.01 / 0.005 / 0.01), not the monotonic drop first predicted; 0.01 is the safe 33M choice. The bracket earned its keep — a fixed 0.02 understated the win at every scale.
• Caveat: all points are the short-horizon bg_small / 2000-iter regime (heavily undertrained at 33M), so the magnitude of the in-regime flagship margin may differ; the direction — Muon robustly helps through 33M — is what transfers. Muon is confirmed for the flagship; Track B (the data mix) is the remaining lever before the flagship run.

11Under Muon, the highway gate still helps (mechanism); MoE does not get rescued — its "win" and the "composition" were capacity (2026-07-09).

The campaign only ever moved one axis at a time, so the optimizer × architecture interactions were blank. An APOL autoresearch loop (Gemini proposer, fast 600-iter proxy) surfaced two leads — "highway composes under Muon" and "Muon rescues MoE, and the two winners compose" — which were then put through the param-matched controls the leads did not include. Eight arms, matched settings, 3 seeds (3.4M, bg_small BPE, 1500 iters, local — so absolute losses are not comparable to the T4/2000-iter numbers above; only the within-run seed ranges are).

• Highway gating composes with Muon — mechanism, not capacity. The near-zero-param highway-scalar (F, 4.9415) still beats plain-Muon (B, 4.9907), ranges separated. Two independently-validated winners — the learned residual gate (Finding 2) and Muon (Finding 9) — stack. This is the one lead that survives clean.
• "Muon rescues MoE" is false — it was capacity. Unmatched moe (D, 4.9515) beats dense (B) only because it adds experts/params; at parameter parity moe-matched (G, 5.0719) loses to dense under Muon by ~0.08, ranges separated — the same penalty Finding 6 found under AdamW. Muon changes nothing about MoE's param-matched deficit.
• The "composition" is false at param parity. E (highway+moe+muon, 4.8986) looked like a clean additive ladder, but both winners there add parameters. The param-clean version (H, highway-scalar + moe-matched, 5.0330) is worse than highway-scalar alone (F, 4.9415): matched-MoE hurts, so there is nothing to compose with the gate.
• Meta. The loop generated a plausible lead that was ~2/3 capacity artifact, and the param-matched control — the campaign's core method — caught it. Useful as a hypothesis generator; not a substitute for the matched control. Consistent with the campaign thesis (Finding 6): MoE's benefit is conditional capacity, not a param-free mechanism, whatever the optimizer. The surviving flagship implication: highway-scalar + Muon compose; MoE does not earn its parameters even under Muon.

12Muon's win survives in-regime — but only at the right LR/schedule (2026-07-09/10).

Findings 9–10 measured Muon at short horizon (bg_small, 2000 iters). The in-regime test (33M, full mixed corpus, ~40k iters, held-out bpc) first appeared to falsify it: Muon at lr 0.01 + cosine gave 1.975 bpc, worse than the recorded AdamW+cosine 1.934. But that was mistuning, not Muon failing — a WSD-schedule LR bracket found it:

Muon at lr 0.005 beats AdamW in-regime by 0.074 bpc. The lesson: Muon's optimal LR drops with the training horizon (0.01 at 2k iters → 0.005 at 40k). Attribution caveat: at lr 0.01, WSD (2.061) was worse than cosine (1.975), so the win is mainly the lower LR, not the schedule — only 0.005 was tried under WSD, not cosine, so WSD-vs-cosine isn't cleanly isolated. But the recipe decision is clean: Muon lr 0.005 is confirmed in-regime, and ckpt_mwsd005 is the best 33M we have. Harness: --optimizer muon/--muon-lr (gpt_alpha/optim.py), --lr-schedule wsd (train.py); the default AdamW path is bit-identical, so all prior numbers stand.

13Per-Head Muon (Kimi K3) is a null at 3.4M parameter parity — and it exposed that Finding 9 under-bracketed plain Muon's LR (2026-07-19).

(The null is scale-gated: Finding 15 shows it becomes a clean win by 15–33M. Read this as the 3.4M point of that curve.) Motivated by Moonshot's Kimi K3 report (docs/kimi-k3.md): their Per-Head Muon slices each attention projection by head and runs Newton-Schulz on every head-block independently, rather than on the whole matrix. Added as --optimizer muon-perhead (gpt_alpha/optim.py; qkv → 3·H row-blocks, proj → H column-blocks; the head_blocks==1 path is bit-identical to plain Muon, so Finding 9 stays comparable). Same 3.4M / bg_small / 2000-iter / T4 protocol as Findings 9–10.

At matched LR, per-head ties or loses (0.01: 5.071 vs plain 5.029; 0.02: 5.185 vs 5.181; 0.04: 5.551 vs 5.540). Its only edge comes from a lower optimal LR — each small head-block gets its own RMS scale, shrinking the effective step, so the optimum shifts ~2× down (0.005 → 0.0025). Comparing each optimizer at its own LR optimum, both fully LR-bracketed, 6 seeds:

Per-head is lower in the mean (0.014, ~1.7× combined std), but the seed ranges overlap ([4.985, 4.989] region) — at 3 seeds the ranges separated and it looked like a win; a 6-seed confirmation broke the separation. By this repo's significance bar (non-overlapping seed ranges), Per-Head Muon does not beat whole-matrix Muon at this scale. Head-block orthogonalization is a null here; Moonshot's motivation (protecting fine-grained heads in a 3T-param compressed-MLA topology) simply doesn't bind at 3.4M with near-square head-blocks (hs = 32). Documented, not deleted.

The load-bearing result is a byproduct: Finding 9 under-bracketed plain Muon. Its published "optimum 0.01 → 5.029" was a grid artifact; a finer low-LR sweep (0.0025 → 5.048, 0.005 → 4.992, 0.01 → 5.029) puts the true 2k-iter optimum at 0.005. So Muon's win over AdamW (5.179) at 3.4M is 0.187, not the 0.150 recorded in Finding 9 (both at adamw_lr 3e-4; Finding 25 later re-anchors this to 0.035 with that side tuned too) — Muon is stronger than reported, and 0.005 is the better default LR at this horizon (consistent with Finding 12's 0.005-in-regime at 40k). Harness: ablate.py arms muon_ulo/muon_vlo (plain, 0.0025/0.005) and muon_ph_* (per-head bracket); run in parallel via cloud/ablate_modal.py::main_parallel (one container per config×seed).

14MuonClip (Kimi K2) rescues Muon's high-LR blowup but doesn't lower the floor — it's a robustness mechanism, dormant at the tuned optimum (2026-07-20).

Second idea from Moonshot's reports: MuonClip = Muon + QK-Clip. After each step, bound every head's max attention logit to tau by rescaling its Q/K projection rows by sqrt(tau/S_max) (--optimizer muonclip, --qk-clip-tau; qk_clip_ + a gated per-head logit monitor in gpt_alpha/optim.py / attention.py, zero-overhead and bit-identical when off). Motivation: aggressive Muon updates can explode attention logits, an instability AdamW masks; Moonshot used it to train K2 on 15.5T tokens spike-free. Same 3.4M / bg_small / 2000-iter / T4 protocol; tau=100 (their K2 value).

The logits confirm the instability is real at high LR here: on structured data at muon-lr 0.04, the peak qk logit climbs 0.4 → 113 → 382 → 738 over a few hundred iters (measured locally). At muon-lr ≤ 0.01 they stay under 100, so QK-Clip never fires:

Two clean conclusions. (1) QK-Clip works as designed: where logits explode (0.04) it bounds them and rescues 0.242 nats, turning the catastrophic high-LR cliff into a gentle slope; where they don't (≤0.01) it is provably inert, reproducing the plain-Muon seeds to the digit. (2) It does not beat Muon — MuonClip's best (4.994 @ 0.005) ties Muon's best (4.992 @ 0.005), because the optimum sits in the regime where the clip never fires. So MuonClip is a stability/robustness mechanism, not a floor-lowering one: it widens the usable LR band and buys insurance against logit explosion, which matters at Moonshot's scale (spike-free 15.5T-token runs) but is dormant at our tuned 3.4M optimum. Recipe unchanged (Muon lr 0.005); MuonClip stays in the toolbox as LR-robustness insurance, most relevant if logit explosion ever appears at the good LR at larger scale. Together with Finding 13 (Per-Head Muon, null), both K3/K2 Muon variants leave the recipe where Finding 9/12 put it: plain Muon, lr 0.005.

15Per-Head Muon is a scale-gated guardrail: dormant at the bottom of the sub-Chinchilla range, it switches on around 15M / head_size 64 (2026-07-20).

Finding 13 called Per-Head Muon (Kimi K3) a null at 3.4M and predicted the threshold sat at larger head_size. A sweep across four sub-Chinchilla sizes (all below Chinchilla's smallest ~70M model) confirms the prediction and turns the null into a scaling curve. Same bg_small / 2000-iter / T4 protocol; each size runs the full Muon LR bracket {0.0025, 0.005, 0.01} and the Per-Head bracket {0.00125, 0.0025, 0.005}, 3 seeds, best-of-bracket vs best-of-bracket:

The best-of-bracket margin grows monotonically (−0.012 → −0.033 at 3 seeds). Per-Head Muon is a dormant guardrail below ~10M and its advantage strengthens through the sub-Chinchilla regime, not only at frontier MLA scale. The Muon baselines reproduce Finding 10 to the digit (15M 4.624, 33M 4.504), so the comparison is clean across the measurement gap. Per-Head's optimal LR (0.0025 → 0.005) and Muon's (0.005 → 0.01) both drift up with scale.

Confound resolved: head dimension, not capacity, is the axis. At a fixed 13.77M params (n_embd 384, n_layer 6), varying only n_head so head_size ∈ {32, 64, 128} at identical param count (3 seeds, best-of-bracket):

A 13.8M model with 32-wide heads reproduces the tiny-model null; the same model with 64/128-wide heads separates. The threshold sits between hs 32 and 64, then saturates. This matches Finding 13's mechanism: whole-matrix orthogonalization only smears per-head subspaces once head-blocks are large enough to carry them.

Six-seed reality check (strict significance). Adding seeds 3–5 to the top two scale points: 33M holds (margin −0.029, ranges separated by +0.013, every Per-Head seed below every Muon seed); 15M's means stay apart (margin −0.024) but the seed ranges touch (gap −0.0005) — the 3-seed separation did not survive, the same fragility that erased the 3.4M whisker. So by the non-overlapping-ranges bar, Per-Head is a confirmed win at 29M and a means-separated-but-touching effect at 13.8M; the robust claim is the monotonic margin in both scale and head dimension, with strict separation reached near the top of the range.

Six seeds on the head-dimension axis too (2026-08-01): hs 128 holds where hs 64 touches. The six-seed check above covered the two top scale points, and the hs 64 point it hardened is the same configuration as the 15M scale row. The head-dimension sweep's other separated cell, hs 128, had never been six-seeded. It has now, same protocol, seeds 3 to 5 added to the recorded 0 to 2, arms muon_ph_vlo and muon_vlo at 13.77M (n_embd 384, n_head 3, n_layer 6), T4:

Margin −0.0266, ranges separated by +0.0079, every Per-Head seed below every Muon seed. So at a fixed 13.77M parameters the effect touches at head_size 64 and separates at head_size 128: the strict-significance boundary sits on the head-dimension axis, not only on the scale axis, which is what Finding 13's mechanism predicted (whole-matrix orthogonalization only smears per-head subspaces once the head blocks are large enough to carry them). The claim that survives six seeds is therefore stronger than the one recorded above: strict separation is confirmed at the top of both axes, 29M on scale and hs 128 on head dimension, with hs 64 the measured boundary where it fails.

Cost and a process note: this cell cost ~$1 on a T4. A second run at hs 64 cost the same and should not have happened, because FINDINGS.md already carried its six-seed result; the TODO item said "hs 64/128" and was followed without checking which cell was outstanding. It did reproduce the published number to within 0.001, which is a pipeline check nobody asked for. Read the findings cell, not the TODO summary. Draft: docs/papers/subchinchilla_guardrails_draft.md (P15). Harness: cloud/ablate_modal.py::main_parallel (run sequentially — shared app name); arms muon_ulo/vlo/lo, muon_ph_ulo2/ulo/vlo.

16The highway gate's advantage grows with depth — and under a fixed recipe, depth pays for the gated stack but not the plain one (2026-07-26).

Finding 2 measured highway gating at one depth (n_layer=4), which left it untested against its own original motivation: Srivastava, Greff & Schmidhuber proposed gates to make deep stacks trainable. Protocol identical to Finding 4's rerun (192d/6H, block 128, batch 32, 2000 iters, AdamW 3e-4 with warmup 200 + cosine, dropout 0.2, rebuilt bg_small corpus, T4, 3 seeds), varying only depth and the residual mode. highway-scalar is the gated arm so that the gate stays ≤0.08% of params at every depth — a widening gap cannot be gate capacity growing with layer count.

Harness validity check. d4_plain is the modern config, and it returned 5.1793 with seeds 5.175, 5.181, 5.182 — Finding 4's published modern to four decimals, seed for seed. The depth arms are therefore measured on the same footing as the rest of this document.

• The gap widens monotonically with depth (−0.114 → −0.201 → −0.240 → −0.247), separated at all four depths, and saturates between 16 and 32. Finding 2's mechanism claim strengthens with depth rather than being an artifact of the shallow setting it was measured in.
• Depth does not pay for plain residuals here. 4 → 32 layers costs 0.048 val loss upward while parameters grow 4.7× (3.34M → 15.74M). Only the endpoints separate (d32_plain's best 5.209 > d4_plain's worst 5.182); every adjacent step overlaps, so the honest statement is that plain depth buys nothing across this range and the extremes are separated-worse, not that each step degrades.
• Depth pays for the gated stack, up to a point. 4 → 8 → 16 improves 0.064 then 0.038, each separated; 32 then regresses 0.016 (also separated from 16). So the gate converts depth from inert-to-harmful into a real if bounded gain, with the turn between 16 and 32 layers.
• Not the ResNet degradation problem, strictly. Every arm here already has residuals; He et al.'s degradation is about plain feed-forward stacks without them. What this measures is a weaker cousin: with ordinary residuals the depth axis stops paying at this scale, and putting a learned gate on the residual restores it partway.
• The effect is corpus-sensitive. At 4 layers the gap is −0.114 here against −0.033 for the same arms on the pre-rebuild corpus (Finding 2), a 3.4× difference. Finding 4 already recorded curiosity flipping sign across that same rebuild. Gaps in this document travel across corpora in direction more reliably than in magnitude.

The LR confound, stated plainly. All arms share one learning rate, tuned at 4 layers. Deep transformers usually want a lower LR or depth-scaled init, so "plain depth does not pay" is a claim about this fixed recipe, not about plain residual depth in general — some of the plain arms' flatness is plausibly an optimization failure the gate happens to absorb. That ambiguity does not touch the within-depth gaps (both arms get the identical recipe at each depth), which is where the highway claim lives. The follow-up that would settle it is a per-depth LR bracket on the plain arm; until it runs, the cross-depth column is suggestive and the gap column is the result. Summary: ablate_depth.json on gpt-alpha-vol.

Written up 2026-07-31 as the depth half of P11's §4 (docs/papers/selectivity_campaign_draft.md and p11_selectivity.tex). The d4 row also closed that draft's standing editorial question: it had been quoting Finding 2's highway gap from the pre-rebuild corpus for want of an in-regime number, and this sweep supplies one measured against the same modern baseline.

17Fast weights tie softmax at parity; the decay gate is the largest single architecture effect in this document; and the published delta deficit was the positional axis (2026-07-28).

--attn gated-delta (Gated DeltaNet / Kimi Delta Attention, built 2026-07-20) had never been measured. Measuring it required its correct baseline, which did not exist: fast-weight configs carry pos=learned (RoPE was held not to apply), so every previous delta-vs-softmax comparison in this project also swapped the positional encoding. The single-axis pos arm had never been run on the rebuilt corpus. All three arms here, protocol identical to Findings 4/16 (192d/6H, block 128, batch 32, 2000 iters, AdamW 3e-4, T4, 3 seeds); modern is the recorded number.

• Fast weights and softmax are the same model at parity — an exact null. delta 5.3584 vs pos 5.3585: a gap of 0.0001, delta's range straddling pos's. Both 3.37M, both learned positions, so this is matched on parameters and on the positional axis. Whatever DeltaNet's O(d²) recurrent state does at 128-token context, SDPA does equally well and neither is ahead.
• The 0.176 delta deficit reported in fw_beta_FINDINGS.md was a confound. That comparison was delta(learned) against modern(RoPE), and pos prices RoPE alone at 0.180. The qualitative conclusion there survives — a tie is still "the recurrent memory buys nothing SDPA doesn't already have" — but the number and its mechanistic gloss ("the O(d²) state is a bottleneck") do not. Corrected in that document.
• RoPE beats learned positions by 0.180, separated, and with fewer parameters (3.34M vs 3.37M). Finding 1 only ever measured the four-axis baseline flip; this isolates one of them.
• The decay gate wins by 0.206 against its matched baseline (gated_delta vs delta), fully separated, the largest single-mechanism margin in this document: highway 0.060 (Finding 2), modern-over-baseline 0.127 (Finding 1), Muon 0.187 (Findings 9/13, at adamw_lr 3e-4 — Finding 25 re-anchors that one to 0.035, which widens this gap rather than threatening it). It also beats modern by 0.027 while carrying the 0.180 positional handicap.
• Not yet a clean mechanism claim: the gate is +4.5% params (3.52M vs 3.37M). The ratio is much better than highway's per-parameter return (+9% bought 0.028 there), but this document's own rule asks for a param-matched arm, not a favourable ratio. Until it runs, "the decay gate is worth 0.206" carries a capacity asterisk. Resolved by Finding 19 — the control ran the same day and the asterisk comes off.
• Two registered predictions, both falsified. (1) That delta on T4 would land near the 5.088 "CUDA point" quoted in fw_beta and overturn the published number on device grounds: it did not — T4 reproduced the MPS number to within seed noise (5.3584 vs 5.355), so delta is one config where MPS and CUDA agree and that 5.088 figure describes some other setup. (2) That the positional mismatch was a caveat on the gated_delta margin: it was instead the entire published delta effect. Summaries: ablate_{gated_delta_20260728-191948,delta_20260728-193949, pos_20260728-194206}.json on gpt-alpha-vol.

18Positional encoding, three architectures, three different signs — softmax needs it, plain fast weights ignore it, a gated fast-weight arm is hurt by it (2026-07-28).

Finding 17 priced the positional axis at 0.180 and left an obvious question: every fast-weight config in this repo carries pos=learned because RoPE was held not to apply — an assumption, never a measurement. But a decay gate is itself a recency mechanism, so position may be redundant given one. Each architecture measured with and without positional encoding, everything else fixed (192d/6H, block 128, batch 32, 2000 iters, AdamW 3e-4, T4, 3 seeds). pos=none is NoPE (Kazemnejad et al. 2023), already in the codebase from the length-generalization strand.

• Softmax needs position supplied externally. Removing it costs 0.037, separated. Expected, and it anchors the other two rows as a positive control: the measurement can detect a positional effect when one exists.
• Plain fast weights are indifferent. 5.3584 vs 5.3536, ranges overlapping. The recurrence carries enough order information on its own that learned embeddings add nothing.
• The gated arm is actively harmed by position. 5.1487 → 5.1175, separated. Not redundancy — interference. Learned position embeddings and a learned decay gate are both encoding recency, and running both is worse than running the gate alone.
• delta_nope is what makes that attributable to the gate. It was included specifically to stop the result being read as "fast weights don't need position": they don't, but only the gated variant improves without it. Without this arm the claim would have been over-general.
• The best arm in this document at parameter parity is now a fast-weight one. gated_delta_nope (3.35M) beats modern (3.34M, softmax+RoPE) by 0.062, separated, using no positional encoding at all. Caveat on how to quote it: two axes differ (attention mechanism and positional scheme), so it is a configuration result. The single-axis claims are the three rows above, each of which compares an architecture against itself.
• Second caveat, on the softmax row: pos and pos_none are not param-matched — learned position embeddings are ~0.025M (3.37M vs 3.34M). The direction is safe; the magnitude carries an asterisk. The other two rows are matched to within the same 0.025M and in the opposite direction to their effect, so they understate rather than flatter.
• Still not the leader overall. Finding 16's d4_hw (highway-scalar, same protocol) is 5.0656. It sits on a different axis (residual vs attention), which makes the composability question a measurement rather than a guess; it is wired and unrun.
• Registered predictions: four written down before the run. Two confirmed (softmax degrades; the dissociation), one discriminated as intended (delta_nope), and the primary one falsified in the favourable direction — "gated_delta_nope holds within ~0.02" understated an improvement of 0.031 with separated ranges. Summary: ablate_gated_delta_nope+2_20260728-205601.json on gpt-alpha-vol.

19The decay gate is a mechanism, not capacity: a per-head scalar at 1/32 the gate parameters matches the published per-channel form (2026-07-28, recorded 2026-07-31).

Finding 17 left the gate's 0.206 margin under a capacity asterisk and named the arm that would settle it. That arm — gated_delta_scalar, one α per head broadcast across its channels instead of one per channel — ran the same evening and appears in Finding 18's table, but its comparison against the per-channel form was never written down here. It was found missing on 2026-07-31 while re-checking the P18 draft, which had been resting its first contribution on numbers this document did not carry. Same protocol as Findings 17/18; the scalar arm reuses the per-channel kernels unchanged, differing only in the gate's output width.

• The asterisk comes off, and the margin gets slightly larger. The scalar gate beats plain delta by 0.210, fully separated (its worst seed 5.1597 below delta's best 5.3484), while costing 0.0046M — 0.14% of the ungated model against the per-channel form's 4.4%. Whatever the decay gate is doing, it is not the 0.148M parameters the published form spends on it.
• A null against the refinement, not against gating — and the two arms are both published designs. 5.1487 vs 5.1521, ranges almost fully overlapping. Worth stating precisely, because the natural phrasing ("our scalar control beats the published gate") is wrong: Gated DeltaNet's own α is a data-dependent scalar per head, inherited from Mamba2. The channel-wise form is the gated-linear-attention line's, presented by Kimi Delta Attention as its refinement over that coarser gate. So this arm is not an invented control, and the null says the refinement doesn't pay here while independently reproducing GDN's granularity choice. It is also measured on its least favourable terrain: fine-grained forgetting is a long-context feature and 128 tokens is where it should matter least. The 512-token comparison is wired (gd_ctx512/gds_ctx512) and is the falsifier; until it runs this is bounded to short context.
• The analogy from the other gate in this document does not carry. For highway residual gating, the per-channel form buys a real 0.028 over the scalar one (Finding 2). Two gates, same question, opposite answers — which is an argument for measuring the scalar control per gate rather than reasoning by analogy from one to another.
• Summary: ablate_gated_delta_scalar_20260728-203221.json on gpt-alpha-vol. Every parameter count in the table was recomputed from the model code on 2026-07-31 and matches to four decimals.

20The positional axis is about recency, and it appears twice: across memory mechanisms, and inside the choice of signal (2026-07-31).

Finding 18 left an ordering hypothesis. If explicit position is a compensation for memory that does not forget, then its value should fall monotonically as a mechanism carries more recency internally, and the two untested cells should fill in at the ends. Five configurations, same protocol as Findings 17/18; ablate_linear+4_20260731-000242.json.

(a) The mechanism axis. Δ is (loss without position) minus (loss with), so positive means the mechanism wanted the signal. linear accumulates without decay and without error correction, so it should carry the least internal recency of the three fast-weight forms.

• The ordering holds and the significance does not. +0.037 > +0.006 > −0.005 > −0.031 is monotone as predicted, but only the two poles separate. linear 5.3879 ± 0.0025 against linear_nope 5.3943 ± 0.0023 is directionally on the softmax side, and the ranges overlap by 0.0004 (linear_nope's best 5.3911 sits under linear's worst 5.3915). The honest statement is an axis with two significant ends and two indistinguishable middle rungs. linear leans the predicted way without earning the claim on its own, and at 0.006 against a 0.002 spread it would take many more seeds to settle.
• The registered prediction was that linear patterns with softmax rather than with delta, because a sum of outer products cannot tell recent from old. That is what the sign says and what the ranges decline to confirm.

(b) The same axis inside the positional scheme. If recency is what attention is short of, a signal shaped as a recency penalty should beat one shaped as an index. ALiBi and hard-ALiBi (Press et al. 2021; hard windows per Jelassi et al. 2024) had never been measured on LM loss here: the length-generalization strand added them for extrapolation only.

• Recency-shaped position beats index-shaped by 0.136, separated, and on fewer parameters (ALiBi's bias is a fixed buffer; learned embeddings cost 0.0245M). The axis shows itself a second time, one level down from Finding 18.
• Rotary still wins, by 0.043. Relative-but-not-decaying beats decaying, which is consistent with the reading: RoPE supplies order without discarding long-range content, and a distance penalty pays for its recency by giving some of that up.
• hard-ALiBi does not beat plain ALiBi (5.2262 against 5.2226, overlapping). The stronger copy mechanism in the length-generalization setting is not the better language model here. Registered in advance and confirmed.

(c) A control that strengthens Finding 17 rather than qualifying it. gated_delta_none is the per-channel gate with no positional encoding, 3.50M, 5.1054 ± 0.0074 (5.1043, 5.0969, 5.1150). It beats the per-channel gate with learned positions by 0.047, separated, which is the Finding 18 effect reproduced on the other gate parameterization. Against the per-head scalar at NoPE (5.1175 ± 0.0151) it overlaps: the two gate forms tie under both positional settings, so Finding 19's granularity null is not an artifact of having been measured with learned positions in place. It remains bounded to 128-token context, which is the open falsifier.

21Finding 16's LR bracket ran (2026-08-02) — and it hit the exact trap it was registered against, so the depth question is still open, with a bigger surprise sitting underneath it.

The Modal spend limit that blocked this since 2026-07-26 is cleared. Same protocol as Finding 16 (192d/6H, block 128, batch 32, 2000 iters, T4, 3 seeds), n_layer ∈ {4, 32}, residual ∈ {plain, highway-scalar}, LR ∈ {1e-4, 2e-4, 6e-4} — 3e-4 deliberately omitted; it is the recorded modern/Finding-16 baseline, 5.1793 at d4_plain.

Every one of the four series is monotonically improving through the top of the bracket, each step separated (no seed overlap between adjacent LRs in any row). That is precisely the trap the run's own pre-registered verdict rule warned about ("watch for the winner landing at a bracket edge and extend before reading it," Finding 13's lesson) — and it happened on all four series at once, not one. No best-of-bracket reading is valid yet. The depth-vs-optimization question Finding 16 opened is still open: extending upward (1.2e-3, 2.4e-3 are the natural next rungs, same cost class, ~$2) is required before d32_plain can honestly be compared against d4_plain at each one's own optimum.

The bigger and unplanned-for result is what this bracket says about the recipe's own baseline. d4_plain is the modern config nearly everywhere else in this document, published at 5.1793 (lr 3e-4, "tuned at 4 layers" per Finding 16). At lr 6e-4 it reaches 4.7981 — a 0.381 improvement, separated, and still climbing. That is larger than the entire highway-gate effect Finding 16 measured at any depth (0.114 to 0.247). The four known points (1e-4 → 2e-4 → 3e-4 → 6e-4: 5.897 → 5.446 → 5.179 → 4.798) are monotone, so this is not a non-monotonic surprise, just confirmation that 3e-4 sits well short of the optimum for this 2000-iteration, 4-layer config — the same config used as the LR-tuning anchor for every depth in Finding 16, and for most of the modern-vs-baseline comparisons elsewhere in this document.

What this does and does not touch. Within-comparison rankings at matched LR (almost every finding in this document, including Finding 16's own within-depth highway-vs-plain gap) are unaffected — both arms in a comparison share the same LR, so a shared miscalibration cancels in the difference. What is now suspect is any claim that treats 3e-4 as near-optimal for the short-horizon (2000-iter) ablation harness specifically, rather than as a fixed value both arms share. This echoes the project's own repeated lesson that an optimizer's LR optimum moves with horizon (Finding 12's short-horizon Muon mistuning); this is the first sign it may not be Muon-specific. Registering here, not yet resolved: does 3e-4 undertuning explain any of this document's harder-to-place results (e.g. Finding 4's null margins), or does it wash out because those comparisons are also matched-LR? That is a different, larger audit than this bracket and is not attempted here. Summary: ablate_depth_lr.json on gpt-alpha-vol.

The bracket extended (2026-08-02): {1.2e-3, 2.4e-3}, same 8-arm shape, and the cross-depth verdict survives — for the right reason, not the naive one. Full picture, all five rungs:

First, the robustness result, and it's the strongest one here: the highway-vs-plain gap widens with depth at every single one of the five tested learning rates, not only at the original 3e-4 — gap-at-32 exceeds gap-at-4 by 0.078→0.180, 0.108→0.217, 0.114→0.247, 0.073→0.241, 0.069→0.152, 0.054→0.123. Whatever LR miscalibration Finding 21's first half found, it does not touch this comparison, because both arms at a given depth share it.

Applying the pre-registered verdict rule literally is now the wrong question, and here's why. Best-of-bracket d32_plain is 4.659, which does beat d4_plain's published 5.1793 — a literal read of the rule says "confounded, it was optimization." But that rule implicitly assumed 5.1793 was near d4_plain's own optimum, which the first half of this finding already showed is false: d4_plain reaches 4.616 in this very bracket. The honest comparison is best-tuned against best-tuned, not best-tuned against a number now known to be undertuned.

Best-tuned vs best-tuned, both of Finding 16's original claims hold, separated:

• Plain depth still does not pay. d4_plain best (4.616, at 2.4e-3) beats d32_plain best (4.659, at 1.2e-3) — separated (d4's worst seed 4.619 below d32's best seed 4.644). Four layers of plain residual, properly tuned, still beats thirty-two.
• Highway-gated depth still pays. d32_hw best (4.507, at 1.2e-3) beats d4_hw best (4.563, at 2.4e-3) — separated (d32's worst seed 4.527 below d4's best seed 4.558). Reversed direction from the plain arm, exactly as Finding 16 originally claimed, now confirmed at each arm's own best LR rather than at a shared possibly-unfair one.

What's still open, honestly. Not every arm has clearly found its peak. d4_plain is still separated-improving at 2.4e-3 (true optimum unlocated — extending further could only push it lower, which would strengthen "depth doesn't pay," not reverse it). d4_hw has essentially flattened between 1.2e-3 and 2.4e-3 (separated by 0.0001 — a plateau in practice). Both depth-32 arms tick worse at 2.4e-3 than 1.2e-3, suggesting a peak near 1.2e-3, but neither is cleanly separated from 2.4e-3 by this project's non-overlapping-seed-range bar (d32_plain overlaps outright; d32_hw misses separation by 0.0005) — directionally a turnover, not a confirmed one. Chasing the exact peaks further is low priority: both qualitative verdicts already hold robustly across five LR rungs, and the open part (exactly where each curve bottoms out) doesn't change either one. Summary: ablate_depth_lr2.json on gpt-alpha-vol.

Caveats

These results are specific to this setup and should not be over-generalized:

• Small scale. ~3.4M-param models, 2000 iterations, 15 MB of text. Mechanisms like RoPE (short context) and DeltaNet (needs long sequences) are expected to matter more at larger scale; a null here is not a null in general.
• Single corpus, single language, single hardware, 3 seeds. Enough to separate the significant effects above, but not a broad claim.
• Depth: measured to 32 layers, and the recipe is not tuned per depth. Findings 1–15 are all n_layer=4; Finding 16 adds the 4/8/16/32 sweep and shows the highway gap widening with depth. But every arm in that sweep shares one 4-layer-tuned learning rate, so its cross-depth column (plain residual depth not paying off) is confounded with optimization and should not be quoted as an architecture result. The within-depth gaps are clean. A per-depth LR bracket is the open follow-up.
• Fast-weight attention (linear/delta) not included in this run — its O(T) Python recurrence was too slow for a 3-seed sweep. An earlier short single-seed pass had it matching softmax within noise. Closed 2026-07-11 after the chunkwise rewrite made the arm cheap: delta trails modern 5.355 ± 0.007 vs 5.179, fully separated — the single-seed impression did not survive. Numbers, protocol, and the write-gate autopsy that motivated the run: findings/interpretability/fw_beta_FINDINGS.md. Superseded 2026-07-28 (Finding 17): that 0.176 gap was the positional axis, not attention. Against its own baseline (pos, same 3.37M, same learned positions) delta ties exactly, and the single-seed "matches softmax within noise" impression was right after all — it was the 3-seed interpretation that was confounded, not the measurement.
• Not tuned per-config. All configs share one learning rate; a per-config LR sweep could shift the margins.

Reproduce

uv run python ablate.py --data bg_small.txt --tokenizer tokenizer.json \
  --configs modern baseline highway highway_scalar hierarchical curiosity \
  --seeds 0 1 2 --max-iters 2000

(Or run Section 10 of colab_train.ipynb on a T4.)

Finding 4's rerun, on a Modal T4 (data upload steps in the wrapper's docstring):

uv run modal run cloud/ablate_modal.py::main --configs "modern curiosity progress" --seeds "0 1 2"

The depth sweep (open, see Limitations). Each cell gets its own container, which matters here because the 32-layer arms cost ~8× a 4-layer one:

uv run modal run cloud/ablate_modal.py::main_parallel \
  --configs "d4_plain d4_hw d8_plain d8_hw d16_plain d16_hw d32_plain d32_hw" \
  --seeds "0 1 2" --max-iters 2000 --out "ablate_depth.json"

(--out is optional; without it the summary auto-names itself per run. Either way the wrapper refuses to overwrite an existing file on the volume unless you pass --force.)

The fast-weight arms (Finding 17). Run as three separate invocations, not one sweep, because pos is the baseline for the other two and had to exist before either could be read:

uv run modal run cloud/ablate_modal.py::main --configs "gated_delta" --seeds "0 1 2"
uv run modal run cloud/ablate_modal.py::main --configs "delta" --seeds "0 1 2"
uv run modal run cloud/ablate_modal.py::main --configs "pos" --seeds "0 1 2"
uv run modal run cloud/ablate_modal.py::main --configs "gated_delta_scalar" --seeds "0 1 2"

The positional dissociation (Finding 18) — one sweep, each architecture against itself:

uv run modal run cloud/ablate_modal.py::main --configs "gated_delta_nope delta_nope pos_none" --seeds "0 1 2"

Finding 16's open follow-up, the depth × LR bracket. Run (2026-08-02), see Finding 21 — the 2026-07-26 attempt died before any cell started (the Modal workspace was over its spend limit); the limit cleared and the six-rung bracket below ran, then extended by two more rungs when the winner landed at the original bracket's edge (Finding 13's exact trap). 3e-4 is absent from the first command on purpose — it is Finding 16's published number:

uv run modal run cloud/ablate_modal.py::main_parallel \
  --configs "d4_plain_lr1e4 d4_plain_lr2e4 d4_plain_lr6e4 d4_hw_lr1e4 d4_hw_lr2e4 d4_hw_lr6e4 \
             d32_plain_lr1e4 d32_plain_lr2e4 d32_plain_lr6e4 d32_hw_lr1e4 d32_hw_lr2e4 d32_hw_lr6e4" \
  --seeds "0 1 2" --max-iters 2000 --out "ablate_depth_lr.json"

uv run modal run cloud/ablate_modal.py::main_parallel \
  --configs "d4_plain_lr1200u4 d4_plain_lr2400u4 d4_hw_lr1200u4 d4_hw_lr2400u4 \
             d32_plain_lr1200u4 d32_plain_lr2400u4 d32_hw_lr1200u4 d32_hw_lr2400u4" \
  --seeds "0 1 2" --max-iters 2000 --out "ablate_depth_lr2.json"

Still open, and named as such in Finding 21: exactly where d4_plain and the two depth-32 arms bottom out past 2.4e-3, which does not change either qualitative verdict already confirmed best-tuned-vs-best-tuned. If the winning LR lands at an edge of a bracket again, extend it in that direction before reading a best-of-bracket number — Finding 13 records that exact trap.

Read it as four within-depth gaps, not as one table sorted by loss: params grow with depth, so cross-depth absolute losses are not param-matched. The claim under test is whether hw − plain widens from d4 to d32.

22The composability matrix ran (2026-08-08): the two gates stack, but most of the second one is redundant — and the stacked arm is the best measured at this protocol.

The question Findings 17-20 left open was whether this project's two best mechanisms compose: highway-scalar (residual axis, Finding 2/16) and gated-delta-scalar with NoPE (attention axis, Finding 20). Both are learned gates, which is why the registered prior was that they would partially cancel — the project's own synthesis is that what wins is learned write-gating, and these may gate the same thing twice. Four arms, 3 seeds, T4, 2000 iters, the same bg_small protocol as the recorded table, so no baseline was re-run.

Recorded arms it reads against: modern 5.1793, d4_hw 5.0656 [5.0565, 5.0821], gated_delta_nope 5.1175 [5.0961, 5.1290], muon_vlo 4.992 [4.985, 4.998].

• The architecture-only cell separates, and only just. gdn_hw's worst seed (5.0538) sits below d4_hw's best (5.0565) — non-overlapping, but with 0.0027 between the ranges. This is the one number in the table that changes meaning depending on whether you compare means ± std or actual seed ranges, so the raw arrays were pulled from the volume rather than inferred.
• Separated is not additive, and the registered prior was right. From modern 5.1793, highway alone buys 0.1137 and the decay gate alone buys 0.0618; additive would predict 5.0038. Actual is 5.0507 — 73% of additive, so roughly three quarters of the decay gate's standalone value disappears once highway is present. The two gates do overlap in what they do.
• Muon composes, as predicted. hw_muon (4.9276) beats plain muon_vlo (4.992), separated. This is the 2000-iter comparison point that did not exist: Finding 11 measured highway × Muon at 1500 iters, so its absolute numbers could not be quoted against this table.
• The full stack is the best arm measured here. gdn_hw_muon 4.8806 beats hw_muon 4.9276 with non-overlapping ranges, so the decay gate does earn a place on top of highway + Muon.

The one cell that should not be believed yet. gdn_muon (4.9934) overlaps muon_vlo (4.992): the decay gate adds nothing on top of Muon alone, while adding 0.047 on top of highway + Muon. That is a three-way interaction resting on 3 seeds at std ~0.016, and it is also exactly the pattern the pre-registration warned about: Muon's lr 0.005 is the optimum measured for the dense softmax architecture (Findings 9/13), and Finding 9's whole lesson was that a flat Muon arm means bracket the LR before believing the null. gdn_muon is the flat arm. Treat that cell as unresolved rather than negative until it is bracketed.

What this does and does not license. It is the first evidence that the locked flagship recipe should take three levers rather than two. It does not license changing the recipe yet: every arm here is 3.4M, 2000 iters, and 128-token context, while the flagship is 30M, multi-epoch and long — and fast-weight arms are precisely the family whose behaviour is expected to move with context length. The long-context arms (gd_ctx512/gds_ctx512/modern_ctx512) are the gate on that decision. Raw JSON ablate_gdn_hw+3_20260808-130609.json on gpt-alpha-vol.

Addendum, same day — the unresolved cell resolved, and it was the LR. The bracket ran (gdn_muon_ulo/gdn_muon_lo, 2 flanking arms x 3 seeds; the 0.005 centre is the recorded arm and was not re-run):

• The tie with plain Muon was an LR artifact, exactly as the pre-registration warned. At 0.0025 the gated arm is [4.9511, 4.9807] against muon_vlo's [4.9850, 4.9980] — separated, a 0.022 win. "The decay gate adds nothing on top of Muon" is withdrawn. This is the second time in this project that the dense-softmax LR alone would have shipped a null a finer sweep reversed; Finding 9 was the first, and it is why the caveat was registered before the run rather than after it.

• The optimum is enclosed — extension ran the same day and the bracket is closed. The first three points were monotone, which meant 4.9696 was a bound rather than an optimum, so the bracket was extended downward rather than quoted (Findings 13 and 21 each paid for that rule; this was the third instance). Two more arms, 3 seeds each:

  lr	0.000625	0.00125	0.0025	0.005	0.01
  val	5.2235 ± 0.0260	5.0542 ± 0.0162	4.9696 ± 0.0132	4.9934 ± 0.0172	5.0857 ± 0.0131

  A clean U with an interior minimum. Both flanks are separated from 0.0025 ([5.0313, 5.0666] and [5.0686, 5.1005] against [4.9511, 4.9807]), so the optimum is genuinely enclosed and 4.9696 is the arm's best. Raw JSON ablate_gdn_muon_ulo+1_20260808-132459.json and ablate_gdn_muon_xlo+1_20260808-135308.json on gpt-alpha-vol.

• The asterisk on the matrix is real but small, and smaller than first stated. Every Muon arm in the matrix ran at 0.005 while the gated-delta minimum sits at 0.0025, so gdn_muon and gdn_hw_muon were mistuned and the matrix's gated margins understate rather than overstate. But 0.0025 and 0.005 are not separated — [4.9511, 4.9807] against [4.9694, 5.0087] overlap — so the mistuning is worth at most ~0.024 and is not itself significant. An earlier version of this addendum said the gated family "demonstrably wants lower" and called 4.8806 a lower bound without that qualification; the enclosing run shows the LR curve is flat between those two points and only punishes you outside them. gdn_hw_muon at 0.0025 is still the companion arm worth running, but it should be expected to move 4.8806 by a hair, not to reorder the table.

23The long-context falsifier ran (2026-08-08): the granularity null survives its own worst terrain, and the fast-weight advantage over softmax does not.

Finding 17's "per-channel decay buys nothing over per-head" was measured at 128 tokens, which is where fine-grained forgetting should matter least — long context is the entire point of a per-channel forget gate. Both docs/NOVELTY.md and the P18 draft bound the claim to short context because of it. Three arms at block 512, 3 seeds, T4. Absolute losses are not comparable to the 128-token table (4x the tokens per step changes the task); only within-512 gaps are readable.

• The registered quantity, per-channel against per-head, still ties. [4.7907, 4.8463] against [4.8173, 4.8623] overlap. The null holds at 4x the context, on the terrain most favourable to the design it nulls, so Finding 19's claim no longer needs its short-context bound — the falsifier was run and it did not falsify. The point estimate does move 0.019 in per-channel's favour where at 128 the two were level, so the honest phrasing is "still a tie, with the direction now weakly favouring per-channel," not "no effect at any length."
• The ordering against softmax reverses, and this is the consequential half. At 128, gated_delta_nope beat modern by 0.062, separated (Finding 20). At 512 modern_ctx512 is nominally ahead of both gated arms. It is not separated from either, so this is a direction, not a result — but it is the opposite direction, and it lands on the axis the flagship cares about.
• Confound, stated because it blocks the stronger reading. modern_ctx512 carries RoPE while both gated arms carry NoPE, so that comparison moves architecture and position at once. At 128 NoPE was the better setting for the gated arm (Finding 20), but there is no reason to assume that survives 4x the context — a decay gate is a recency mechanism, and recency is exactly what gets harder to supply implicitly as the window grows. Resolving it needs gated_delta_scalar at 512 with RoPE, or modern at 512 with NoPE. Until then the softmax lead is uninterpretable as an architecture claim.
• Net effect on the recipe: it stays locked. Finding 22's case for a third lever rested on the 128-token table, and the gated arms' advantage over softmax does not visibly survive to 512. That is not a refutation, because of the confound above and because the gated arms were also LR-undertuned (see the addendum). It is enough to say the matrix alone does not justify changing a recipe whose target is 30M and long. Raw JSON ablate_gd_ctx512+2_20260808-131940.json.

Addendum, same day — the de-confound was attempted as a 2x2 and one cell of it cannot exist. The plan was to complete {softmax, gated} x {RoPE, NoPE} at 512. The gated + RoPE arm came back bit-identical to gated + NoPE — seeds [4.8173, 4.8623, 4.8248] in both, same parameter count. Cause, confirmed in the code rather than inferred: FastWeightAttention takes no positional argument at all and RoPE is implemented only in MultiHeadAttention, so --pos rope on a fast-weight arm is a silent no-op equivalent to NoPE. Learned-absolute positions do act on that path, because they are added at the embedding level in gpt.py — which is why Finding 20's gated positional contrast used learned-vs-none and why every fast-weight entry in CONFIGS sets pos explicitly. The arm cost ~$2 to reproduce a number already on the volume. Two guards were added so it cannot recur: ablate.py now refuses a config combining a fast-weight attn with RoPE before any GPU time is spent, and Block warns at construction (warn, not raise, so a checkpoint carrying that combination stays loadable through rebuild()).

What the run did establish, and it is worth more than the cell that failed:

• Softmax's dependence on position grows sharply with context. Removing it costs 0.394 at 512 against 0.037 at 128 (Finding 20's positive control) — an order of magnitude for a 4x window. Nobody here had measured that, and it is the sharpest available statement of the recency thesis: softmax re-reads an undecaying past, so the further back that past extends, the more it needs an external index.
• It also resizes the confound I claimed above. Finding 23 called the softmax-vs-gated comparison "confounded" while reading differences of 0.02 to 0.04. The confounding variable is worth 0.394 — an order of magnitude larger than the effects being read off it. At matched NoPE the ordering does not merely fail to reverse, it flips hard: gated beats softmax by 0.356, separated.
• But matched-NoPE is not the fair comparison either, and this is the honest bottom line. Softmax with no positional signal is a crippled configuration, not a serious arm. Judged best-tuned against best-tuned — the standard Finding 21 settled on — softmax's best is RoPE at 4.7967 and the gated arm's best available is NoPE at 4.8348, a 0.038 gap whose ranges touch (4.8177 against 4.8173) and so is not separated. Finding 23's original reading therefore survives as "softmax is level with or slightly ahead at 512, best-vs-best", and does not survive as any claim about architecture at matched position.
• Still owed: gds_ctx512_learned (gated-delta-scalar at 512 with learned absolute positions), which is the contrast Finding 20 actually ran at 128 and the only one that makes the 128-vs-512 comparison like-for-like. Until it exists, whether the gated arm's preference for NoPE survives the longer window is unmeasured. Raw JSON ablate_gds_ctx512_rope+1_20260808-135310.json — the filename preserves the mistaken arm name on purpose.

Second addendum, same day — the owed arm ran, and the positional preference inverts with context. Full table at block 512, 3 seeds each:

• Finding 20's positional result is scale-bounded, and the sign flips. At 128 tokens NoPE beat learned by 0.031, separated — "the gated arm is actively harmed by position." At 512 learned beats NoPE by 0.0575, also separated (4.8122 against 4.8173, no overlap). The gated arm wants position at long context and does not want it at short. docs/NOVELTY.md and the P18 draft state the 128-token version without a scale qualifier and must be corrected before either ships.
• It also supersedes this file's own reading from two hours earlier. The first addendum concluded "softmax is level or slightly ahead at 512, best-vs-best." Best-vs-best is now softmax 4.7967 against gated 4.7773 — the gated arm nominally ahead by 0.019, with ranges overlapping heavily, so the honest verdict is a tie, not a softmax lead. The earlier statement was made while the gated arm's best positional setting was unmeasured; it is withdrawn.
• PARAMETER CONFOUND, and it is the reason none of this is quotable yet. Learned positions cost block_size x n_embd = 512 x 192 = 98,304 params, so the winning arm carries +3.0% over gated+NoPE and +3.3% over softmax+RoPE (RoPE is parameter-free). The asymmetry is what matters: at 128 the winner was NoPE, the smaller model, so that result was clean and capacity could not explain it. At 512 the winner is the larger model, so capacity is a live alternative explanation for the whole inversion. Finding 17 set the precedent — its 0.206 decay-gate win carried a +4.5% asterisk and required the param-matched gated-delta-scalar control, which is why that config exists at all. 3.0% is the same territory and needs the same treatment.
• The control that would settle it: a gated + NoPE arm at 512 given ~0.098M extra parameters somewhere positionally inert (a slightly wider FFN via ffn_hidden), so the comparison is capacity-matched. If it closes the 0.0575 gap, the inversion is capacity. If it does not, the inversion is positional and Finding 20 is genuinely scale-bounded. Note ALiBi cannot serve as the parameter-free alternative here: like RoPE it lives in MultiHeadAttention and never reaches the fast-weight path. Raw JSON ablate_gds_ctx512_learned_20260808-164742.json.

Third addendum, same day — the control ran, and the inversion is half capacity and half a tie. gds_ctx512_wide gives the NoPE arm the same budget through a wider FFN (ffn_hidden 555 against the SwiGLU default 512, solved by construction: 3,452,400 parameters against the learned arm's 3,451,632, deliberately 768 over so that a control failing to close the gap rules out capacity a fortiori). --ffn-hidden was added to train.py for this; the field already existed in gpt.py because grow_wider mutates it, and simply had no route in from the CLI.

• The 0.0575 splits almost evenly. Capacity alone accounts for 0.0275 (48%): the wider-FFN NoPE arm recovers nearly half the gap without touching position. The residual positional effect is 0.0300 — and it does not separate, [4.8032, 4.8133] against [4.7493, 4.8122].
• So the claim is neither "the sign reverses" nor "NoPE wins at every length". At parameter parity, learned against NoPE at 512 is a tie. The correct statement is that the interference effect is real and separated at 128 tokens, where the winner is also the smaller model, and decays to a tie by 512. That is a milder scale bound than the second addendum claimed, and it supersedes it.
• Underpowered, and it should be said. The learned arm's seed spread is 6x the control's (std 0.0261 against 0.0043; range 0.063 against 0.010), so the overlap is partly a power problem rather than a clean null. A 6-seed repeat of the learned arm is what would turn "does not separate" into "is a tie". Until then, read the residual 0.0300 as unresolved-but-small, not as zero. Raw JSON ablate_gds_ctx512_wide_20260808-173553.json.

24The third lever's go/no-go ran (2026-08-08), and the answer is no: against the incumbent this recipe actually uses, the decay gate's sign flips with context length.

Finding 22 made gdn_hw_muon (4.8806) the best arm ever measured at this protocol, which put a third lever on the table for the flagship recipe. The case against it had narrowed to one objection — wrong regime — and Finding 23 spoke to it only partly, because that run compared the gated arm against plain softmax, when the lever in the recipe is highway. "Does the gate still pay on top of highway at longer context?" had never been asked. Two arms at block 512, 3 seeds, T4, identical protocol to Finding 23 so the two are directly comparable.

• The gate is worth +0.015 at 128 tokens and −0.092 at 512. At 128, gdn_hw (5.0507) beat d4_hw (5.0656). Here the same pair reverses, separated ([4.7575, 4.7736] against [4.8431, 4.8678], no overlap), and the margin against it is 6x the margin it won by at the shorter window. The two arms are param-identical at 3.35M, so capacity explains none of it.
• This is the arm Finding 23 could not supply, and it moves that finding's conclusion from "does not survive" to "inverts". Finding 23's gated arms trailed softmax with touching ranges; measured against highway instead, the ranges separate. The recipe's incumbent is the stronger comparison and it is also the harsher one.
• Verdict: the flagship recipe stays locked at two levers (Muon + highway-scalar). The single axis anyone argued would rescue the third — longer context, since fast-weight arms are the family whose behaviour is expected to change with it — was tested and moved against it.
• The Muon LR bracket for gdn_hw_muon is now enclosed, and it settles an interaction. The companion arm Finding 22's addendum asked for came back the other way: 0.0025 gives 4.9094 [4.9031, 4.9173], worse than 0.005's 4.8806 [4.8615, 4.8993] and separated — so 4.8806 was not understated by mistuning. That left 0.005 on the high edge of its own bracket, the shape Findings 13/21/22 each paid to learn not to quote, so 0.01 ran too: 4.9215 [4.9120, 4.9285]. A clean U, both flanks separated from the centre, so 4.8806 is an enclosed optimum rather than a bound. Note what this says about the arms, not just the LR: without highway, gdn_muon preferred 0.0025 to 0.005; adding highway moves the Muon optimum back up.
• Honest limit. All four arms are 3.4M at 2000 iters, and the flagship is 30M and multi-epoch, so this does not prove the gate would lose at 30M — it removes the reason to bet on it. The configs (hw_ctx512, gdn_hw_ctx512, gdn_hw_muon_ulo, gdn_hw_muon_lo) are in ablate.py if a larger-scale rerun is ever wanted. Raw JSON ablate_gdn_hw_ctx512+1_20260808-200644.json, ablate_gdn_hw_muon_ulo_20260808-200642.json, ablate_gdn_hw_muon_lo_20260808-201457.json.

25Muon's AdamW side was starved by a shared default, and fixing it shrinks Muon's margin from 0.187 to 0.035 (2026-08-12).

gpt_alpha/optim.py runs Muon on the 2D weights at --muon-lr and plain AdamW on the embeddings, head and norms at --learning-rate. That second flag defaults to 3e-4 and every Muon number in this repo has used it, including Findings 9-11 and the 29M pair. An APOL autoresearch loop (muon-lr) flagged the pairing; the confirmation ran as a 2x2 factorial rather than re-running the loop's winner, because the loop reports a point in a joint space and not a decomposition. 2000 iters, 3 seeds, T4, bg_small, ablate.py.

• The loop's winner was half right, and that is the methodological point. muon_lr 0.05 is harmful in both columns (+0.55, +0.40) — a clean main effect with no interaction. Because the tuned pairing still beat the default by 0.19, adopting the loop's winner directly would have shipped a harmful muon_lr while correctly crediting the AdamW-side gain. Only the off-diagonal arms separate them.

• Both optimizers were under-tuned, so the comparison had to be re-run on both sides. Re-tuning Muon's AdamW side against a baseline left at its default would repeat the same error with the sign flipped. Bracketed to closure (each arm turns over on both sides):

  pure AdamW	val loss		Muon (AdamW side 3e-2)	val loss
  1e-4	5.8970		0.005	4.6359
  3e-4 (old default)	5.1867		0.01	4.6055
  1e-3	4.6726		0.02	4.5872 [4.576, 4.594]
  3e-3	4.6226 [4.613, 4.629]		0.04	4.8396
  1e-2	5.1859		0.08	5.1738
  3e-2	5.4314

• Re-anchored margin: 0.035 (4.6226 vs 4.5872), ranges still separated (Muon's worst 4.594 < AdamW's best 4.613). Finding 9's direction stands; its size is about a fifth. This is a re-anchor at changed settings, not a retraction — 0.187 remains correct at its own configuration (both arms at 3e-4), the way a bpc figure stays correct at its own --max-chars. Quote either number with the AdamW LR it was measured at.

• Muon's AdamW side wants a different LR from standalone AdamW: 3e-2 for the embeddings/head/norms subset, a setting at which pure AdamW is catastrophic and unstable (5.4314 ± 0.1102). Different parameter subsets, different optima — so the fix is not "raise the default", it is that these are two knobs. Variance blowing up past each optimum (AdamW std 0.007 → 0.045 → 0.110) is why the outer arms had to be measured rather than extrapolated.

• Architecture effects measured under the starved optimizer are inflated, non-uniformly. Re-running the arch arms at the tuned LR: highway-scalar's advantage over plain falls from ~0.25 to 0.054 (still separated), and MoE's apparent capacity win vanishes — 6.89M ties 3.35M dense, where starved it won by 0.064 with separated ranges. At parity MoE still loses (0.041), so Findings 5/6 hold under both settings. The decomposition: 92% of the total gain is the LR, 8% is the architecture.

• Honest limits. 3.4M at 2000 iters only. Finding 10's scale trend (3.4M/15M/33M margins 0.187/0.230/0.220) was not re-anchored here — every point was measured at 3e-4 — so that table should not be quoted at those values. Finding 26 re-anchors all three points. Guarded by apol bench muon-tuned-margin. Raw JSON ablate_modern+7_bg_small_20260812-114908.json, ablate_muon_adamw_3e3+3_bg_small_20260812-120110.json, ablate_arch2_plain+3_bg_small_20260812-121031.json.

26Finding 25's scale trend re-anchors too, and the trend's shape changes, not just its size (2026-08-14).

Finding 25 fixed the 3.4M margin (0.187 → 0.035) by tuning both optimizers' AdamW-side LR rather than leaving it at the shared 3e-4 default. Finding 10's own scale trend (3.4M/15M/33M, 0.187/0.230/0.220) inherited the same handicap at every point and was flagged but not re-measured. It now is, with each optimizer's own optimum re-bracketed per scale, not inherited from 3.4M — the first attempt at this (2026-08-13) stopped mid-run when a low-edge probe at 33M found adamw_lr 3e-3 beating the 3e-2 point the bracket had been built around, which put the whole axis in doubt at every scale. Walking the axis down one rung at a time resolved it differently at each scale rather than confirming a single "under-bracketed" story: 3.4M closes with an interior optimum at 3e-2 (lower is worse, muon_adamw_1e3 4.7702 → muon_adamw_3e3 4.6513 → muon_adamw_1e2 4.5911, approaching the known 3e-2 point 4.5872 from below); 15M is a flat plateau across 1e-3 to 3e-2; 33M has a sharp interior minimum at 1e-3 (muon01_adamw3e4, literally the original starved default, scores 4.4836 — worse than both its neighbours), closing the exact edge that broke the first attempt. ablate.py (muon_adamw_1e3, muon01_adamw3e4), cloud/ablate_modal.py::main_parallel, bg_small, 2000 iters, T4, seed-0 brackets before any 3-seed spend, per the project's own bracket-before-committing discipline.

Three-seed, both sides tuned per scale, every point separated:

• The shape changes, not only the size. The published under-starved trend (0.187/0.230/0.220) rises then plateaus/dips. The re-anchored trend (0.0354/0.0910/0.1782) grows monotonically, roughly doubling at each step (2.6× from 3.4M to 15M, 2.0× from 15M to 33M) — this is the answer to the honest-limits caveat Finding 25 registered ("if the shrink is scale-dependent, the trend's shape could change too"). It did.
• The correction is friendliest exactly where the flagship recipe lives. It shrinks the margin most at the smallest scale tested (0.187 → 0.035, 5.3×) and least at the largest (0.220 → 0.178, 1.2×). The case for Muon at 30M+ is not weakened by this re-anchor — the growth trend is steeper under the fix than it was under the handicap, not milder.
• The AdamW-side optimum genuinely drifts with scale, but it is a bracketed drift, not an open edge. 3e-2 at 3.4M, a plateau spanning 1e-3 to 3e-2 at 15M, a sharp 1e-3 at 33M. The first attempt's fear — "the axis is under-bracketed at every scale and lower values keep winning" — does not hold once each scale is walked down to its own turnover point.
• Honest limits. All three points are still the 2000-iter bracket, not an in-regime run — whether the LR fix itself, not only its magnitude, survives past this horizon is unmeasured (the 29M pair's own convergence run, Phase 24's third bullet, is the closest thing to an answer and is itself unmeasured at a tuned AdamW side). apol bench muon-tuned-margin still guards only the 3.4M point. Raw JSON: p24_scale15m_bracket.json, p24_scale15m_stage2.json, p24_scale33m_bracket.json, phase24_15m_edge2.json, phase24_33m_edge2.json, phase24_33m_edge3.json, phase24_15m_confirm.json, phase24_33m_confirm.json (all on gpt-alpha-vol).

On the site

The campaign (Findings 1-2, 4-9, and the depth sweep at 16) is packaged for a general reader at /selectivity, led by the verdict rather than the wins: five imposed mechanisms, five losses, and the two that worked being the ones the network chose for itself. Finding 15 has its own page at /scaling-laws, and /write-gate is the mechanistic footnote: an audit of what a learned gate actually writes hard (structure, not surprise), which is a candidate explanation for why the error-derived signals in Findings 3/4 kept failing. Both tables on the /selectivity page keep their own baseline, since the pre- and post-rebuild corpora are not comparable.

Findings 17-19 are at /position, built around the three-way dissociation rather than the headline arm: softmax needs the position signal, plain fast weights are indifferent to it, and the decay-gated variant is harmed by it. The page states the middle row as the control that makes the third attributable to the gate rather than to recurrence in general, and labels the best arm a configuration result, since it moves two axes at once. The correction that produced the study (Finding 17's 0.176 deficit was the positional axis) is on the page and is also the site's fifth home-page null.