Everything we told the model to focus on made it worse

Before testing anything exotic, the modern architecture had to be checked against the plain one it replaced — same data, same budget, three different random starts each, and a result only counts if the runs do not overlap.

Two things worth pausing on. The modern stack beats the original by a wide, clean margin while having fewer parameters — the old baseline spends 4.95M and still loses, so this is architecture rather than size.

And the gated version wins with essentially no extra parameters at all. A gate that is one number per token — 0.01M more parameters than the reference — beats it outright. That is the controlled result the rest of the campaign is built on: the mechanism helps, not the capacity.

That is also what lets the full gate's advantage be split rather than just quoted. The param-matched arm sits between the two, so the distance from the reference up to it is mechanism and the distance from it up to the full gate is capacity:

Without that middle arm the whole thing would have read as “more parameters help” in a costume. With it, more than half the win is available for 0.01M parameters.

Each row changes one thing about where effort goes and nothing else. Higher is worse. These losses are on a rebuilt corpus, so they do not compare with the table above — only within this table.

The interventions span every level you could apply one. Two reweight the loss — by surprise, and by how fast the model is improving, which was supposed to fix what surprise got wrong.

One changes the routing, sending tokens to experts by competence. One strips the learned part out of that routing to isolate sparsity alone. Two sparsify the gradient itself, keeping only the largest updates.

The gradient arms are the cleanest, because they show a dose-response: keep 25% and it is bad, keep 10% and it is much worse. An effect that scales with the dose is hard to explain away as noise.

Five mechanisms, four levels of the system, one direction. Telling the network what to care about consistently made it worse at its job.

The most informative arm is the fourth one. A mixture-of-experts model beats the dense reference, which looks like a win for selectivity — but it also has twice the parameters, so the comparison is worthless as it stands.

Matched on parameters instead, three things were tangled together: the model updates only a quarter of its capacity each step, each token sees a narrower network, and the router learns which expert suits which token.

Separating them: dense beats the parameter-matched sparse model decisively; the learned routing is worth a real 0.052; and freezing the router at random — sparsity with no learning in it — is the worst arm of all.

So the useful part was never the selectivity. It was the specialization the network worked out for itself. Strip that out and keep only the sparsity, and you get the worst result on the page.

Two winners, and neither is a hand-written rule about what matters.

Learned gates. The highway gate above, and the mixture-of-experts router. In both cases the network decides how much to write and where to send things, and nobody supplies the criterion.

A different update rule. Swapping the optimizer for Muon — same architecture, same parameters, only the way updates are applied changes — is the campaign's one clear positive, worth 0.187 at the learning rate everything else here was measured at.

That margin has been corrected twice, in opposite directions, and the second time is the more interesting one. A coarse sweep first understated it: 0.150 became 0.187 once the bracket was widened. Then the other learning rate turned out to be wrong. Muon only replaces the update on the big weight matrices; embeddings, the output head and the norms stay on AdamW, and that side had been sharing a single default with everything else. Tune it and both sides improve — the margin closes to 0.035, with the seed ranges still separated.

So the direction survives and the size is about a fifth of what was published. The campaign's conclusion is unchanged, and it is worth saying plainly: five mechanisms we imposed by hand lost, and the one borrowed win is real but small.

And the honest label: Muon is not ours. It is a published optimizer, replicated here. The campaign's contribution is the controlled comparison and the decomposition above — not the one thing in it that won.

Gates were originally proposed to make deep stacks trainable, and the result above was measured at one shallow depth. So we swept it. Gap is the gated stack minus the plain one; more negative means the gate helps more.

The advantage widens all the way from 4 to 16 layers and then flattens, separated at every depth. Meanwhile plain depth buys nothing here — going from 4 to 32 layers makes the plain stack slightly worse while spending 4.7× the parameters.

The gate converts depth from inert into a real, bounded gain.

The band is the argument. A gate this cheap cannot buy a widening advantage with capacity: it stays at or below 0.08% of parameters at every depth, while the gap it opens more than doubles.

A check worth mentioning: the shallowest plain arm in this sweep is the same configuration as the reference in the earlier table, and it reproduced that published number to four decimals, seed for seed. The sweep is measured on the same footing as everything else here.