A training trick that does nothing until the model is big enough

Frontier labs keep adding refinements to how models are trained. We took two of them from Moonshot's Kimi models and ran them on tiny models, from 1.6 million to 29 million parameters, all far below the smallest model in the (about 70M).

The benefit grows with scale

Per-Head Muon changes how the optimizer updates the attention weights. Against plain Muon at matched settings, its advantage (how much lower a loss it reaches) is near zero for the smallest models and grows steadily larger. Nothing switches on abruptly: the margin is a smooth curve.

0.0000.0180.0360.0121.6Mhead 32· overlap0.0143.4Mhead 32· overlap0.02814Mhead 64≈ ranges touch0.03329Mhead 64✓ separatedPer-Head advantage over Muon (how much lower a loss it reaches)
Per-Head advantage over plain Muon by model size (each at its best setting, 3 random seeds). Confidence from up to 6 seeds: overlap at the two smallest, ranges touching at 14M, cleanly separated at 29M.
plain MuonPer-Head Muon
4.504.755.005.251.6M3.4M14M29M
Each model at its best setting. The two curves sit almost on top of each other at the small end and pull apart as the models grow: the gap is the trick switching on.

It is the head size, not the parameter count

The win appears where the model gets bigger, but bigger models also use wider attention heads, so which one matters? We held the parameter count fixed at 13.8M and changed only the head width (by splitting the same attention budget into more or fewer heads).

The benefit tracks head size, not capacity: a 13.8M model with narrow heads reproduces the tiny-model null, and the same 13.8M model with wide heads wins.

0.0000.0180.0360.015head 3212 heads· overlap0.028head 646 heads✓ separated0.030head 1283 heads✓ separatedPer-Head advantage over Muon (how much lower a loss it reaches)
Per-Head advantage at a fixed 13.8M parameters, varying only the attention head width (3 seeds). The margin rises with head size and then saturates; the head-32 point matches the tiny-model null.

This is why the effect is scoped to training tricks and not a general scaling law: a widely cited result says width and depth barely matter for loss at fixed size. Here they do, for this optimizer refinement. Bigger heads carry more distinct information, and Per-Head Muon only helps once each head has enough of its own structure to protect.

What is real and what is not (the honest version)

We judge a result real only when the per-seed loss ranges of the two methods do not overlap. With three seeds the 14M win looked clean; with six it does not quite hold. The means stay apart, but the ranges touch. At 29M the win survives six seeds outright.

14M parametersMuon4.620Per-Head4.596≈ ranges touch (margin 0.024, not separated at 6 seeds)29M parametersMuon4.504Per-Head4.476✓ separated (margin 0.029, gap 0.013)
Six seeds per arm. Bar = min-to-max seed range; ring = mean. The 29M ranges clear each other; at 14M they touch (every Per-Head seed at or below Muon's).

So the load-bearing claim is the smooth, monotonic margin, in scale and in head size, not a crisp threshold. The apparent switch-on is really the smooth benefit crossing the noise floor: a small, fully-controlled example of the argument that emergent thresholds are often smooth trends seen through a significance test.

Why this matters

Small-scale ablations are how most people, without a supercomputer, decide whether a frontier idea is worth adopting. This result is a caution: a training refinement can read as a flat null at small scale simply because the pathology it guards against has not switched on yet.

The switch can sit on an architectural dial (head size) that a parameter-count-only scaling law hides. Full method, numbers, and controls: docs/papers/subchinchilla_guardrails_draft.md.