Telling a model the time, when it already knows

Ordinary attention re-reads the entire past every time it predicts a word. Nothing fades, and nothing about the machinery says which of those words came first, so order has to arrive from outside.

A fast-weight memory works differently: it keeps a running summary and edits it as it goes, correcting what it already believed rather than piling new material on top. And a decay gate adds one more thing, a learned dial that fades the memory a little before each write, so old content disappears unless something refreshes it.

That last one is worth pausing on. Fading is a way of knowing the time. A memory that decays already treats recent things differently from old ones, without being told what position anything is in. So there is a real question about whether telling it as well is useful, redundant, or worse than useless. No brain has positional embeddings; recency in biological memory comes from traces that fade.

Each architecture compared against itself, with the position signal and then with none at all (NoPE), everything else held fixed. Lower is better, so a line falling to the right means the model was better off without. Three architectures, three directions.

Ordinary attention needs it. Removing the signal costs 0.037, and the three runs separate cleanly from the three with it. That row is the sanity check for the other two: the experiment can detect a positional effect when there is one.

The plain fast-weight memory does not care. 5.3584 against 5.3536, with the runs overlapping. Whatever ordering information a running summary carries, it is enough that the extra signal adds nothing on top.

The memory that forgets is actively harmed by it, at this context length. Taking the position signal away improves it by 0.031, with the runs separating, and removes 25,000 parameters at the same time. Note which way that cuts: the arm that wins is the smaller one, so extra capacity cannot be the explanation.

That third sign does not survive longer context, and we could not fully explain why. At four times the window the comparison reverses on its face, position winning by 0.058. But learned positions cost 512 × 192 = 98,304 parameters there, so the winner is now the larger arm, exactly reversing the direction just noted.

Giving the no-position arm the same budget through a wider feed-forward layer splits the difference almost evenly: 0.0275 of it is capacity, and the remaining 0.0300 does not separate. At matched size the two tie. So the honest statement is that interference is real and separated at 128 words, and decays to a tie by 512 rather than reversing.

One weakness we would rather state than have found: the position arm’s spread across seeds is six times the control’s, so that tie is partly low power.

Two mechanisms were doing the same job, and running both was worse than running the better one alone. That is interference rather than redundancy, and it is the result this page is named for. The middle row is what lets us pin it on the decay gate: fast weights in general shrug the position signal off, but only the forgetting one prefers its absence.

The same question one level down. A position signal can be an index, saying which slot a word sits in, or it can be a recency penalty, saying only that further back counts for less. If the argument above is right, the recency-shaped kind should do better, because recency is what the model is actually short of.

Expressing position as a recency penalty beats expressing it as an index by 0.136, and the two hard-window variants land together well clear of it. So the ordering shows up twice: across the three memories above, and inside the choice of signal here.

Rotations still win, by 0.043 over the recency penalty. They are relative rather than absolute, which is the property that matters, and they carry it without discarding long-range information the way a decaying penalty does. The honest summary is that recency beats indexing, and relative beats both.

The decay gate is the largest single-mechanism margin in the 128-token comparisons: 0.206 against its matched baseline, ahead of highway (0.060), of modern-over-baseline (0.127) and of Muon (0.187). Those four were all measured at the same shared AdamW learning rate; tuning the one Muon leaves on AdamW pulls its 0.187 down to 0.035, which widens this gap rather than closing it. The question is whether that is the mechanism or the parameters it arrives with, so it was built twice, in the two forms the literature already uses: emitting a separate decay value for every channel, and emitting a single value per attention head.

The cheap version costs 4,600 parameters against 148,000, a factor of 32, and the two are indistinguishable. The gain is roughly 0.21 either way, and essentially none of it is the parameters the finer form spends. Both are published designs rather than a design against a homemade control: the per-head gate is what Gated DeltaNet uses, and the per-channel gate is the refinement Kimi Delta Attention makes on top of it. So this is a null against that refinement, not against gating, and at this size it agrees with the older, cheaper choice.

128 words is where fine-grained forgetting should matter least — long context is the whole reason to want it. At four times the window the two forms still tie, on the terrain that most favours the finer one. The point estimate shifts 0.019 toward it, so read this as a tie leaning slightly that way rather than as no effect at any length.

Fast-weight memory looks 0.176 worse than ordinary attention. It is not. The two models were carrying different position signals: learned vectors for the memory, rotations (RoPE) for attention, on the assumption that rotations do not apply to a memory of this kind.

That difference is worth 0.180, which is the entire gap. Give attention the same learned vectors and the two are level: 5.3584 against 5.3585, a difference of 0.0001. The mechanism comparison is a tie at this size, and what looked like an architecture result was a setting.

One variable, three studies, opposite signs. The same mismatch flattered the fast-weight side of a length-generalisation comparison, by handing attention the positional scheme worst suited to that task. A third study, at a different size and on a different language, had the two level all along.

When two architectures cannot share a setting, that setting is part of the experiment, not part of the background.

A decay gate is a rule for forgetting. Give a model 128 words of context and it pays. Give it 512 and it costs — same size, same opponent, opposite sign.

Two gates, on different axes, do not add up. Highway wiring decides how much each layer writes; the decay gate decides how fast the memory fades. They overlap, and the overlap grows with the window: the longer the context, the less there is left for the second gate to do, until it is doing harm. Which mechanism you already have decides whether the next one is worth adding.

Measured at 3.4M parameters over 2000 steps. What a gate does at 30M, over a longer run, is a different question and not one these numbers answer.