The six-word stall was training length and decoder drift, not memory

The full 4.73M restorer, running client-side in WebAssembly — no server, no network round-trip. Type shlyokavitsa (Latin-keyboard Bulgarian); it restores Cyrillic on the fly.

The obvious guess is that the model runs out of room to hold the earlier text in mind. But it can see about 60 words back - ten times the six it was stuck at - so nothing was being forgotten.

Swapping the two common ways a model keeps track of word order, RoPE and ALiBi, reaches the same length either way, so word-order machinery wasn't the culprit either. Memory and position were never the bottleneck.

A larger model (still tiny - 4.73 million parameters), trained far longer, reaches a frontier of 20 words: full-sentence restoration. How often it gets the whole phrase exactly right, by phrase length - measured on text like the kind it was trained on (the next section measures it on text unlike that):

Within the trained frontier it is a genuinely usable tool. Past it, accuracy decays as expected (matches the length_gen null — no free length generalization). The revised claim: the self-curriculum does climb to full sentences given capacity and iterations; the earlier "doesn't induce full generalization" read conflated an under-trained small model with a mechanism limit.

The numbers above are measured on the same kind of text the model trained on. Scored instead on 210,000 phrases built from Bulgarian Wikipedia - a corpus it never saw:

0.670 instead of 0.92 at twenty words. The reason is specific rather than general weakness: encyclopedia text is full of names, places and abbreviations, and that is exactly where the Latin keyboard destroys information beyond recovery. “sasht” spells both САЩ (the abbreviation for the USA) and същ (“same”); “faram” spells both фаръм and фарам. No model recovers those from the input alone.

Part of that 25-point drop is the task, not the model - and the honest number for arbitrary Bulgarian is the right-hand column.

The failures above are two different things wearing one label. Either the model picked a wrong-but-legal letter — а where ъ belonged, the genuinely hard call — or it lost its place in the input and dropped, repeated or invented characters, writing something the input never allowed at all (на на групата came back as на на на групата).

The second kind is not a judgement call. It is bookkeeping, and it dominated the long end: at 25 words it accounted for 67% of phrases.

Latin “a” can only be а or ъ; “sh” can only be ш or с+х. So the set of letters allowed at each point is knowable before the model runs. Holding the decoder to that set — constrained decoding — leaves every hard call to the model and makes almost all of the bookkeeping errors impossible to express. Same weights, nothing retrained:

Fifteen points at 25 words, for free. Where it still misses, it now misses by a letter instead of derailing — the character error rate at 25 words falls from 0.111 to 0.013, eight times lower. (These are Wikipedia phrases again, 200 per length, scored the same way for both columns.)

The last cell is not quite zero, and that is the policy being honest rather than a bug. The decoder is allowed the spellings people actually type, not only the ones the training data used — so at 25 words 4% of phrases come back as a legalreading of the input that is not the one the reference happens to use. The alignment is intact; the spelling choice is not the reference's.

It also reveals what the length limit actually is. Counting only the genuine decisions in a phrase, the model gets 97.7% of them right, and that rate is flat from 5 to 20 words - there is no length effect left inside the range it was trained on.

A whole phrase is right only when every decision in it is, so a 20-word phrase needs about 21 coin-flips to all land.

Longer phrases are a training question; higher accuracy at a given length is a per-decision question, and those compound: 2.3% wrong per decision to 0.5% would take twenty-word phrases from 0.68 to about 0.90 without touching the model at all.

One catch: the model only ever trained on one way of writing Bulgarian in Latin letters. Real people use several - j for ж or й, q for я, x for х. Faced with a letter it has never seen, the model does not guess. It quietly swallows it: cheshmata e zalqta s voda comes back as чешмата е залта с вода, a word short of a letter.

Widening the allowed-letter lists to cover those spellings costs nothing on the text measured above - identical scores at every length - and on text written the way people type, it takes the rate of losing-your-place from 40-100% of phrases to none of them. The model that has never seen a q now writes чешмата е залята с вода. Try the last two examples in the box at the top.

What it does not fix: toj still comes back as тож rather than той. The constraint narrows the choice to ж or й; choosing between them is knowledge, and the model has never seen that spelling. Constraints supply structure, not knowledge.

Bulgaria has a real Bulgarian language model: INSAIT’s BgGPT, 2.6 billion parameters. Given five worked examples and the same Wikipedia phrases:

It is not a fair fight in either direction - BgGPT was never trained for this and got five examples, while this model does nothing else and had a curriculum built for it. That is the point.

Restoring shlyokavitsa is not a capability that appears when a Bulgarian model gets big enough: it needs a model shaped like the task, reading one character at a time. The same conclusion this project reached from the inside, when word-chunk models six times larger kept failing, now confirmed from the outside.

A 4.73M character model does 20-word restoration at 92% — wherewarm-starting a 29M BPE model (~6× the params, with the missing lexicon provably in its weights) still garbled 2-word phrases (az obicham balgarija → аз обикалгари). BPE shatters noisy Latin input into tokens with no stable relation to the Cyrillic target, so knowledge stored behind the tokenizer can't flow through. Char-level models do character surgery; that is the task, and matching the interface beats ~6× the parameters.

The larger model above is not a video or a call to a server — the whole network runs on your own device, in this tab. Load the page, switch off your internet, and it still works.

Nothing is simplified for the demo: the exported Go build restores the four held-out smoke phrases correctly and passes the same check that its instrumented forward pass is byte-identical to its plain one. The stronger guarantee belongs to the smaller model on the home page: its Go port is verified byte-for-byte identical to the PyTorch original on the port's harness phrases, and its cached decode output-identical to a full recompute across a 40-phrase corpus.