The benchmark that was measuring nothing
Our Bulgarian models are good at judging how natural a sentence is. Are they any good at questions? We put them through two published Bulgarian multiple-choice tests without any extra training, and the useful result was not a score. It was what happened when we ran the same test on a network that had never been trained at all.
The 14M model on this page is the one you can run in your own browser →.
How you test a model that cannot be asked a question
These are base models: they continue text, they do not follow instructions, so you cannot simply ask them “which answer is right?” But you can do .
Put the question down, then put each candidate answer after it, and measure how unsurprised the model is by each one. Whichever ending it finds most natural is its answer. It is the same machinery as the fluency judging next door, pointed at a quiz.
Two tests, both published and both in Bulgarian. EXAMS is 1,472 real school-exam questions — biology, history, geography — so answering needs knowledge the model would have to have stored.
Belebele is 900 reading-comprehension questions where the passage is given, so the answer is on the page rather than in the weights. Four options each, so .
The control that changed the answer
Before reading any score, look at this. Each pair is the 91M flagship against a network with the same architecture and no training whatsoever — random weights.
the trained model beats its untrained twin by 5.0 points (z = 3.1)
the untrained twin scores the same, so the benchmark is not reading the model
A randomly-wired network is not a blank slate. It still has opinions about how long an answer should be and which words are common, and on a four-way multiple-choice test those opinions alone move the score off 25%. On EXAMS they push it below chance, to 22.8%. On Belebele they push it above, to 27.9%.
Which means the 28.1% our best models score on Belebele is not evidence of anything. An untrained network gets the same. No result from that benchmark is claimed here.
EXAMS — a real signal, and a small one
Accuracy with its margin of error, and whether it clears plain guessing. The last column asks the stricter question: does it beat the untrained network?
| model | accuracy (± std. error) | vs chance (25%) | beats untrained? |
|---|---|---|---|
| 13.77M, one epoch | 25.07 ±1.13 | +0.1σ | no |
| 29.37M mixed | 24.66 ±1.12 | −0.3σ | no |
| 91.26M flagship | 27.85 ±1.17 | +2.4σ | yes |
| untrained, 91M shape | 22.83 ±1.09 | −2.0σ | — |
Only the flagship does. It beats its untrained twin by 5.0 points, which at this sample size is a 3.1-sigma gap — a real effect, not noise. The two smaller models are indistinguishable from an untrained network on this test, and the 14M one is the model running on the live page.
Small, though. 27.85% against 25% guessing is not a leaderboard result. Put next to the fact that this same model is far better at Bulgarian text than the 29M one (1.28 vs 1.93 ), it says something specific: at this scale, getting better at predicting Bulgarian buys fluency, not knowledge.
Those are different things, and only one of them scales into exam answers.
Belebele — where every model ties the untrained one
| model | accuracy (± std. error) | vs chance (25%) | beats untrained? |
|---|---|---|---|
| 13.77M, one epoch | 26.78 ±1.48 | +1.2σ | no |
| 29.37M mixed | 28.11 ±1.50 | +2.1σ | no |
| 91.26M flagship | 28.11 ±1.50 | +2.1σ | no |
| untrained, 91M shape | 27.89 ±1.49 | +1.9σ | — |
Read the rows in order and the trap is obvious. Three trained models sit above chance, two of them by more than two sigma — the shape of a result. The untrained network sits there with them. Whatever lifts scores to about 28% on this test is a property of the answer options and the scoring rule, not of anything a model learned.
Without the last row, this page would have reported a benchmark number for all three models and been wrong. That is the argument for always running the untrained arm, and it is now a house rule rather than a lesson learned once.