Every model trained here

The 91M flagship

768d · 12 layers · 12 heads · block 256 · BPE 8192

Mixed Bulgarian - books, news, forum, wiki, plus a FineWeb-2/FineWiki slice. ~1.83B training tokens (Chinchilla-optimal for this size), 111,400 iterations on a Modal A10G, highway-scalar residuals.

val loss

2.6689

bits per character, external held-out

1.2813

used as a grammar judge

92.3% paired preference, AUC 0.77on 26 real error/correction pairs. Gemini 3.5-flash scores 88.5% / AUC 0.85 on the same set and the confidence intervals overlap, so this is a tie, not a win.

zero-shot EXAMS

27.85%chance 25%. The only model here that beats an untrained net of its own shape (22.83%, z = 3.1). Small in absolute terms: much better at Bulgarian text (1.2813 bpc) than at Bulgarian questions.

against a 2.6B Bulgarian model

wins on judging, loses on compressionhead-to-head with two generations of INSAIT’s BgGPT through identical code, on 2000 identical pairs and one device: fluency preference 0.9305 against Gemma-2-2.6B’s 0.9175 (exact McNemar p = 0.0292, a win, though marginal under a correction for the two comparisons run) and against Gemma-3-4B’s 0.846 (p effectively 0). Bits per character 1.3041 vs 1.1600 and 1.1683 - still a clear loss. 28× and 47× smaller. Both opponents are instruct-tuned and this metric reads raw-text likelihood, which instruction tuning decalibrates; INSAIT’s Gemma-3-27B was not tested.

weightsHugging Face glassbox/gpt-alpha-bg-91m - 365 MB weights-only, optimizer state stripped, verified tensor-identical to the training checkpoint. Public since 2026-08-03.checkpoint ckpt_gpt2bg.pt on the gpt-alpha-vol Modal volume

its concepts, in the SAE browserdownload the weightsckpt_gpt2bg

The best 29M

same shape as mixed-v2, trained with Muon at lr 0.005 on a warmup-stable-decay schedule

The mixed clean corpus, ~40k iterations. The in-regime test of whether the optimizer that wins short ablation runs still wins a full training run — and it is published with its own control, because a single checkpoint is a number and a matched pair is a measurement.

bits per character, external held-out

1.860against 1.9339 for its AdamW twin - same corpus, tokenizer, architecture, batch, dropout and seed, differing only in the optimizer. Muon wins by 0.074 bpc on 1,000 fewer steps. Only at the right learning rate, though: at lr 0.01 the same optimizer loses to AdamW, and at 0.02 it diverges. And it is a comparison at a shared, untuned AdamW learning rate: Muon leaves embeddings, head and norms on AdamW at the same 3e-4 the control uses, and at 3.4M/2000 iters that default was measured to handicap both arms and to inflate Muon's margin fivefold. Unmeasured at 29M to convergence, so neither confirmed nor cleared here.

weightsHugging Face glassbox/gpt-alpha-bg-29m-muon-pair - the two arms in one repo, Muon at the root and the AdamW control under adamw-control/, so the comparison travels together. Staged, still private.checkpoint ckpt_mwsd005.pt on gpt-alpha-vol

the model behind the language half of the DNA studyckpt_mwsd005

29M mixed-v2

512d · 8 layers · 8 heads · block 256 · BPE 8192

The cleaned mixed corpus (books, news, wiki, forum), 672M tokens over 41k iterations on a Modal A10G.

val loss

3.2914Its delta twin scores 3.394, but that pair differs in positional encoding as well as attention; see the comparison page.

bits per character, external held-out

1.9339the control arm for the Muon comparison above, not a strawman: it was this project's best 29M for months, and it ran 1,000 more iterations than the arm that beat it.

weightsHugging Face glassbox/gpt-alpha-bg-29m-muon-pair - the two arms in one repo, Muon at the root and the AdamW control under adamw-control/, so the comparison travels together. Staged, still private.checkpoint ckpt_full33_v2.pt on gpt-alpha-volONNX exported and verified faithful to 1e-5 (export_onnx.py); the onnxruntime-web runner on top of it is not built

an instrumented forward passckpt_full33_v2

29M books-only

512d · 8 layers · 8 heads · block 256 · its own BPE 8192

The same recipe and the same budget as mixed-v2, on a fiction-only corpus. This is the corpus control: the one variable that moves is what the model read.

bits per character, external held-out

2.07650.143 worse than the identical model on the mixed corpus - the diet mattered about 2.7 times more than the recipe change that preceded it. Single seed.

weightscheckpoint ckpt_books33_v2.pt on gpt-alpha-vol

a fiction-trained model reading an encyclopediackpt_books33_v2

29M delta twin

512d · 8 layers · 8 heads · block 256, fast-weight (delta rule) attention instead of softmax, and learned positions instead of RoPE

Same corpus, same tokenizer, same 41k iterations, same residual gating as mixed-v2. Built as an attention-mechanism twin, but it is not a single-variable one: fast-weight configs here carry learned position embeddings because rotary was assumed not to apply to a recurrent state, so this checkpoint differs from mixed-v2 on the positional axis too.

val loss

3.394against 3.291 for its softmax twin. The fast-weight model reads in a visibly different style, and it scores worse here - but the two differ in positional encoding as well as in attention, so the gap is not attributable to the mechanism. On small controlled arms, where both get the same positional scheme, the two tie to 0.0001 and the positional axis alone is worth 0.180.

weightscheckpoint delta_full33_s0.pt on gpt-alpha-vol

the two, side by sidedelta_full33_s0

14M, one epoch

384d · 6 layers · 6 heads · block 256 · BPE 8192

One single pass over the cleaned mixed corpus - 18,300 iterations, ~300M tokens, no example seen twice.

bits per character, external held-out

2.0620better than both 29M books-trained models at under half their size. Single seed, so read it as a strong hint about data diet rather than a measured margin.

zero-shot EXAMS / Belebele

at chance25.1–26.8% against 25% chance on both. Its 91M sibling reaches 27.9% on EXAMS; on Belebele an untrained network of the same shape scores 27.9% too, so nothing here can claim a Belebele result. See bg_zeroshot_FINDINGS.md.

weightscheckpoint ckpt_bg15m.pt on gpt-alpha-volONNX shipped as int8, 17.3 MB, running client-side on this site and public on the Hub as glassbox/gpt-alpha-bg-14m-onnx since 2026-09-06. Quantization costs +0.0018 bpc on the shared held-out (2.0620 to 2.0638) - about a fiftieth of what the corpus mix buys - so quantization-aware training was skipped and the null recorded; the fp32 graph is faithful to PyTorch at 1.7e-5 max logit difference

run it in your browseran instrumented forward passdownload the ONNX graphsckpt_bg15m

The books-era baseline

512d · 8 layers · 8 heads · block 256, plain residuals

The older books corpus, 15k iterations (245M tokens, about three times under Chinchilla). Kept because the two models above are only interpretable against it.

bits per character, external held-out

2.1304

weightscheckpoint ckpt_full_sched.pt on gpt-alpha-vol

ckpt_full_sched

The reproducible 13.8M

384d · 6 layers · 6 heads · BPE 8192

A capped books subset, trained end to end in a free Colab GPU session. Its job is not to be good; its job is that someone else can rebuild it.

weightsnotebook cloud/colab_train.ipynb - rebuilt on demand rather than stored

cloud/colab_train.ipynb

The deployed restorer

256d · 4 layers · 8 heads · block 348 · 60-character vocabulary

Synthetic Latin/Cyrillic pairs generated from a Bulgarian corpus, taught through a self-expanding curriculum over phrase length. Warm-starting it from a BPE language model was tried, and was worse.

port fidelity

byte-for-byte identical to PyTorchon the port's harness phrases, re-checked after the KV-cache rewrite; the cached decode is output-identical to a full recompute across a 40-phrase corpus.

exact match on Wikipedia, plain greedy

0.880 / 0.850 / 0.665 at 3 / 5 / 10 wordsn=200 per length on the shlyokavitsa-pairs test split, CPU reference numerics over frozen weights, 2026-08-03 (uv run restore_eval.py --checkpoint restore_ckpt.pt --per-length 200). Past 10 words it does not degrade, it stops: 0.055 at 15 and 0 for 200 phrases at 20, where the block-348 context and the ~6-word trained frontier run out together. The 4.73M sibling scores 0.955 / 0.915 / 0.810 on the same rows and is still at 0.670 at 20 words.

weightsWebAssembly restorer.wasm, 17 MB with the weights embedded (export_weights.py, then Go), served from this sitecheckpoint restore_ckpt.pt on gpt-alpha-volHugging Face glassbox/gpt-alpha-bg-restorer - both checkpoints, both WASM builds, and generated model.safetensors + config.json for each. Public since 2026-08-02.

type into it, watch it thinkand watch it faildownload the weightsthe same model as a standalone demorestore_ckpt

The 20-word restorer

256d · 6 layers · 8 heads · block 688 · 60-character vocabulary

The same curriculum, more capacity and 60k iterations on a Modal A10G. Built to test whether the earlier model's six-word ceiling was a mechanism or just a budget.

exact match at 20 words, in-corpus

0.920.98 at 5 words, 0.94 at 15, then 0.75 at 24 and 0.23 at 28 - past the trained frontier it decays, exactly as the length-generalization null predicts.

exact match at 20 words, on Wikipedia

0.720the same model on pairs built from Bulgarian Wikipedia, which it never trained on, at 200 phrases per length with constrained decoding: 0.965 at 3 words, 0.820 at 10, 0.720 at 20. Named entities are where the romanization’s irrecoverable collisions live - "sasht" spells both САЩ and същ - so part of the drop is the task, not the model. The in-corpus number above is a held-out-sample number; this one is the distribution-shift number.

versus a 2.6B general model

0.720 against 0.000at 20 words on that Wikipedia split, INSAIT’s BgGPT-Gemma-2-2.6B-IT prompted 5-shot scores 0.100 at three words and 0.000 from five words up. Restoration does not fall out of Bulgarian language modelling at 550× the size.

weightsWebAssembly restore_big.wasm, 21 MB, under its own build tag so both restorers can coexist in the pagecheckpoint restore_big.pt on gpt-alpha-volHugging Face glassbox/gpt-alpha-bg-restorer - shares the repo with the deployed restorer above (restore_big.pt, restore_big.wasm, generated small/model.safetensors + small/config.json). Public since 2026-08-02.

the ceiling that was not onedownload the weightsthe pair set it is scored onrestore_big

Grammar corrector, warm-started

the flagship, fine-tuned on error/correction pairs

Supervised fine-tuning on real (erroneous, correct) Bulgarian pairs, then two attempts to improve on it at the reinforcement-learning and inference-time-search level. Shipped as a CLI with a similarity guard, because its documented failure mode is paraphrasing sentences full of unfamiliar words rather than correcting them.

in-distribution exact match

47.3%

real cases fixed character-perfect

13 of 26against 4 of 26 for the 3.4M from-scratch corrector: base quality dominates this transfer. An earlier run of the same recipe scored 12 of 26 - a one-case difference between runs, worth knowing before quoting either number.

RL and search on top

null twiceGRPO costs 3.3 points in-distribution; best-of-N sampling with perplexity reranking (10 samples, with and without a minimal-edit constraint) leaves the real-case score untouched at 13 of 26. On the cases greedy decoding gets wrong, the sample pool simply never contains the right answer.

weightscheckpoint bg_corrector_91m.sft.pt on gpt-alpha-volCLI correct.py — stdin/--text/--input, --json, and a rewrite guard that returns the input unchanged when the model paraphrases instead of correcting

bg_correct_rl.py

Grammar corrector, from scratch

block 256 · BPE 8192, trained from nothing

The same supervised curriculum as the warm-started corrector, with no pretrained base underneath it.

in-distribution exact match

22.0%

real cases fixed character-perfect

4 of 26

weightscheckpoint trained by experiments/bulgarian/bg_correct_rl.py; not kept on a volume

bg_correct_rl.py

The fluency scorer

the flagship, read as a judge rather than a generator

No extra training: the model's own perplexity is the score. Which correction reads more like Bulgarian is a question a language model already answers.

prefers the correct sentence

92.3% of 26 pairsAUC 0.77. Gemini 3.5-flash: 88.5%, AUC 0.85 - overlapping intervals, so a tie with a frontier model on this narrow task. A 3B general model (qwen2.5) scores 65.4%. The 26 cases are hand-curated for known error types (article misuse etc.), harder by design.

prefers the correct sentence, n=200 random sample

100.0% of 200 pairsAUC 0.85. A different, easier protocol (random real thebogko pairs, not the curated 26) - Gemini 98.0% (AUC 0.94), qwen2.5:3b 69.2% (AUC 0.65). Not a stricter version of the 92.3% result above and does not correct it; both independent judges are near-ceiling on short everyday errors, so the curated 26 remain the number to cite for the Gemini tie.

weightsno weights of its own score.py reads the flagship checkpoint

score.py

Bacterial genome model

256d · 6 layers · 4 heads · block 512 · 4-character vocabulary (A/C/G/T)

12 bacterial genomes, ~54M coding-dense bases, on a Modal A10G. Same train.py as the language models, zero code changes.

codon detectors found by the SAE

10.5σagainst 2.3σ for the same architecture with random weights - the model learned to organise DNA into codons rather than reflecting structure already present in the input.

weightscheckpoint bact_dna.pt on gpt-alpha-vol

an instrumented forward pass over DNAlanguage versus DNAbact_dna

E. coli genome model

128d · 4 layers · block 256 · 4-character vocabulary

The E. coli K-12 genome (NCBI NC_000913.3, strain MG1655), 4,641,652 bases, 2000 iterations. The feasibility model: can this codebase read a genome at all.

bits per base

1.90against 2.0 for random DNA. Scaling this model up raised the untrained control's scores as much as the trained model's, which is how the study found its own probes were confounded.

weightscheckpoint ecoli_dna.pt on gpt-alpha-vol

why its probes needed a controlecoli

SELFIES molecule generator

128d · 4 layers · 4 heads · block 128 · SELFIES vocabulary

~20k ZINC molecules. Because it writes SELFIES rather than SMILES, every sample decodes to a chemically valid molecule by construction - a property of the representation, not something the model learned.

validity

100%by construction, from step 0.

mean drug-likeness (QED)

0.605

after RL toward drug-likeness

0.747a genetic algorithm with no learned parameters at all reaches 0.940 on the same objective.

Celecoxib rediscovery, with replay and a diversity filter

top-1 0.952this reproduces published REINVENT rather than beating it, and Celecoxib is a saturated task where the standard baselines score 1.000. The honest test is the full 20-benchmark suite against Graph GA's 0.899, and that has not been run.

weightscheckpoint smiles_sf_ckpt.pt (prior) and smiles_rl_ckpt.pt (RL) on gpt-alpha-smiles-volmodel card docs/hf_molgen_card.md is written; the weights are not uploaded yet

smiles_sf_ckpt

SELFIES generator, 10M

384d · 6 layers · 6 heads · block 128, same SELFIES setup

The MOSES drug-like set - a different corpus from the 0.8M model, which is why the two are not a clean scale comparison and only the within-model results are quoted.

val loss

0.5620

mean drug-likeness, before and after RL

0.802 → 0.848

features shared by RL and the genetic algorithm

0 of 20and their feature shifts are anti-correlated at −0.40. Two optimizers, the same competence, opposite internal machinery - the effect gets sharper with scale, not weaker.

weightscheckpoint smiles_big_ckpt.pt and smiles_big_rl_ckpt.pt on gpt-alpha-smiles-vol

mol_sae scale-up

SMILES generator, 25M

512d · 8 layers, character-level SMILES

The full MOSES set, ~1.9M molecules, 20k steps on a Modal A10G. Measured on the standard distribution-learning benchmark, including the metric that actually discriminates.

validity / uniqueness

99.53% / 100%

FCD against the MOSES test set

0.888on-distribution, but two to ten times the leaderboard models. Not state of the art, and it memorises about 22% of the training set.

novelty

78.0%

weightscheckpoint smiles_ckpt.pt on gpt-alpha-smiles-vol

moses_bench.py

SMILES generator, combined set

512d · 8 layers, character-level SMILES

A broader 3.4M-molecule set - MOSES plus ChEMBL and ZINC bioactives - benchmarked exactly like its MOSES-only sibling.

validity / novelty

98.70% / 90.17%

FCD against the MOSES test set

2.227worse than the MOSES-only model on this metric, and that is correct behaviour rather than a regression: it models a wider slice of chemistry, so it sits further from one benchmark’s specific distribution while covering more novel chemistry.

weightscheckpoint smiles_combined_ckpt.pt on gpt-alpha-smiles-vol

smiles_combined_ckpt

Log-template classifier

64d · 4 layers · 4 heads · block 192 · 63-character vocabulary

1,400 labelled HDFS log lines across 14 templates, 3 seeds. The model is never shown a template string - only raw lines and their label.

test accuracy

0.9978 ± 0.0016against a regex-per-template ceiling of 1.0000. The ceiling is deliberately optimistic: its patterns were read straight off the ground truth, which is the expensive part in real life.

macro-F1

0.9391 ± 0.0431the spread is three rare templates, where one miss costs a full point.

weightscheckpoint logs/hdfs_cls_seed{0,1,2}.pt on gpt-alpha-vol

train_logs.py

Text-JEPA encoders

192d · 6 heads · 4 layers · block 128, character-level

13 configurations × 3 seeds on Tiny Shakespeare, Modal T4. A self-contained sub-codebase asking whether a predict-in-latent-space objective can replace next-token prediction for text.

probe cross-entropy, pure JEPA

3.19against 2.80 for a randomly initialised encoder and 1.67 for ordinary next-token prediction. The objective produced representations worse than no training at all.

why

the anti-collapse anchor was an accidenta prefix leak in the setup was holding the representations apart; removing it collapsed them. VICReg restores the geometry and still does not restore the outcome.

weightsrun logs per-arm metrics on the text-jepa-vol volume; the encoders themselves are transient

jepa/

The growable tiny GPT

32d, block 32, expanded by function-preserving growth (Net2Net)

A pet-sized model that expands its own network as it is fed text, with two studies behind it: whether function-preserving growth keeps a network plastic under continual learning, and a spiking (leaky integrate-and-fire) brain trained by surrogate gradients as an alternative to the gated recurrent one.

status

pilots, not resultsthe plasticity effect barely appeared at this scale and its baseline is too weak to be a fair comparison; the spiking brain runs but is unbenchmarked. Both were measured on an Apple GPU, so by this project’s own device rule neither could be published even if the numbers had held.

weightsportable format the .pet file the browser toy loads (pet/format.ts); the study arms are transient

pet/

GPT-2 small

768d · 12 layers · 12 heads, open weights from 2019

Nobody here trained this. It is on the site because an instrument that only works on its author's own models is not an instrument.

traced through the same viewer

logits verified identical to an uninstrumented passthe guided tour's induction scorer finds a head unprompted: L5·H5, putting 92% of its attention on the token that followed the previous " book". Corrected 2026-07-31 — the earlier claim here named a different head on the strength of an attention sink; the autopsy is Finding 4 in interp_FINDINGS.md.

weightsHugging Face openai/gpt2, loaded straight from the hub by hf_trace.py

inside GPT-2openai/gpt2