One Engineer, $998, and a 3.8B Model That Clears GPT-2 on Every Eval
Hugo Vergnes, an engineer who built this in the evenings, published a complete documented training run of a 3.8B-parameter language model this week. The model, little-lm, scored 0.384 on CORE — materially ahead of GPT-2 (0.2565) and Andrej Karpathy’s nanochat d32 (0.310), which cost roughly the same thousand dollars on H100s.
The benchmark comparison:
| Model | Params | Hardware | Time | Cost | CORE |
|---|---|---|---|---|---|
| GPT-2 (OpenAI, 2019) | 1.5B | — | — | — | 0.256 |
| nanochat d32 | ~1B | 8x H100 | ~33h | ~$1,000 | 0.310 |
| little-lm (1024 ctx) | 3.848B | 8x B200 | 35.9h | $820 | 0.338 |
| little-lm (2048 ctx) | 3.848B | 8x B200 | 43h | $998 | 0.384 |
B200s delivered better value per compute-dollar than H100s for this workload. The architecture is Llama-style: RMSNorm, RoPE, grouped-query attention with 24 query heads and 8 KV heads, relu squared MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings. Those value embeddings account for 721M of the 3.848B parameters — 19% of the model — running as 14 vocabulary-times-kv-dim tables across alternate layers.
What the First Run Got Wrong
Vergnes started with an 858M-parameter Llama trained on FineWeb-Edu, six days on a single A100. Result: PIQA 60.45%. GPT-2 at 124M parameters scores 63%. Six days of compute to build something worse than a 2019 model seven times smaller.
The loss curve explained it. Cosine learning rate decay to zero went completely flat after 70% of training steps — the final 30% of compute produced essentially nothing. Peak LR of 2.5e-4 was too conservative for that scale. AdamW across all parameters when Muon should have been used for the matrix weights.
Five changes between that failure and little-lm:
- Trapezoidal LR schedule. Holds a useful learning rate much later than cosine-to-zero.
- Higher peak LR. Aggressive is fine at sub-billion scale.
- Muon optimizer for matrix parameters. Meaningfully better per-token efficiency at this scale.
- Better data than FineWeb-Edu.
- Scale to 3.8B with value embeddings.
The Infrastructure Argument
Vergnes frames the project partly as a case for engineering discipline in AI training. The final run was fully specified by YAML config: model, dataset, optimizer, schedule, callbacks. No code edits per experiment — components self-register into a global registry and resolve by name. Swapping an optimizer or dataset is a one-line diff.
The argument: good infrastructure pays for itself at the first convergence problem you hit. If you can read a config and know exactly what will happen — no hidden mechanics — you can debug faster and run more experiments per dollar.
The Wider Point
The $1,000 reachability frontier has moved. In 2019, $1,000 of cloud compute was a toy. In 2026, it buys a 3.8B model scoring above GPT-2 on CORE with 43 hours of B200 time. The absolute numbers still sit well below frontier — 0.384 CORE is not 0.8-something — but the trajectory matters.
As Vergnes put it: “As the frontier moves, $1,000 takes you further and further.” The write-up is on his GitHub page. The model is Llama-licensed and available.