NanoGPT Speedrun Record Smashed to 39.9 Seconds, Beating Last 45 Records Combined
ziv_ravid · x · 2026-09-29
Hyperstition's Deven Pietrzak cut the NanoGPT speedrun record from 73.889s to 39.914s (124M model on 8×H100 reaching FineWeb val loss 3.28) — by percentage, a bigger improvement than the previous 45 world records combined.
Three core changes:
- ANVIL II: algorithmic optimizer improvement beating a Muon variant on val loss
- Sampled softmax: cuts the vocab scored in forward/backward for most of the run
- Much larger sparse n-gram table: scaled from 640M to 65B entries (2 OOM) for local token patterns
The writeup includes an ablation breakdown: sampled softmax contributes 8.41s, larger table 7.06s, ANVIL+averaging 5.09s, FP8 4.95s, smaller model 3.82s, CUDA graphs 2.99s, and more. Code and measurements are open-sourced; frontier-scaling methods are redacted except ANVIL II, whose extensions claim 62% pretraining cost cuts.
Related event: NanoGPT speedrun record slashed to 39.9 seconds(4 posts)→
More from Infra
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16