New NanoGPT Speedrun Record at 74.6s Achieved via Prefix Token Prediction
surmenok · x · 2026-08-03
Jan Varho set a new NanoGPT Speedrun world record at 74.6 seconds (−0.8s). The core innovation is the introduction of an auxiliary loss on prefix token prediction.
The mechanism works by giving partial credit during early training if the model predicts the longest token-prefix of the target token (e.g., predicting " banana" when the target is " bananas"). This approach can stack with Multi-Token Prediction (MTP), effectively reducing both training steps and overall time.
Related event: NanoGPT Training Speedrun Records Shattered, Hitting 74.6 Seconds(2 posts)→
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16