NanoGPT Speedrun Record Smashed to 39.9 Seconds, Beating Last 45 Records Combined

ziv_ravid · x · 2026-09-29

Hyperstition's Deven Pietrzak cut the NanoGPT speedrun record from 73.889s to 39.914s (124M model on 8×H100 reaching FineWeb val loss 3.28) — by percentage, a bigger improvement than the previous 45 world records combined.

Three core changes:

The writeup includes an ablation breakdown: sampled softmax contributes 8.41s, larger table 7.06s, ANVIL+averaging 5.09s, FP8 4.95s, smaller model 3.82s, CUDA graphs 2.99s, and more. Code and measurements are open-sourced; frontier-scaling methods are redacted except ANVIL II, whose extensions claim 62% pretraining cost cuts.

Related event: NanoGPT speedrun record slashed to 39.9 seconds(4 posts)→

Original post →

More from Infra

Infra channel →