NanoGPT speedrun record falls to 39.9s, -46% via flop-level skipping tricks
yacinelearning · x · 2026-09-29
modded-nanogpt has a new record: devenpzak (MIT Math) cut training time from 67.6s to 39.9s (-34.0s, -46% on the same hardware); PR #360 was merged and independently reproduced by four sources. The new paradigm: optimize at the individual flop level rather than just matmuls or fancier ops—skip low-value flops entirely. Key tricks include sampled softmax (8s saved, skipping lmhead fwd/bwd for tokens absent from a batch), sparse optimizer steps only for ngram embeddings that occurred in the batch (beta1=0, beta2 applied retroactively), full-stack fp8, a new embedding table, and ANVIL2.
Related event: NanoGPT speedrun record slashed to 39.9 seconds(4 posts)→
More from Research
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- Ropedia Xperience 2 embodied AI dataset lands on Hugging Face under CC BY-NC license — AdinaYakup · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- LatamGPT releases 178.4M-document open Latin American corpus on Hugging Face — OmarUFlorez · 2026-11-16