NanoGPT Speedrun Record Falls to 67.6s With Canonical Token Masking (-15 Training Steps)
kellerjordan0 · x · 2026-09-19
Jan Varho set a new NanoGPT Speedrun world record of 67.6s (-0.4s) in modded-nanogpt with a simple idea: canonical token masking.
- Multi-token prediction pushes probability mass onto non-canonical tokens: e.g. "eter" can never follow "Hyper" in the tokenized dataset, yet the model learns to predict it
- Explicitly masking impossible continuations improves val loss and prevents the model from generating token pairs never seen in training
- The approach is a local-conditioning variant of arXiv:2506.07956; since masking is eval-only, it's provably better for loss
- Measured savings: 15 training steps; PR #350 has been merged
A tokenization-level observation translates into real training speedup — simple idea, clear payoff.
More from Research
- CoT monitoring isn't an audit log: model explanations barely change when decisions flip — ziv_ravid · 2026-09-19
- CoT may not be faithful: filler tokens add 13 points, models keep reasoning after committing — ziv_ravid · 2026-09-19
- TokenRhythm unveils NeoHorse-1: 4B/9B models claiming first closed-loop step toward RSI — jiqizhixin · 2026-09-19
- Training an LLM to paint by writing code: RL meets creative tasks with a judge model — measure_plan · 2026-09-19
- AI provers on a Bittensor subnet close six decades-old Erdős problems in ten days — markjeffrey · 2026-09-19
- Stanford's Anshul Kundaje: gene expression AI models capped by data, not scale — anshulkundaje · 2026-09-19