New NanoGPT Speedrun Record Hits 73.8s via Sophisticated FP8 Optimization
kellerjordan0 · x · 2026-08-10
The NanoGPT Speedrun challenge has a new world record, shaving 0.8s off the previous time to hit 73.8 seconds.
This speedup comes from an incredibly sophisticated FP8 implementation applied to the MLP down-projection forward pass. The 280-line optimization includes several key techniques:
- Integrating the activation quantization step into the prior kernel.
- Implementing delayed amax scaling.
- Fusing weight quantization and transpose operations together.
The backward pass continues to use the original BF16 activations and weights.
More from Infra
- Trillions of Daily Tokens: Inside the Profitable Economics of 'Token Factories' — 智东西 · 2026-08-10
- End of Silicon? CVD Diamond Wafers Promise Massive Compute Leaps — nptacek · 2026-08-10
- The Hidden Bottleneck of Advanced AI Chips: TRUMPF Laser Systems — JosephJacks_ · 2026-08-10
- Benchmarking MiniMax H3 on RTX PRO 6000: 7 Acceleration Workflows Compared — WhoopJack · 2026-08-10
- MiniMax-H3 Low-VRAM Setup: Save 10GB+ VRAM and Eliminate Swapping — Annual_Mess_1839 · 2026-08-10
- Decouple Agents and Models: Run Your Agent on a Raspberry Pi — max_paperclips · 2026-08-10