New NanoGPT Speedrun Record at 75.4s via Optimized Triton Kernel
kellerjordan0 · x · 2026-08-02
The NanoGPT Speedrun Track 1 has a new world record at 75.4 seconds (-0.6s), set by congml and Recursive.
The speedup comes from an optimized Triton kernel for the ReLU^2 MLP. During the forward pass of relu(X@W1)^2@W2, the standard approach saves both relu(X@W1) and its square for the backward pass. This PR only saves the squared version and reconstructs the former during the backward pass as a fused epilogue, significantly reducing memory traffic.
Related event: NanoGPT Training Speedrun Records Shattered, Hitting 74.6 Seconds(2 posts)→
More from Research
- AI Conference Peer Review is Broken: Unresponsive Reviewers and Misguided Feedback — RexDouglass · 2026-08-03
- Deep dive into Protein Language Models: The foundation behind Raygun — anshulkundaje · 2026-08-03
- The AI That Builds the Next AI Wins the Race: The Loop Is the Moat — hamostaf04 · 2026-08-03
- NeurIPS 2026 meta-reviews: do they include decisions? Authors confused — CantKillTheLifeless · 2026-08-03
- Google Paper: Safety Tuning to Suppress AI Consciousness Degrades Human-like Values — Promptmethus · 2026-08-03
- Quantizing KV Cache for DeepSeek V4 Flash Significantly Degrades Quality — erazortt · 2026-08-03