New NanoGPT Speedrun Record at 75.4s via Optimized Triton Kernel
kellerjordan0 · x · 2026-08-02
The NanoGPT Speedrun Track 1 has a new world record at 75.4 seconds (-0.6s), set by congml and Recursive.
The speedup comes from an optimized Triton kernel for the ReLU^2 MLP. During the forward pass of relu(X@W1)^2@W2, the standard approach saves both relu(X@W1) and its square for the backward pass. This PR only saves the squared version and reconstructs the former during the backward pass as a fused epilogue, significantly reducing memory traffic.
Related event: NanoGPT Training Speedrun Records Shattered, Hitting 74.6 Seconds(2 posts)→
More from Research
- Tessellation-free geometric queries on displaced surfaces cut caustics variance 10x — ssh4net · 2026-10-16
- MIT's Buehler: AI as 'a new language of science,' three years on — ProfBuehlerMIT · 2026-10-16
- Mere availability of AI advice collapses human "I don't know" from 43% to 3% — ValerioCapraro · 2026-10-16
- One Bias Score, Two Factors: POSTECH Paper Decomposes Option Bias in MC Evaluation — Hyunjong Ok · 2026-10-16
- When the KV Cache Is Slower Than the GPU: Remote KV Tiers Backfire in Production — MultiverseComputingCAI · 2026-10-16
- Artificial Analysis launches Decision Index for the emerging decision model category — ArtificialAnlys · 2026-10-16