NanoGPT Speedrun hits new 68.0s record with 96-dim QK and packed FP8 attention

kellerjordan0 · x · 2026-09-18

modded-nanogpt claims a new Track 1 world record: 68.0 seconds on 8xH100, a 5.15% same-node speedup (5.8s), delivered via a heavily involved 1k+ line Triton PR from Nvidia engineer Glitchfix.

Key technical changes:

Q/K head width reduced from 128 to 96 while keeping V and the residual stream at 128 dims — the argument is that routing vectors don't need the same width as value vectors, so normalized Q/K can use a smaller geometry without narrowing the content path;
Q, K, and V projections packed into a shared FP8 projection, with new Triton kernels covering layout, normalization, RoPE, and backward work;
QK Norm, RoPE, KeyOffset, and paired head layout fused into a single Triton kernel; MLP backward and QKV forward/backward moved to FP8 across 4 different FP8 scaling protocols, plus register-aware epilogue placement.

Keller Jordan flags two takeaways: if sub-128 QK dims win at nano scale, maybe >128 wins at Hero scale — and this Nvidia engineer is "super legit."

Original post →

NanoGPT Speedrun hits new 68.0s record with 96-dim QK and packed FP8 attention

More from Infra