NanoGPT speedrun record hits 76.3 with a 7% step-count cut from a custom attention tweak

kellerjordan0 · x · 2026-07-27

New NanoGPT speedrun world record set at 76.3 by @Lisennlp, cutting 2.9 seconds off the previous best.

• The run uses a lightweight Dynamically Composable MHA tweak on the final attention layer.
• It reduces step count by 7%.
• The method adds a secondary local window of 112 tokens, recomputes it on the same Q/K, then mixes per-head attention maps into one unified map after softmax.
• That shared map is applied to each head’s values with a per-head scale.
• Every mixing and scaling operation is computed per token from a MUDD MLP, and the whole thing runs in a custom kernel.
• The technique is inspired by the original Dynamically Composable MHA idea.

Original post →

More from Research

• NVIDIA says AdamW hits a scale ceiling as SOAP and Muon beat it on trillion-token runs — omarsar0 · 2026-07-27
• A proposal calls for LLM prompts to replace peer review’s first pass — ChenhaoTan · 2026-07-27
• Study says AI coding gains may depend more on project size than on the model — MelodicStep6956 · 2026-07-27
• A legal-document agent stack uses Google ADK, MCP, and four parsers to cut hallucinations to zero — MaryamMiradi · 2026-07-27
• FreeStyle mines community LoRAs to improve style-content dual-reference image generation — jiqizhixin · 2026-07-27
• ACL 2026 paper proposes two scores to tell when VLM explanations are trustworthy — BrihiJ · 2026-07-27

Research channel →