COLM paper on the softmax gradient bottleneck; author briefly crushed the nanogpt speedrun record with a new LM head
yoavartzi · x · 2026-10-06
At COLM, Nathan is presenting a paper on the softmax — or gradient — bottleneck (Thursday, 4:40pm). Yoav Artzi amplifies it, adding that Nathan briefly held the nanogpt speedrun record for a few hours using a new LM head, and invites people to talk to him about it.
More from Research
- LenVM lands COLM 2026 Spotlight: value model predicts remaining generation length for efficient reasoning — xwang_lk · 2026-10-06
- I tested 20+ ways to make a cheap coding model act like an expensive one — KangarooAnxious9394 · 2026-10-06
- 90+ Opus 5.5 Agents Find Two Room-Temperature Magnetic Semiconductor Candidates in 3 Days — scaling01 · 2026-10-06
- TRAAC paper: adaptive thinking budgets fix under- and overthinking in reasoning models — EliasEskin · 2026-10-06
- Illustrated essay 'Mathematical Illustration: Proof, Intuition and AI' by Harriss & Stange — S_Conradi · 2026-10-06
- Study: 'Metacognitive laziness' — generative AI harms learning processes and performance — ArtificialOther · 2026-10-06