FlashNorm: two lines of algebra buy 33-35% speedup — and a CUDA race that made the model echo the past
AI Engineer · youtube · 2026-09-20
AI Engineer interview with Filip Makraduli (paper co-written with Nils Graef) on FlashNorm: an RMS norm layer does almost no arithmetic yet can launch 33 times per decode step. The fix folds the norm's gain into projection weights offline, defers the scalar divide so tensor cores and CUDA cores run in parallel, and drops redundant double-normalization (scale invariance) — a 33-35% speedup on norm+projection, compatible with torch.compile and quantized checkpoints. The deferral required hand-written CUDA; an implicit join between streams let post-scale read a stale buffer, making the model repeat outputs with a one-step lag — fixed by explicit stream-end sync. Deploying modified checkpoints required Superlinked's open inference engine, since kernel surgery isn't possible on rented endpoints.
More from Infra
- Meme: we stole fire from the gods just to max out AI compute — bronzeagepapi · 2026-09-20
- SemiAnalysis tells neoclouds to join Nvidia's embargoed CVE program before breaches hit — typewriters · 2026-09-20
- Qwen3.8-27B on a 7900XTX hits 40 tok/s with 240K context for local agentic coding — W61k3r · 2026-09-20
- "There are more inference workloads in Heaven and Earth, Horatio" — a quip on overfit optimization — charles_irl · 2026-09-20
- Kimi subscriptions return after roughly two months, suggesting Moonshot found more compute — ChrisGPT · 2026-09-20
- HN: How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip — petrusenko_max · 2026-09-20