FlashNorm: two lines of algebra buy 33-35% speedup — and a CUDA race that made the model echo the past
AI Engineer · youtube · 2026-09-20
AI Engineer interview with Filip Makraduli (paper co-written with Nils Graef) on FlashNorm: an RMS norm layer does almost no arithmetic yet can launch 33 times per decode step. The fix folds the norm's gain into projection weights offline, defers the scalar divide so tensor cores and CUDA cores run in parallel, and drops redundant double-normalization (scale invariance) — a 33-35% speedup on norm+projection, compatible with torch.compile and quantized checkpoints. The deferral required hand-written CUDA; an implicit join between streams let post-scale read a stale buffer, making the model repeat outputs with a one-step lag — fixed by explicit stream-end sync. Deploying modified checkpoints required Superlinked's open inference engine, since kernel surgery isn't possible on rented endpoints.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04