New attention kernels meet real models: where interpretability overlaps systems research
QuentinAnthon15 · x · 2026-11-06
Announcing new attention kernel work, the author credits co-authors: Ganesh Bikshandi for BF16 and HipKittens groundwork, and Praneeth Medepalli and Stephen Ebert for foundational quantization work that supplied the building blocks for understanding the numerics. Many complexities emerged when testing the kernels on real models and input distributions; tracing problematic layers yielded numerous insights, revealing surprising overlap between interpretability and systems research.
More from Infra
- Polymarket poll: 56% of voters say AI and data centers shaped their midterm vote — KyeGomezB · 2026-11-06
- MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows — QuentinAnthon15 · 2026-11-06
- Texas data center bill would let all permitted projects finish, just with stricter consumer rules — VraserX · 2026-11-06
- GitHub rebuilds Git storage for thousands of concurrent AI agents — arpit_bhayani · 2026-11-06
- OpenAI built Pando for Codex: 5x lower Git push latency, ~1PB transferred daily — sven_ai · 2026-11-06
- AI bubble debate: liquidity drying up, but are OpenAI and Anthropic really immune? — teortaxesTex · 2026-11-06