ZHA achieves subquadratic prefill and decode, converts existing models via distillation
BerenMillidge · x · 2026-11-03
Beren Millidge 介绍其 ZHA（零阶层次注意力/稀疏层次注意力）方法的核心结果：prefill 与 decode 两阶段均达到亚二次复杂度；在 SOTA 评测上以强模型匹配现有稠密注意力与 MSA 的表现；并且可以通过蒸馏直接转换现有模型，无需从头训练。他补充称方法使用了正确的树形数据结构，并认为注意力本质上是稀疏且分层的，因此该方法「不得不行」。
Related event: Zyphra Unveils ZHA, a Tree-Structured Subquadratic Attention(3 posts)→
More from Infra
- Ubuntu's delayed CVE-2026-98432 fix is in -proposed, likely held back by database regression fixes — bytebot · 2026-11-03
- The Circuit: Anthropic's S-1, Intel's questions, Marvell and custom silicon bets — BenBajarin · 2026-11-03
- Analyst: Palantir earnings could hurt the AI trade more than help it, squeezing rivals alongside Nvidia — pdamodaran · 2026-11-03
- Inference providers can't survive without DeepSeek-grade KV-cache engineering, argues thread — AccBalanced · 2026-11-03
- Inference is a craft, not an appliance — many providers won't survive, says founder — AccBalanced · 2026-11-03
- Agent-designed nested Kubernetes horror: kcp partitions on bare metal under KubeVirt under guest clusters — sloppenheimer · 2026-11-03