Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer
yacineMTB · x · 2026-11-03
Zyphra released a paper introducing Zyphra Hierarchical Attention (ZHA), a sparse attention method that builds a tree of recursively compressed key summaries over the KV cache, traversed top-down by queries. Total cost including routing grows sublinearly in sequence length, yielding overall subquadratic (but superlinear) attention scaling.
Key points:
- Existing fixes either lose expressiveness (sliding-window/linear attention) or only give constant-factor speedups (sparse attention with quadratic indexers)
- ZHA can be cheaply retrofitted to pretrained models via a short KL distillation over a few billion tokens
- Maintains performance on long-context retrieval, reasoning, math, and agentic coding evals, and is robust to very deep trees
Related event: Zyphra Unveils ZHA, a Tree-Structured Subquadratic Attention(3 posts)→
More from Models
- Custom fine-tunes now beat frontier models at ~1/20th the cost, says investor — bindureddy · 2026-11-03
- OpenAI updates Model Spec: intent over literal instruction-following — sven_ai · 2026-11-03
- LiquidAI's LFM3-1.7B tops HF trending with 1k likes and 42.4k downloads in 5 days — maximelabonne · 2026-11-03
- 'Gemini is your friend and alignment is complicated': another eyebrow-raising model reply — vedantmisra · 2026-11-03
- Pedro Domingos: Humans are much more resistant to data than AI is — pmddomingos · 2026-11-03
- OpenAI says search-like queries in ChatGPT are one-fifth of Google's volume — firstadopter · 2026-11-03