Zyphra Introduces Hierarchical Attention, Claiming up to 400x Speedups at 1B-Token Contexts
teortaxesTex · x · 2026-11-03
Zyphra announced Hierarchical Attention (ZHA), its path to true subquadratic attention. The KV cache is recursively compressed into a tree of summaries; at inference each query starts at the root and, scored against learned reference queries, uses adaptive top-p routing to descend toward raw keys — making the search itself sparse while keeping the full KV cache for precise retrieval.
- Projected speedups: 35x over dense attention and 5x over MSA at 10M tokens, rising to 400x and 55x at 1B tokens.
- Unlike fixed-state models, ZHA preserves the full KV cache; unlike indexer-based sparse methods, it avoids residual quadratic costs.
- Conversion is cheap: freeze a trained dense-attention backbone and learn reference queries with only 4B tokens.
- Tested on open models from 4B to 35B with 256K–512K contexts. Author Beren Millidge called solving subquadratic attention a multi-year goal finally realized.
More from Research
- Stanford and NUS hearables amplify hidden sound textures in real time at UIST 2026 — HanGuo97 · 2026-11-03
- Three LLM agents share one Godot escape room: logging what agents say vs. what they do — ViceGodIm · 2026-11-03
- Steering Qwen-Image-2.1 with Qwen3-VL's internal states, no training needed — zacharynado · 2026-11-03
- Report finds LLMs surprisingly good at reading and writing neural network weights — omarsar0 · 2026-11-03
- Researcher urges critics to read the prompts in the 'weight-space programming' post first — anton_lozhkov · 2026-11-03
- New paper: standard CoT monitorability metric is only an upper bound, improvable via self-distillation and RL — armancohan · 2026-11-03