Zyphra's ZHA sparse attention keeps 35B model quality while reading just 1-6% of KV-cache

burny_tech · x · 2026-11-08

Zyphra Research published a paper introducing Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention method targeting the quadratic-scaling bottleneck of long-context LLMs.

How it works

Builds a tree of recursively compressed key summaries over the KV cache; queries traverse it top-down to narrow down which tokens deserve attention
Total cost including routing and final attention grows sublinearly with sequence length, yielding overall subquadratic (but superlinear) scaling

Key results

Can be retrofitted onto pretrained models with only a short 1B-token KL distillation stage
On a 35B-parameter model, scores 93.3% vs 96% on RULER at 256K context while attending to just 1-6% of the KV-cache
Maintains performance on long-context retrieval, reasoning, math, and agentic coding benchmarks
Robust to tree geometry up to 6 levels deep; performance model shows dramatic wins beyond 1M tokens of context

The authors argue the approach could ultimately extend effective context windows to hundreds of millions or billions of tokens.

Original post →

Zyphra's ZHA sparse attention keeps 35B model quality while reading just 1-6% of KV-cache

More from Infra