Fathom Speeds Million-Token KV Scans 1.67x with Per-Query Read Depth
Vivek Kalyanarangan · hf · 2026-09-17
When agentic sessions hit a million tokens with many sessions resident, the KV cache scan that ranks all n keys bounds decoding. Fathom lets each query decide how many bits of each key channel to read.
- The 4-bit K cache is stored channel-major as bit planes; a prefix of t planes is exactly the channel's t-bit quantizer, with bit budgets allocated via reverse water-filling
- On Qwen3-8B at 1M tokens, decode steps are 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki and SparQ r=32
- At SparQ's 68-bit GPU time it reads 18% fewer bytes with lower attention error; on RULER tasks every scan matches exact top-k decoding
- On real coding-agent sessions, 92 bits reaches the step agreement of the most accurate 136-bit scan; it reuses the 4-bit K copy quantized serving stacks already hold
More from Infra
- The new AI stack starts below models: energy, grid and data centers — ingliguori · 2026-09-17
- US House Passes Bipartisan Bill Making Data Centers Cover More Grid Costs — Polymarket · 2026-09-17
- Dev trains a 2B Apache-licensed model on 24GB VRAM using OLMo embeddings and a 1B Engram table — NineThreeTilNow · 2026-09-17
- Applied Materials to invest $5bn in India as country boosts chip support — pstAsiatech · 2026-09-17
- Cloud AI Costs 10-20x More Than On-Premises, Expert Warns of Repeat of Early Cloud Mistakes — DavidLinthicum · 2026-09-17
- Huawei says AI chip demand outstrips supply, plans two new AI chips by 2027 — sunychoudhary · 2026-09-17