Baidu's Virtual Head Attention cuts KV cache 4-7x with zero custom kernels
burny_tech · x · 2026-10-19
Baidu and Peking University researchers propose Virtual Head Attention (VHA), tackling the "impossible triangle" of KV cache size, training/prefill FLOPs, and model quality in long-context LLMs.
- Method: two lightweight linear ops — Q Premix applies a near-identity transform within each KV group to restore virtual-head diversity from halved physical heads, and Linear Postmix fuses inter-head features via a low-rank residual I+ABᵀ; no nonlinearities introduced.
- Key selling point: at inference, Premix and Postmix fold into the query and output projections, reducing the model to a standard GQA-2 architecture that reuses all existing GQA optimizations with zero custom kernels — unlike MLA.
- Results: only 2 KV heads reach GQA-4 level in large models; across 0.6B/1.7B/4B dense and 8B-A1B MoE pretrained on 50B tokens at 4K context, VHA consistently beats GQA baselines, cutting KV cache 4x at 4B dense (2x MoE) and training/prefill FLOPs by 7.9% (1.7B); an industrial 3B-A3B model outperforms MLA and GQA-4 by +0.702.60% on a 12-benchmark suite; 8B-A1B MoE scores 84.4 avg RULER 8K-128K vs GQA's 73.0 with 2 KV heads instead of 4.
Code is open-sourced (PaddlePaddle/VHA).
More from Infra
- vLLM cuts container startup time 14-18% with one uv bytecode flag — vllm_project · 2026-10-19
- Why neoclouds get acquired by outsiders, never by each other — AccBalanced · 2026-10-19
- Jev Router cuts coding agent costs 67-85% by routing tasks to cheap open models — HowDevelop · 2026-10-19
- Server memory spot growth slows to +0.6% as storage stocks fall 5-7% in a day — tengyanAI · 2026-10-19
- RTX Spark vs M5 Max: local Qwen 3.8 27B benchmark shows NVIDIA wins on compute — draginol · 2026-10-19
- Biscotti: a self-hosted Instinct-like iMessage assistant in ~850 LOC — lucasgelfond · 2026-10-19