Virtual Head Attention: 4x smaller KV cache, beats GQA on long context (RULER 84.4 vs 73.0)
pmttyji · reddit · 2026-10-23
A new paper proposes Virtual Head Attention (VHA), resolving the 'impossible triangle' of KV cache size, training/prefill FLOPs, and model quality with two lightweight linear ops (Q Premix and Linear Postmix) that fold into projections at inference, reducing to standard GQA-2 and reusing all existing GQA optimizations.
Results: across four scales with 4K-context 50B-token pretraining, VHA beats GQA baselines by +0.05% to +0.81%, cuts KV cache 4x vs GQA-8 (2x at MoE scale), and lowers training/prefill FLOPs by 7.9%. On a 30B-A3B MoE trained on 500B tokens at 8K context, VHA outperforms MLA and GQA-4 by +0.70%/+2.60%; a 10B-A1B long-context comparison reaches RULER 84.4 vs 73.0 for GQA with only two KV heads. Paper and code (PaddlePaddle/VHA) are public.
More from Infra
- PsiQuantum's 12-inch thin-film BTO wafer spotted in person, and it looks stunning — clarejtbirch · 2026-10-24
- BofA's data center map: servers cost $31.4M per MW, dwarfing every other component — TiernanRayTech · 2026-10-24
- Chip-Backed Debt Is on Track to Become a Trillion-Dollar Market, Concentrating Risk — dinabass · 2026-10-24
- Applied Compute Featured on NVIDIA GTC Keynote for Enterprise Post-Training Platform — rhythmrg · 2026-10-24
- OpenAI details how it scaled ChatGPT storage with multi-tenancy and global caching for 12x faster reads — OpenAI News · 2026-10-24
- Google Cloud's API Gateway now supports streaming endpoints for LLM routing — rseroter · 2026-10-24