HyQuant: Hybrid-Precision Quantization for LLM Attention
Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang
cs.AI
2026-08-28
HyQuant keeps vertical-line tokens and a local window in full precision, 4-bit for the rest. LongBench stays near FA2; 32K decode kernels speed up 3.58×, end-to-end 1.17×.
Long chain-of-thought traces push context to tens of thousands of tokens. Prefill is GEMM-bound; decode rereads the KV cache and hits memory bandwidth. Attention is not uniform. Heatmaps often show a few positions attended by many later queries, forming vertical lines that cover well under 5% of tokens. Uniform low-bit quantization plants error on those high-score positions, and later queries amplify it.
Prior work either drops the long tail with a sparsity mask (MInference) or mixes precision across layers (KVTuner). Few methods keep the vertical line in full precision at token granularity.
HyQuant splits keys into vertical-line positions, a recent sliding window, and the quantized remainder. The vertical-line score is accumulated column mass; the top-ρ fraction is taken from the non-window prefix, 5% by default. Window size is fixed. Identification is a light reduction every 64 tokens, reported as 3% to 5% of runtime.
In prefill, quantized and full-precision paths share one FlashAttention-style kernel: most GEMMs run low-bit, vertical lines and the window stay FP16/BF16, with a single online softmax. In decode the KV cache is stored the same way, and dequantization is fused into the scan so a full-precision cache is never materialized. That differs from KIVI/KVTuner implementations that dequantize first, then run standard attention.
On Llama-3.1-8B, global top-1% / top-5% keys cover 58.83% / 64.09% of attention mass, and 85.63% once a W=128 window is added; Qwen3-8B sits at 47.30% / 54.10% / 82.53%. On Qwen3-8B layer 28, keeping only the top-1%/5% in full precision with the rest in 4-bit brings attention-output MSE close to uniform 8-bit.
LongBench v1, Qwen3-8B thinking: HyQuant 45.04 average versus FlashAttention-2 44.59, KIVI 37.68, SageAttention 38.13, KVTuner 40.45. Llama-3.1-8B-Instruct: 46.73 versus FA2 46.63. Math on Qwen3-8B: GSM8K 96.52 versus FA2 95.88 and KIVI 92.48; MATH500 78.73 versus FA2 80.14, a small drop.
On H100, decode-kernel speedup is 1.32× at 1K prefix and 3.58× at 32K (6.354 to 1.775 ms/token). End-to-end decode versus FA2 runs from 1.04× to 1.17×. KIVI/KVTuner land at 0.69–0.80× relative to FA2 because they dequantize then attend. At 32K on one 80GB card, HyQuant still runs batch 32 at 231.6 token/s; FA2 and the baselines OOM.
The vertical line that MInference used as a sparsity mask is reused here as a precision budget: the tail stays, just at low bit-width. Fused kernels skip the bandwidth tax of "dequantize, then standard attention," which is why kernel speedup shows up at long prefixes. The design is aimed at long CoT decode; short context gains less.
A few LongBench scores sit slightly above FA2. The paper treats that as eval noise, not as quantization making the model smarter.
Short tasks gain little from vertical-line retention. Runs are on H100; high-end GPUs at short context are often not memory-bound, so end-to-end speedup shrinks. The largest model is Qwen3-32B, not 80B. Agent and coding workloads are untested. MATH500 is below FA2 (78.73 vs 80.14), so "near lossless" is mainly a LongBench and GSM8K claim. Keeping 5% of tokens in full precision adds about 15% non-window KV versus strict 4-bit, plus the window itself.