BitDecoding-FP4: native FP4 KV cache decoding via algorithm-system co-design
kalyan_kpl · x · 2026-11-20
A new paper presents BitDecoding-FP4, an algorithm-system co-design enabling LLM decoding with native 4-bit floating-point (FP4) KV caches while preserving BF16-level accuracy.
Problem
- KV caches consume substantial GPU memory and bandwidth in long-context inference.
- Existing FP4 approaches degrade accuracy due to query quantization errors, softmax probability underflow, and key-channel outliers; accuracy-preserving 4-bit methods add overhead that limits throughput.
Approach
- Dual-Residual Query Quantization: represents queries with two FP4 tensors, cutting attention logit quantization error from 60% to 1%.
- Underflow-Aware Probability Scaling: prevents small probabilities from underflowing during quantization.
A directly applicable engineering direction for memory/bandwidth bottlenecks in long-context serving.
More from Infra
- Prefill beats decode: SGLang wins agent runs on DGX Spark despite slower tok/s — AIFlow_ML · 2026-11-20
- Nvidia supplier commitments jump from $87B to $203B as forecasts miss low — tengyanAI · 2026-11-20
- Qwen3.8-Flash-Next on a single Intel Arc B70: 2.5K prefill, 47 tok/s — WizardlyBump17 · 2026-11-20
- SQLite is ~35% faster than the filesystem for 10KB blobs — here's why the syscall overhead adds up — arpit_bhayani · 2026-11-20
- NVIDIA's NarrowSpec: Halving Draft Hidden Width Still Speeds Up Speculative Decoding 1.44x — nvidia · 2026-11-20
- Tencent Hy4 lands in vLLM day-0: 774B MoE with 49B active, 1M context, 8-GPU FP8 recipe — vllm_project · 2026-11-20