LoopVQ Hits Sub-1-bit KV Cache Quantization With Up to 6.1x Throughput
SeoulNationalUniv · hf · 2026-11-05
Seoul National University proposes LoopVQ, a KV cache vector quantization method for looped transformers that jointly codes channels across loop steps and fine-tunes codebooks. At 0.5 bits per element it beats existing KV quantization methods on general, long-CoT and long-context tasks; combined with TaSQ it retains most BF16 accuracy at 0.25 bits. Implemented in SGLang, it nearly halves decode latency at 32K context and boosts peak throughput up to 4.3x/6.1x on the 1.4B/2.6B Ouro models.
More from Infra
- Dev: skip any GPU under 256GB for local AI, prices are about to fall — AIFlow_ML · 2026-11-05
- int8-vllm fork rewrites serving stack, hits 1000+ tok/s on 4x MI100 — 1ncehost · 2026-11-05
- Together Compiler: agents re-tune inference for up to 2.6x TPS gains — simran_s_arora · 2026-11-05
- US economy runs on just 18% electricity, transport only 1% electrified — NinaDSchick · 2026-11-05
- AI industry's new defaults: 15-25% of planned compute at risk, Anthropic IPO haircut — sytelus · 2026-11-05
- hxiao: inference overfitting is real, Together Compiler hillclimbs TPS on task token distributions — simran_s_arora · 2026-11-05