FP4 KV cache with no degradation sparks debate on aggressive inference compression
stochasticchasm · x · 2026-09-11
Engineers are discussing an aggressive inference optimization: FP4 KV cache with no noticeable degradation, combined with compressing KVs via CSA, leaves models barely using any KV cache.
The author adds nuance: within a single request you still need the KV cache in HBM/RAM, but between requests re-prefilling is negligible compute at 500K context. Kernel reduction order and similar details may matter a lot for performance.
Related event: Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship(44 posts)→
More from Infra
- EdgeRock open-sourced: 72 MiB KV cache at 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27
- Neon rebuilds its docs search entirely on its own stack: Lakebase Search, Functions, AI Gateway — matei_zaharia · 2026-10-27
- PyTorch details 2.12x faster on-device streaming ASR on Arm SME2 via ExecuTorch — PyTorch · 2026-10-27