vLLM contributor: lossless KV cache quantization viable but not worth it
AlpinDale · x · 2026-11-07
vLLM contributor AlpinDale assessed applying the NF12 approach to lossless KV cache quantization: it is feasible and might approach BF16 KV performance with effort, but the memory savings are small and the added serving-engine complexity too high, so he is passing on it.
More from Infra
- Open-source fork runs 47GiB MoE on 2019 Radeon VII with 49% faster decode — milpster · 2026-11-07
- SmolLM-135M running in pure SQLite at ~40 seconds per token — cneuralnetwork · 2026-11-07
- NUS professor mocks Jensen Huang: he doesn't need a Chinese spy to sell chips to China — pmddomingos · 2026-11-07
- Anthropic's amended S-1 admits mounting losses amid $110B in annual compute commitments — GaryMarcus · 2026-11-07
- Xepak: declarative local builder for REST, RPC, MCP and RAG servers that agents can write DSL for — rumatoest · 2026-11-07
- US AI scale now hinges on coordinating power, components, agentic workflows and humanoid pilots — ingliguori · 2026-11-07