Octave KV lands in vLLM: 4.6x KV cache capacity on AMD GPUs with 3-bit cache
solyarisoftware · x · 2026-11-03
- vLLM has merged ROCMOCTAVE, a native 3-4 bit KV cache backend for AMD GPUs (RDNA3 gfx1100 and MI300X gfx942), enabled via --kv-cache-dtype octavekv3 and related options.
- Capacity: 4.6x the cached tokens of FP16 on 4x RX 7900 XTX with Qwen3.8-27B; on one MI300X, 1.95x the tokens of AITER's fp8 cache (53 vs 27 concurrent 128k requests).
- Quality: GSM8K and a 126k needle-in-a-haystack test match fp16/fp8 on both architectures.
- Speed: on RDNA3 at 380k, decode is 4.6x faster than TurboQuant and MTP verification 11x faster; MI300X single-request decode matches AITER fp8.
- Works with MTP and CUDA graphs; the author reports production use on RDNA3 since October 2026. Note it compresses the KV cache, not model weights, enabling longer conversations or more concurrent requests within the same VRAM.
More from Infra
- 500M-row benchmark: hand-written BFS beats NebulaGraph in MySQL, 2.7ms vs 23.6ms — JeremyCMorgan · 2026-11-03
- Anthropic open-sources FlashProve: provable inference costs ~165x more than raw inference — arthurcolle · 2026-11-03
- PostHog re-rebuilds its data warehouse five months after switching from ClickHouse to DuckDB — teodorio · 2026-11-03
- Local MoE inference on a Tesla V100: 55 tok/s decode with 8,109 experts cached in 31.6GB VRAM — Traditional_Bell8153 · 2026-11-03
- Cities Worldwide Build AI Services on NVIDIA Nemotron, Cosmos and Omniverse — NVIDIAAI · 2026-11-03
- Anthropic spends $2.4B ARR on data vendors — just 6% of its compute bill — himanshustwts · 2026-11-03