Qwen 27B at Full 262k Context on a 16GB RX 9070 XT: 80+ t/s Decode With Adaptive KV Streaming
bottlestreet3d · reddit · 2026-10-17
The author implements adaptive KV streaming plus RDNA 4/ROCm kernel optimizations in llama.cpp, running Qwen 27B (IQ4XS weights, Q8/Q4 KV cache) with its full 262,144-token context on a single 16GB RX 9070 XT at 80+ t/s decode with MTP.
Problem: the model needs 14.3GB and 262k context requires 7.4GB of KV cache, causing OOM past 65k tokens on 16GB.
Approach:
- Ported CUDA adaptive KV streaming to ROCm/HIP: full KV cache stays in host RAM (7.4GB), bounded working sets (450MiB prefill, 2,100MiB decode) stream over PCIe SDMA overlapping compute
- Incorporated community kernel optimizations (multi-column mmvq, vectorized loads, SwiGLU fusion, async HIP graphs)
- Added sibling activation reuse, in-register fused residual+RMSNorm, online Log-Sum-Exp chunked prefill with pipelined prefetch, persistent graph arenas, and optimized speculative decoding
Results: decode 36.5→9.5 t/s (1k→262k), 71.8–86.5 t/s with MTP at short contexts; prefill 1368→586 t/s. A 231W power limit costs only 3.6% prefill latency. Code is open source.
More from Infra
- MiniCPM-V 4.7 runs locally at 100 tok/s using only ~3 GB of memory — NielsRogge · 2026-10-17
- Power-management chips: the invisible layer deciding AI data center energy efficiency — Olivier__OG · 2026-10-17
- MiniCPM-V 4.7 runs a 35B multimodal model in 3GB of memory, fully on-device — alifcoder · 2026-10-17
- Full FP8 n-gram table visibly improves local Qwen3.8-Flash-Next at zero speed cost — sillynoobhorse · 2026-10-17
- Local Nemotron 3.5 on one A100 hits 169 tok/s, processes 100 clinical notes for 2.4 cents — MaziyarPanahi · 2026-10-17
- NVIDIA's open-source AIPerf benchmarking tool and Triton GPU tutorials for inference measurement — techNmak · 2026-10-17