Hot-swappable speculative decoding boosts Qwen 27B on 16GB CUDA
tsangberg · reddit · 2026-09-13
Reddit user tsangberg built on Raymond's KV cache streaming fork of llama.cpp to add hot-swappable speculative decoding, open-sourced as llama.cpp-adaptive-kv-streaming.
- The base idea: a VRAM pool is used differently during prompt processing vs decoding, streaming KV data from host RAM when context exceeds VRAM, yielding much higher token generation throughput than standard offloading.
- On top of that, the author ejects the MTP or DFlash2 speculative model when VRAM runs tight and reloads it once context shrinks (compaction), using spare VRAM for spec decoding.
- He has run Qwen 3.8 27B UD-IQ4XS on a 5060 Ti (16GB) this way since release, and shares page-level tuning curves showing the optimal point to eject the spec model.
- The approach targets 16GB cards but the mechanism should help users with more VRAM too.
More from coding & agent
- Grok Bot went from zero to launch in 7 weeks: ex-Cursor growth lead Roman Ugarte tells the story — lennysan · 2026-09-14
- Cognition launches SWE-2: frontier-level coding model at up to 70% lower cost — gethackteam · 2026-09-14
- Dev: With Agents, You Can Fix Anything on Linux With a Prompt — steipete · 2026-09-14
- Marmel 0.9.0: autonomous coding agent tuned to run local models like gemma 4 12b — Naiw80 · 2026-09-14
- Which coding agent is everyone actually using? bough reads agent session histories — VoidEqualZero · 2026-09-14
- Dev builds agent harness for RTS games, optimizing info density and decisions per minute — coding_is_tedious · 2026-09-14