Hot-swappable speculative decoding boosts Qwen 27B on 16GB CUDA
tsangberg · reddit · 2026-09-13
Reddit user tsangberg built on Raymond's KV cache streaming fork of llama.cpp to add hot-swappable speculative decoding, open-sourced as llama.cpp-adaptive-kv-streaming.
- The base idea: a VRAM pool is used differently during prompt processing vs decoding, streaming KV data from host RAM when context exceeds VRAM, yielding much higher token generation throughput than standard offloading.
- On top of that, the author ejects the MTP or DFlash2 speculative model when VRAM runs tight and reloads it once context shrinks (compaction), using spare VRAM for spec decoding.
- He has run Qwen 3.8 27B UD-IQ4XS on a 5060 Ti (16GB) this way since release, and shares page-level tuning curves showing the optimal point to eject the spec model.
- The approach targets 16GB cards but the mechanism should help users with more VRAM too.
More from coding & agent
- Anthropic quietly ships server-side rule banning game rewrites in Claude Code — petrusenko_max · 2026-10-31
- Bootstrapping competitive edge with frontier tokens — curious_vii · 2026-10-31
- Element Forge: open-source editable AI VFX for real footage in ComfyUI — zitroffell · 2026-10-30
- Gym-Anything Adds Audit Agents That Try to Break Simulations With Legal Robot Actions — wellecks · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- This 5-question interview prompt makes Grok design your first two custom bots in minutes — nateliason · 2026-10-30