Run Qwen 27B on 16GB VRAM: llama.cpp MTP mod adds 17% speed, more context
ea_man · reddit · 2026-09-01
The author built MTP Compact Rollback, a low-VRAM modification of llama.cpp's multi-token-prediction speculative decoding, letting 16GB cards run models like Qwen 27B while keeping decent context.
The idea: more speculative tokens means more speed, but rollback states eat VRAM and context. The new flag --spec-mtp-cr-depth 1 keeps only one immediate rollback snapshot while still allowing MTP to draft 5 tokens — depth-5 speed at depth-1 context cost.
Measured results (16GB, well-tuned IQ4 quant):
| Mode | Max draft n | Context | Speed |
|---|---|---|---|
| Standard MTP | 2 | 72,192 | 39.53 t/s |
| Compact Rollback | 5 | 77,312 | 46.39 t/s |
That's +5k context and a 17.35% speedup; coding workloads see 80-98% acceptance rates.
Pairs with Adaptive MTP (--spec-draft-adaptive), which dynamically adjusts draft depth from recent acceptance — high n-max for code, lower for creative prose in long reasoning traces. Near-zero overhead, recommended always on.
Full recommended flags for coding on 27B and A3B:
--spec-mtp-cr-depth 1 --spec-draft-adaptive \
--spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.80 --spec-draft-n-max 5 \
--cache-type-k-draft q40 --cache-type-v-draft q40 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32
Build: clone llama.cpp at the pinned commit, apply the author's patch; example scripts and docs linked in the post.
More from coding & agent
- Code should be explainable, not necessarily readable, in the post-AI era — kieranklaassen · 2026-09-01
- Dev uses AI-generated textbooks with custom tool for interactive learning — generativist · 2026-09-01
- Agents Cannot Read LinkedIn: Social Platforms Block AI Crawlers via Robots.txt — Dry_Steak30 · 2026-09-01
- State-Driven Protocol Stops LLM Over-Fixing and Context Collapse Without Vector DB — wenger2026-12 · 2026-09-01
- Using Rust Runtime Constraints to Guide AI Agents in Writing Concurrency-Safe Code — doodlestein · 2026-09-01
- Intent Hailed as the GOAT of Laptop Dev Environments — Wattenberger · 2026-09-01