Run Qwen 27B on 16GB VRAM: llama.cpp MTP mod adds 17% speed, more context

ea_man · reddit · 2026-09-01

The author built MTP Compact Rollback, a low-VRAM modification of llama.cpp's multi-token-prediction speculative decoding, letting 16GB cards run models like Qwen 27B while keeping decent context.

The idea: more speculative tokens means more speed, but rollback states eat VRAM and context. The new flag --spec-mtp-cr-depth 1 keeps only one immediate rollback snapshot while still allowing MTP to draft 5 tokens — depth-5 speed at depth-1 context cost.

Measured results (16GB, well-tuned IQ4 quant):

|---|---|---|---|

| Standard MTP | 2 | 72,192 | 39.53 t/s |

| Compact Rollback | 5 | 77,312 | 46.39 t/s |

That's +5k context and a 17.35% speedup; coding workloads see 80-98% acceptance rates.

Pairs with Adaptive MTP (--spec-draft-adaptive), which dynamically adjusts draft depth from recent acceptance — high n-max for code, lower for creative prose in long reasoning traces. Near-zero overhead, recommended always on.

Full recommended flags for coding on 27B and A3B:

--spec-mtp-cr-depth 1 --spec-draft-adaptive \

--spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.80 --spec-draft-n-max 5 \

--cache-type-k-draft q40 --cache-type-v-draft q40 \

--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32

Build: clone llama.cpp at the pinned commit, apply the author's patch; example scripts and docs linked in the post.

Original post →

Run Qwen 27B on 16GB VRAM: llama.cpp MTP mod adds 17% speed, more context

More from coding & agent