llama.cpp adaptive MTP PR speeds up code generation by up to 100%
Look_0ver_There · reddit · 2026-08-18
PR #27210 adds an adaptive MTP mode to llama.cpp: a simple counting-style state machine dynamically picks the speculative decoding draft depth, so users no longer need to tune MTP depth manually.
Measured results:
- Dense/hard prose runs about 3% slower than fixed MTP=3 (some prose scenarios average higher).
- Code generation is 10-15% faster; recalling code from the thinking phase is over 50% faster; when the model rewrites a whole file from memory, generation is up to 100% faster.
- Recalled prose gains 20-30%.
- Higher temperatures shrink the advantage, though code still does slightly better.
Recommended config: --spec-type draft-mtp-adaptive --spec-draft-n-max 12 (depth range 3-12), with --spec-draft-n-min-adaptive to lower the floor.
More from Infra
- Mesh LLM: Turning Piles of Computers into One Shared AI Machine — alex_verem · 2026-08-18
- Mesh LLM: A Third Option Between Crypto Rigs and Cloud Subscriptions — alex_verem · 2026-08-18
- Qwen3.8-9B MLX port runs on 16GB Macs with fast speed — alexcovo_eth · 2026-08-18
- Qwen 3.8 on Apple Silicon speeds up nearly 3x using AI-written kernels — alexcovo_eth · 2026-08-18
- Bills Went Up After OpenAI's 80% Price Cut: Jevons Paradox Hits Inference — AccBalanced · 2026-08-18
- Why Stripe Bought Metronome for $1B Instead of Building It — mattturck · 2026-08-18