Testing MTP combined with ngram-mod for coding speed
YetAnotherAnonymoose · reddit · 2026-08-18
A developer tested speculative decoding by combining MTP with ngram-mod in llama.cpp. The task involved repeating a code block 10 times.
Results:
- Initial Boost: TPS hit 156 on the first repetition (avg 60 with MTP only), showing huge bursts of output followed by short pauses.
- Performance Decay: Subsequent bursts slowed down progressively, with TPS dropping from 169 to 82.
Issue:
The author questions whether this is a parameter tuning issue or if the ngram combo is generally not worth it. This is particularly puzzling for models like Qwen3.8, which love to repeat 'thought' code blocks verbatim—an ideal case for ngram.
More from Infra
- Volcengine OpenViking: Self-evolving context DB for agents — volcengine · 2026-08-18
- FreeToken framework claims major MoE inference speedups — wavefnx · 2026-08-18
- FreeToken Claims Faster MoE Inference vs. llama.cpp and Ollama — wavefnx · 2026-08-18
- Epoch AI: Musk Is the Only Frontier Lab CEO Building Data Centers — soumitrashukla9 · 2026-08-18
- AI server demand polarizes MLCC lead times, high-end hits 10 months — zephyr_z9 · 2026-08-18
- DeepSeek V4 Flash Benchmarks: n_max=3 Yields 1.39× Speedup — Responsible_Pain3278 · 2026-08-18