Benchmarking MTP Prefetching for MoE Experts

zyxciss · reddit · 2026-07-17

In a CPU/GPU offload scenario for qwen3.6 35b a3b, the author uses the model's native MTP head to predict the next token and prefetch the likely experts in the background, effectively hiding PCIe transfer latency during the current token's computation.

Initial instrumentation on llama.cpp reveals:

Prefetching based on the "last used expert" yields a mere 20.7% hit rate, rendering it practically useless.
With MTP-guided prediction, the top-8 hit rate reaches 78% and top-16 hits 90%, though larger K values incur higher bandwidth overhead.
A clear hot expert long-tail effect is observed: the top 64 experts (out of 256) account for 51% of total usage.
The current baseline is around 35–36 tok/s, with a theoretical limit of 200 tok/s if all experts reside in VRAM, indicating the bottleneck is PCIe transfer rather than compute power.

The author seeks feedback on the viability of this approach: whether similar implementations exist, if router-only forward overhead negates the gains, and if it breaks down at larger batch sizes. They prefer forking llama.cpp over rewriting an engine entirely.

Original post →

Benchmarking MTP Prefetching for MoE Experts

More from coding & agent