Benchmarking MTP Prefetching for MoE Experts
zyxciss · reddit · 2026-07-17
In a CPU/GPU offload scenario for **qwen3.6 35b a3b**, the author uses the model's native **MTP head** to predict the next token and prefetch the likely experts in the background, effectively hiding PCIe transfer latency during the current token's computation. Initial instrumentation on llama.cpp reveals: - Prefetching based on the "last used expert" yields a mere **20.7%** hit rate, rendering it practically useless. - With MTP-guided prediction, the **top-8 hit rate reaches 78%** and **top-16 hits 90%**, though larger K values incur higher bandwidth overhead. - A clear **hot expert** long-tail effect is observed: the top 64 experts (out of 256) account for **51%** of total usage. - The current baseline is around **35–36 tok/s**, with a theoretical limit of **~200 tok/s** if all experts reside in VRAM, indicating the bottleneck is PCIe transfer rather than compute power. The author seeks feedback on the viability of this approach: whether similar implementations exist, if router-only forward overhead negates the gains, and if it breaks down at larger batch sizes. They prefer forking llama.cpp over rewriting an engine entirely.
More from coding & agent
- Codex turns out 123 screensavers in one playful batch — intellectronica · 2026-07-21
- Grok Build adds `grok doctor`, resumable sessions and remote image paste — mark_k · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- CHAP defines approvals, handoffs, and audit logs for human-agent workflows — DeliveryTechnical199 · 2026-07-21
- The author says Codex reached 20x and is now debugging spec decoding on a hybrid parallel setup — TheZachMueller · 2026-07-21
- Axcess adds an MCP connector for WCAG accessibility checks that scanners miss — modelcontextprotocol · 2026-07-21