Sort prompts before batch MoE inference: a free 7–17% speedup in stock vLLM
yb2698 · x · 2026-10-20
A zero-cost MoE batch inference trick: sort your prompts before calling llm.generate.
Why it works: each decode step reads every expert picked by any request in the batch. Mixed-domain batches touch more experts; same-domain batches overlap, improving expert cache hits.
Measured on stock vLLM with identical requests, only the order changed:
- Qwen3-30B-A3B: +12–17%
- gpt-oss-120b: +7–12%
Just one argsort — no model or framework changes needed.
More from Infra
- Morgan Stanley: $700B may reach Nvidia by 2027, but US data centers face 32GW power gap — rohanpaul_ai · 2026-10-20
- Stop button often doesn't stop the LLM: tokens keep generating unless the app hangs up — No-Peanut-6988 · 2026-10-20
- ComfyUI hits 140k stars: text-to-image generation now runs on your own GPU — alex_verem · 2026-10-20
- Immich at 116k stars: face search across your photo library without Google — alex_verem · 2026-10-20
- whisper.cpp at 54k stars: hours of offline transcription on a normal laptop — alex_verem · 2026-10-20
- Five AI tasks that needed a data center in 2020 now run at home in 2026 — alex_verem · 2026-10-20