Benchmarking a Qwen MoE on 4×RTX 3090: expert hit rate drives decode speed

maxigs0 · reddit · 2026-11-13

A detailed Reddit benchmark of unsloth UD-Q4/Q5 quants of a Qwen MoE via Strata on 4×RTX 3090s (no NVLink, PCIe 3.0) yields reusable findings:

Decode speed tracks how many experts fit in VRAM: going 2→3 cards raises expert hit rate from 88% to 95% and cuts ms/step from 33 to 26-27 (+25-35% tok/s); a 4th card adds little to decode.
Prompt reading is PCIe-bound: x8 pairs read prompts half as fast as x16; all four cards read 2× faster than any 2-3.
Q5 vs Q4: only 0.55% lower perplexity but 28% slower decode and 35-50% slower prompt reads.
The Q80 n-gram table gives no measurable gain over IQ4NL but frees 27GB RAM via NVMe mmap.
200K chat: a cold 190K prompt takes 91s once; then short turns start in 0.3s and decode at 78 tok/s.

Includes full per-card-combination tables (ms/step, tok/s, prompt read times, expert hit rate) — a practical reference for local MoE serving.

More from Infra