Qwen MoE on 4x RTX 3090: expert hit rate drives decode, PCIe drives prefill

maxigs0 · reddit · 2026-11-13

A detailed benchmark of unsloth UD-Q4/Q5 quantized Qwen MoE via Strata on 4x RTX 3090 (PCIe 3.0, no NVLink) yields counterintuitive findings:

Decode speed depends on experts fitting in VRAM: going from 2 to 3 cards raises expert hit rate from 88% to 95%, cutting per-step time from 33ms to 26-27ms (+25-35% tok/s). A 4th card adds little to decode.
Prompt reading depends on PCIe width: two x8 cards read prompts half as fast as two x16; three cards are no faster than two, but all four read twice as fast as any 2 or 3.
PCIe width doesn't affect decode speed.
Q5 vs Q4: Q5 lowers perplexity by only 0.55% but decodes 28% slower; a 3rd card restores Q5 decode to Q4-on-2-cards speed.
The Q80 n-gram table gives no measurable quality gain but frees 27GB RAM via NVMe mmap.
200K context: a cold 190K prompt takes 91s once; afterward, short turns start in 0.3s and decode at 78 tok/s.

Full tables across card combinations included — valuable data for running long-context MoE models on consumer GPUs.

More from Infra