Real benchmark finds Gemma MoE is 20% cheaper and 25.5% faster than dense Gemma
Practical-Koala2831 · reddit · 2026-07-21
A real-world benchmark compared Dense gemma-4-31b-it with MoE gemma-4-26b-a4b-it on the same 100 prompts via 200 live OpenRouter API calls.
- Cost: MoE was 20% cheaper per query.
- Latency: MoE was 25.5% faster on average.
- Output quality: token output was unchanged.
- Tail latency: the advantage narrowed at P50 to 27.3% and at P95 to 12.9%, suggesting both models hit the same infrastructure ceiling under peak load.
The author argues the common claim that MoE models are cheaper holds up in practice, but notes that teams with strict SLAs should test their own tail latency before switching. At scale, the post estimates the 20% gap could mean about $2,970/month at 100M daily queries.
More from Infra
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- NVIDIA claims HGX Rubin NVL8 delivers 10x tokens per megawatt vs B200 — thursdai_pod · 2026-10-27
- NVIDIA Claims HGX Rubin NVL8 Delivers 10x More Tokens per Megawatt Than B200 — thursdai_pod · 2026-10-27
- Together lets multiple LoRA post-training experiments share the same GPU capacity — togethercompute · 2026-10-27
- Commoditization dilemma: either human labor or GPUs become the commodity — _AustinCalvert_ · 2026-10-27