Patched vLLM+FlashInfer Pushes Gemma 4 31B to 150 tok/s on a Single B300, Beating SGLang
abhijithneil · x · 2026-09-13
- The author boosted Gemma 4 31B throughput on a single B300 GPU from a 46.7 tok/s eager baseline to 150 tok/s in non-lossy configurations, beating SGLang's best of 139 tok/s
- Key obstacle: Gemma 4's ten global layers use headdim 512, which most attention kernels don't support — FlashInfer in SGLang handled it, but vLLM's path capped at headdim 256
- The author forked vLLM and wrote a patch to lift the headdim 256 limit, and vLLM+FlashInfer hit 150 tok/s vs SGLang+FlashInfer's 139 tok/s
- A blog post with details is coming soon
More from Infra
- Would LITE accepting MRVL's merger offer prove its moat was overstated? — BenBajarin · 2026-10-30
- Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30