Gemma hits 2.85x speedup on vLLM: 37,900 to 108,000+ tokens/sec on one A100
patloeber · x · 2026-10-27
- Using Gemini 4 Argon as a coding assistant, the Gemma team pushed gemma-4-E2B-it throughput on a single A100 under vLLM from 37,900 to 108,000+ tokens/sec — a 2.85x speedup with 100% token parity and zero quality drop, done in days rather than months.
- The work spans 8 PRs by vLLM collaborator lucianommartins, including fused MTP speculative decoding layers with sparse gather-GEMV LM head, fused Triton kernels for QKV norm/RoPE/pre-post-FFN norms/MoE routing, tuned Triton FusedMoE configs for Gemma 4 26B-A4B (E=128), and multimodal fusions (vision/audio RMSNorm, batched image embedder).
- A concrete showcase of LLM-assisted kernel optimization landing in production inference.
More from Infra
- Anthropic reportedly allocates 72% of compute to training, not serving — abhiadesai · 2026-10-27
- AI boom fuels a cloud computing 'land grab' as hyperscalers scramble to keep pace — DavidLinthicum · 2026-10-27
- Categorica open-sources native Prometheus exporter module for NGINX, dropping the sidecar — Technovangelist · 2026-10-27
- NVIDIA shows one-click Unsloth setup on RTX Spark for local training and RL — danielhanchen · 2026-10-27
- Bloomberg data story details Anthropic's surging revenue vs. staggering compute bills — nmasc_ · 2026-10-27
- GPU FP32-accumulation unlocks explained: 4090 hits 331 TFLOPS, but Blackwell stays locked — panchovix · 2026-10-27