Gemma optimized to break 120k tokens/sec on a single A100 in vLLM, a ~2.9x speedup
_philschmid · x · 2026-10-14
- The team optimized Gemma 4 E2B inference, pushing past 120,000+ tokens/sec on a single A100 with vLLM: 124k prefill and 111k end-to-end, a 2.9x speedup with zero quality degradation.
- The demo shows 64 live Python simulations generated in parallel.
- A useful reference for small-model throughput after inference-stack optimization, especially for local/low-cost deployment.
Related event: Argon Optimizes Gemma to Exceed 120K Tokens/s on a Single A100(2 posts)→
More from Infra
- Musk: Micron's memory pricing is reasonable while rivals are 'extortionate' — elonmusk · 2026-10-14
- OpenRouter's Average Token Price Nearly Halved in 2026, Yet Daily Spend Tripled — maferase · 2026-10-14
- Someone Is Running Qwen 27B at ~20 tok/s on a Jailbroken PS5's GPU — LatentSpacer · 2026-10-14
- ASE Panels Spotted at OCP Showfloor, Reportedly Destined for OpenAI's Venice Data Center — BenBajarin · 2026-10-14
- Semicon West and OCP point to next bottlenecks: high-kW liquid cooling, 800VDC, semicap — BenBajarin · 2026-10-14
- Starship launch cadence capped near 145/year at current sites, a bottleneck for orbital compute — JOBhakdi · 2026-10-14