Emmy cuts Gemma 4-12B TTFT on RTX 5090, but steady-state throughput barely changes
NoVibeCoding · reddit · 2026-08-04
The author presents Emmy, a vLLM plugin and benchmarking/deployment tool for faster LLM inference on RTX GPUs.
The main claim is that compiler-generated GEMM and FlashAttention kernels can reduce time-to-first-token (TTFT), but not token generation speed, since TPOT remains memory-bound. The post benchmarks Gemma 4-12B on an RTX 5090 and compares stock vLLM, vLLM + Emmy, Emmy with FASTMATH, and llama.cpp across multiple prompt lengths and concurrency levels.
Key results:
- TTFT improves in several settings, especially with FASTMATH.
- Steady-state token throughput is often similar to stock vLLM, sometimes slightly worse due to plugin overhead.
- Speculative decoding with MTP is also smoke-tested to check for quality or performance regressions.
The repo and Docker image are provided for anyone who wants to reproduce the setup.
More from Infra
- Nuclear Startup Valar Raises $1B Led by Sequoia to Scale Reactors — kleffew94 · 2026-08-04
- AI API revenue still trails hyperscaler capex by a wide margin in 2025 chart — SurpriseDog9000 · 2026-08-04
- U.S. heartland backlash grows as AI data centers reshape local communities — altryne · 2026-08-04
- Semiconductors and data centers are being built far slower than AI demand — robleclerc · 2026-08-04
- Gemma 4 31B can use over 13× more KV-cache memory than DeepSeek V4 Flash — teortaxesTex · 2026-08-04
- MCP server brings structured compile, flash and stateful GDB to embedded boards — Historical_Court795 · 2026-08-04