Hands-on LLM inference: boosting tokens-per-second with a 31B Gemma model
abhijithneil · x · 2026-09-22
A tutorial on self-hosting LLMs and improving inference speed (TPS, Tokens per Second). The author notes that inference engineering in 2026 still has no one-step playbook — frameworks are still evolving to accommodate different architectures, so this step requires understanding GPU kernels, model configuration, architectures, and ML theory.
Using a 31B-parameter Gemma 4 LLM, the author demonstrates how to measure and improve generation throughput (e.g., how fast a 500-word answer to "What is machine learning?" completes). A practical starting point for engineers entering inference engineering.
More from Infra
- When Does Local AI Hardware Beat Cloud Models for Coding? Measure Cost per Accepted Change — Rama_Surasani_ · 2026-09-22
- rakyll: fully managed platforms fail on composability — bespoke and open source stacks win — rakyll · 2026-09-22
- Local Qwen 27B agent logs into Amazon and buys paper autonomously in one run — fuzhongkai · 2026-09-22
- Qwen Code TUI loses a transcript line on every rows-only shrink, blamed on ink 7.0.3 — SnowCore8 · 2026-09-22
- 60 Minutes: US golf courses use more than twice as much water as data centers — SumitGup · 2026-09-22
- ROCm vs Vulkan on R9700 + Strix Halo: ROCm still wins for DeepSeek, Vulkan closes in — Hrethric · 2026-09-22