Right-sizing context windows: Gemma 4 26B voice app fits on a single H100
Ctbhatia · reddit · 2026-10-17
An engineer shares deployment sizing for Gemma 4 26B (gemma-4-26B-A4B-it) on a voice app: reserving memory for the native 131K window pushes you to bigger hardware, but their real max prompt was 15K tokens. Capping the serving window at 49K let the checkpoint plus KV cache fit on a single H100.
Alternatives they benchmarked:
- Two H100s at tensor parallelism 2: no significant latency gain since one card already sufficed
- One H200 at full window: 24% more output per GPU but only 10% lower p50 time-to-first-token
Neither earned its cost. The cap doesn't make the model faster — it makes each replica cheaper. The author asks: do you size hardware by real max prompt or native window? Full comparison on their blog.
More from Infra
- Kimi K3 three months later: 1.56TB is too big to run locally and rivals have caught up — Sanbi_Ai · 2026-10-17
- Epoch AI chart: xAI owns fewer GPUs than CoreWeave, far behind hyperscalers — bookwormengr · 2026-10-17
- Nvidia CEO Jensen Huang says AI stock selloff is temporary: "All of this is going to pass" — Polymarket · 2026-10-17
- Cloudflare rolls out full ZDR on Workers AI: no data retention, no training on your data — michellechen · 2026-10-17
- Modal explains its speed moat: ImageFS tiered filesystem now tricking VM ext4 — dejavucoder · 2026-10-17
- AWS built a production eval framework on Bedrock AgentCore: agents fail silently — AWS ML Blog · 2026-10-17