Gewell: custom Gemma 4 inference engine cuts KV cache VRAM to 0.625x, losslessly
stoppableDissolution · reddit · 2026-09-21
A developer built Gewell, an engine for high-concurrency Gemma 4 31B on Blackwell. Key insight: Gemma's tied K/V weights on global attention layers and 25% RoPE on K allow mathematically lossless 0.625x KV cache VRAM — vLLM and llama.cpp store full K and V. The extra unsqueeze math is quickly outweighed by reading less memory, enabling more context and cached prefixes.
Its cache manager targets writer+critic style data-generation loops: vLLM's LRU evicts reusable prefixes from intermittent prompts, while Gewell offers smarter eviction, explicit cache hints, and checkpoint merging to keep multiple sessions warm. GEMM shapes are profiled offline and hardcoded, so startup takes seconds. Benchmarks: TTFT slightly behind vLLM on large batches, but higher overall t/s on the intended workload.
More from Infra
- Building a 4-GPU local LLM rig: why Threadripper beats LGA1700 on PCIe lanes — El_90 · 2026-09-21
- Komlós conjecture solution announced, with overlooked implications for neural network quantization — stevenstrogatz · 2026-09-21
- Nvidia Names 5 Companies Using AI for Clean Energy, Grid Reviews Cut From 45 Days to 2 Minutes — nordicinst · 2026-09-21
- Mozilla AI runs a local 30B model end-to-end to open a real bugfix PR, fully offline — mozilla-ai · 2026-09-21
- NVIDIA spotlights 5 AI clean-energy companies, grid review cut from 45 days to 2 minutes — NVIDIA Blog · 2026-09-21
- Cohere Labs launches Local AI community program for local inference and hardware tuning — Cohere_Labs · 2026-09-21