Forking vLLM With Custom Patches to Boost Gemma 4 31B Inference Speed
metalvendetta · reddit · 2026-09-20
A Reddit user documents their journey improving tokens-per-second for Gemma 4 31B: TPS optimization requires understanding the model architecture, so they forked vLLM and wrote their own patch to make a custom configuration work. The full article is written for beginners as a mental model for TPS optimization on any model.
The author argues that being a mere "mecha pilot" (tuning knobs from outside) isn't enough for inference optimization — you have to dive into the codebase and architecture.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04