Maple prunes MoE below 2 BPW to run Gemma 4 26B at 45 TPS on a 6GB GPU
AmSach · hn · 2026-11-08
Maple is a Show HN project offering sub-2-BPW MoE pruning that squeezes a 26B-class Gemma 4 model onto a 6GB GPU at roughly 45 tokens/s, aiming to make large MoE models runnable locally on consumer hardware. Open-sourced on GitHub.
More from Infra
- Batch APIs: the 50% LLM discount only pays off after model choice and prompt caching — arpit_bhayani · 2026-11-13
- RTX Pro 6000 bricks mid-Qwen run, owner jokes it "died of a broken heart" over Jensen news — wine_dark · 2026-11-13
- Bittensor's Chutes Handles 256 Concurrent Requests on a Single RTX 5090 at ~18k tok/s — markjeffrey · 2026-11-13
- Google expands TPU v8 lineup: 8tx cuts latency 2x, 8i Max doubles silicon per rack — Beth_Kindig · 2026-11-13
- Horace He to share two mental models for reasoning about inference at SGLang Summit — cHHillee · 2026-11-13
- Used A100 prices fall another 20-30% as sellers rush ahead of a collapse — Bascilian · 2026-11-13