Kapitsa inference engine hits 346 tok/s on Qwen 35B, runs 180B via SSD offload on M5 Max
DotaMate · reddit · 2026-11-02
Kapitsa, an open-source inference engine optimized for Apple Silicon with hand-written Metal kernels, released v0.2.26 with benchmarks on a 2026 MacBook Pro M5 Max (128GB).
Key results (decode / 4K prefill):
- Qwen3.6 35B-A3B: 346 tok/s, 6,036 tok/s prefill
- Clover 35B-A3B (block-diffusion MoE): 304 tok/s
- Recursive Flint 40B: 265 tok/s
- Nemotron Bolt 74B: 154 tok/s
- Qwen3.8 27B: 143 tok/s
- Qwen Flash-Next 180B: 108 tok/s, or 61 tok/s on a 48GB MacBook via SSD offloading of experts
Against competing engines: 1.23x faster than Splash on the 35B, 2.2x over oMLX on Bolt, and the only engine running Clover on Mac. The 35B still writes 175 tok/s at 32K context.
Why it's fast: KFlash speculative decoding verifies a tree of drafter guesses in one pass instead of a chain, nearly free on Apple GPUs' 8-row tiles, without changing outputs. kapitsa serve exposes OpenAI/Anthropic APIs with streaming, tool calls and JSON-schema output; cached follow-up turns start in 0.04–0.13s. Community benchmark submissions go in via GitHub PR with kapitsa bench --submit.
Related event: Kapitsa 0.2.26 Benchmarks: Qwen 35B Hits 346 tok/s on M5 Max(2 posts)→
More from Infra
- Viral space datacenter story called clickbait: quote is about launch costs, up to 100x Earth build cost — ns123abc · 2026-11-02
- Five-year compute map: NVIDIA holds 93% of GPU units, custom ASICs to pass GPUs by 2027 — pdamodaran · 2026-11-02
- Space datacenter heat is just engineering; cosmic rays are the real challenge — shashib · 2026-11-02
- Louisiana's $300B+ project pipeline includes Meta Hyperion, Amazon and Hut 8 AI data centers — MatthewChang · 2026-11-02
- AI data centers destroy 250,000 trees, undercutting 'paperless' eco promises — im_mansigupta · 2026-11-02
- "Someone must solve the cost of serving coworking agents" — hyhieu226 · 2026-11-02