Kapitsa hits 346 tok/s on a 35B MoE and runs a 180B model on 48GB Macs via SSD offloading
DotaMate · reddit · 2026-11-02
The open-source Apple Silicon inference engine Kapitsa (Apache 2.0, v0.2.26) published M5 Max benchmarks across six models with hand-written Metal kernels.
Key numbers (512-token decode / 4K prefill, median of 3):
- Qwen3.6 35B-A3B: 346 tok/s decode, 6,036 tok/s prefill
- Clover 35B-A3B: 304; Flint 40B: 265; Nemotron Bolt 74B: 154; Qwen3.8 27B: 143
- Qwen Flash-Next 180B: 108 tok/s, only dropping 90→78 at 32K context
Highlights:
- SSD offloading streams non-resident experts from disk: Flash-Next 180B runs at 61 tok/s on a 48GB MacBook Pro
- vs. the fastest competing engine per model: 1.23x over Splash on the 35B, 2.2x over oMLX on Bolt; Clover runs nowhere else on Mac
- KFlash speculative decoding checks a tree of drafts instead of a chain — nearly free on Apple GPUs' 8-row tiles, output unchanged
- One-line brew install; serves OpenAI/Anthropic-compatible local APIs; community benchmark submissions via PR (2KB anonymous data)
More from Infra
- Google's TPU economics: Gemini 3.1 Pro costs 76% less per task than Claude Sonnet 5.5 — Beth_Kindig · 2026-11-02
- Users Say OpenAI and Anthropic's Usage Cuts Are Pushing Them to Local Models — SEND_ME_YOUR_ASSPICS · 2026-11-02
- Spot GPU prices to hit 3-5x contract rates, low-price compute sellers biggest losers — bindureddy · 2026-11-02
- Rumor: Musk to renegotiate TeraFab terms with TSMC, dropping self-build insistence — SumitGup · 2026-11-02
- Crusoe CEO says neocloud debt fears overblown amid 'tidal wave' of compute demand — nathanbenaich · 2026-11-02
- Texas Mega-Project Pipeline Tops $500B, Led by $100B Stargate AI Hub — MatthewChang · 2026-11-02