Open-source kapitsa engine boosts Mac local agent throughput 23% with tree speculative decoding

DotaMate · reddit · 2026-10-24

kapitsa is a new open-source inference engine for Macs that turns speculative decoding into a tree: a drafter guesses a whole tree of tokens and the model verifies it in a single pass, which concurrent requests can share.
With three concurrent agent requests on qwen3.8-27b, total throughput improved up to +23%; a new request sharing the system prompt starts from cache, cutting time-to-first-token from 1.9s to 0.06s (on the 4b).
Single-stream speeds: 63 tok/s on M4 Pro, 140 tok/s on M5 Max. It exposes OpenAI- and Anthropic-compatible APIs on localhost, so any framework with a base URL works, and kapitsa claude --model 27b runs Claude Code against the local model.
Install via Homebrew; requires Apple Silicon (M1+) with 32GB for the 27b (16GB can run the 9b).

More from coding & agent