kapitsa: hand-written Metal inference engine hits 140 tok/s on Qwen3.8-27B for local OpenAI/Anthropic APIs
DotaMate · reddit · 2026-10-24
kapitsa is an open-source, from-scratch C++20/Metal inference engine for Apple Silicon (no MLX, no llama.cpp) that serves both OpenAI- and Anthropic-compatible local APIs. Its kflash speculative decoding verifies a best-first tree of draft candidates in one 8-row pass—bit-identical to plain decoding—reaching 140 tok/s on Qwen3.8-27B 4bit (M5 Max), 81 on M5 Pro, 63 on M4 Pro, well above Ollama/MLX/llama.cpp in like-for-like tests. Concurrent requests share the verify pass (+23% throughput at 3x), prefix caching extends to SSD with --disk-cache, and every default change must pass a logit-equivalence check. Install via brew; 32GB for the 27B, 16GB falls back to 9b.
More from Infra
- Miles v0.1.3 cuts first RL weight update from 99.5s to 3.9s, shares GPUs across train/rollout — hsu_byron · 2026-10-24
- 98%+ Tensor Core utilization shown in LLM training run, drawing admiration — gharik · 2026-10-24
- We spent millions on AI apps with zero revenue — but our old GPUs are now worth more than we paid — stephen280up · 2026-10-24
- Polymarket puts 24% odds on AI data centers launching to space by end of next year — Polymarket · 2026-10-24
- Polymarket jokes wolves are entering Finnish villages after Google cleared forest for AI data centers — Polymarket · 2026-10-24
- llama.cpp PR drops kq_mask for QSA selection, saving gigabytes of VRAM on Qwen Flash Next — jacek2023 · 2026-10-24