Kapitsa runs Hy4-flash on M5 Max at 36 tok/s decode, 971 tok/s prefill at 32K

m4rkGS · reddit · 2026-11-21

Kapitsa, an open-source Apple silicon inference engine, adds Hy4-flash support with benchmarks on a MacBook Pro M5 Max (40-core GPU, 128GB):

Decode (4-bit Kapitsa vs 3.1-bit MLX): 36.4 vs 20.8 tok/s short; 35.1 vs 7.0 at 16K; 34.6 vs 4.4 at 32K — nearly 8x faster at long context.

Prefill: with --profile fast, 971 tok/s across 4K/16K/32K prompts (32K first token in 33s), vs 300 tok/s and 105s on MLX.

Quant: Hy4-flash-MLX-4bit — 8-bit trunk, 4-bit routed experts (group 64), 134GB on disk; all experts stay in RAM on 128GB (88GB peak) with memory-mapped Engram tables read on CPU, which is the key to speed. A 48GB M5 Pro streams experts from SSD at 3.5 tok/s.

Ships OpenAI/Anthropic-compatible APIs, tool calls, thinking toggle; warm cached turns start in 0.15–0.21s at any context length. Install via brew install nace-ai/tap/kapitsa. The repo banner shows Kapitsa hitting 161 tok/s on Qwen3.6 35B writes vs llama.cpp 69 and Ollama 89.

Original post →

More from Infra

Infra channel →