Kapitsa runs Hy4-flash on M5 Max at 36 tok/s decode, 971 tok/s prefill at 32K
m4rkGS · reddit · 2026-11-21
Kapitsa, an open-source Apple silicon inference engine, adds Hy4-flash support with benchmarks on a MacBook Pro M5 Max (40-core GPU, 128GB):
Decode (4-bit Kapitsa vs 3.1-bit MLX): 36.4 vs 20.8 tok/s short; 35.1 vs 7.0 at 16K; 34.6 vs 4.4 at 32K — nearly 8x faster at long context.
Prefill: with --profile fast, 971 tok/s across 4K/16K/32K prompts (32K first token in 33s), vs 300 tok/s and 105s on MLX.
Quant: Hy4-flash-MLX-4bit — 8-bit trunk, 4-bit routed experts (group 64), 134GB on disk; all experts stay in RAM on 128GB (88GB peak) with memory-mapped Engram tables read on CPU, which is the key to speed. A 48GB M5 Pro streams experts from SSD at 3.5 tok/s.
Ships OpenAI/Anthropic-compatible APIs, tool calls, thinking toggle; warm cached turns start in 0.15–0.21s at any context length. Install via brew install nace-ai/tap/kapitsa. The repo banner shows Kapitsa hitting 161 tok/s on Qwen3.6 35B writes vs llama.cpp 69 and Ollama 89.
More from Infra
- Cerebras deep dive: why CPUs, GPUs, TPUs and wafer-scale chips suit LLM workloads differently — sarahookr · 2026-11-21
- Broadcom plunges 9% as Google reportedly cuts 1M AI chips from 2027 plans — PAstynome · 2026-11-21
- Commerce Dept approved 800,000+ advanced AI chip sales to Chinese firms, some military-linked — ylecun · 2026-11-21
- Codex raises WebSocket request limit 8x to 128MB, making large turns 5x faster — charliermarsh · 2026-11-21
- Neon boosts Antigravity cache hit rate ~4%, raising quota effectiveness by at least 20% — cgarciae88 · 2026-11-21
- Epoch AI: trade data consistent with $3B+ of AI chips smuggled to China via Malaysia — Jsevillamol · 2026-11-21