Open-source Kapitsa hits 960+ tok/s prefill on M5 Max with 4-bit Hy4-flash
0zyman23 · reddit · 2026-11-23
A developer added Hy4-flash support to Kapitsa, an open-source Mac inference engine, and benchmarked the 4-bit pack (8-bit main network, 4-bit routed experts) on an M5 Max with 40-core GPU and 128GB RAM. Results: up to 968-971 tok/s prefill on the fast profile, 677 tok/s / 47.5s at 32K context, and steady 35 tok/s decode — far outpacing mlx-lm's 3.1-bit pack (304 tok/s prefill at 32K, which also OOM'd at 4-bit). Peak memory 88.3GB, full model in RAM, only the Engram on SSD; cold prompts, thinking off, temperature 0.6. Install via brew, weights on Hugging Face, raw bench files open-sourced. "Prefill is king these days."
More from Infra
- Alabama Power to cut rates $160M thanks to data center revenue — binarybits · 2026-11-23
- MLX-Serve adds Inkling 2 Mini with ~24 custom kernels, fastest Mac inference yet — SumitGup · 2026-11-23
- One-line MLX fix makes long-context inference up to 10.1x faster on M5 Max — SumitGup · 2026-11-23
- Anthropic's $240B cash pile will go to semiconductors as RSI models demand compute — firstadopter · 2026-11-23
- Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost — eltokh7 · 2026-11-23
- TechCrunch talks to Paris Marx on "Hyperscale" and the data center backlash — TechCrunch AI · 2026-11-23