Open-source Kapitsa hits 960+ tok/s prefill on M5 Max with 4-bit Hy4-flash

0zyman23 · reddit · 2026-11-23

A developer added Hy4-flash support to Kapitsa, an open-source Mac inference engine, and benchmarked the 4-bit pack (8-bit main network, 4-bit routed experts) on an M5 Max with 40-core GPU and 128GB RAM. Results: up to 968-971 tok/s prefill on the fast profile, 677 tok/s / 47.5s at 32K context, and steady 35 tok/s decode — far outpacing mlx-lm's 3.1-bit pack (304 tok/s prefill at 32K, which also OOM'd at 4-bit). Peak memory 88.3GB, full model in RAM, only the Engram on SSD; cold prompts, thinking off, temperature 0.6. Install via brew, weights on Hugging Face, raw bench files open-sourced. "Prefill is king these days."

Original post →

More from Infra

Infra channel →