One-line MLX fix makes long-context inference up to 10.1x faster on M5 Max
SumitGup · x · 2026-11-23
A one-line change in Apple's MLX made long-context inference up to 10.1x faster on M5 Max. The culprit: a hardcoded TK=4 in the attention Metal kernel sdpavector.h, tuned for M1 Pro float16 with 128-dim heads. Newer models like DeepSeek MLA and Gemma 4 use 512-dim heads, causing register spilling. The fix—setting TK to 2 when combined dims exceed 512—cut 60K-context BF16 latency from 14.8ms to 1.46ms (FP32 2.4x, FP16 1.16x). No hardware, model, or compiler changes involved.
More from Infra
- Open-source Kapitsa hits 960+ tok/s prefill on M5 Max with 4-bit Hy4-flash — 0zyman23 · 2026-11-23
- Alabama Power to cut rates $160M thanks to data center revenue — binarybits · 2026-11-23
- MLX-Serve adds Inkling 2 Mini with ~24 custom kernels, fastest Mac inference yet — SumitGup · 2026-11-23
- Anthropic's $240B cash pile will go to semiconductors as RSI models demand compute — firstadopter · 2026-11-23
- Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost — eltokh7 · 2026-11-23
- TechCrunch talks to Paris Marx on "Hyperscale" and the data center backlash — TechCrunch AI · 2026-11-23