One-line MLX fix makes long-context inference up to 10.1x faster on M5 Max

SumitGup · x · 2026-11-23

A one-line change in Apple's MLX made long-context inference up to 10.1x faster on M5 Max. The culprit: a hardcoded TK=4 in the attention Metal kernel sdpavector.h, tuned for M1 Pro float16 with 128-dim heads. Newer models like DeepSeek MLA and Gemma 4 use 512-dim heads, causing register spilling. The fix—setting TK to 2 when combined dims exceed 512—cut 60K-context BF16 latency from 14.8ms to 1.46ms (FP32 2.4x, FP16 1.16x). No hardware, model, or compiler changes involved.

Original post →

More from Infra

Infra channel →