Apple's MLX lands kernel optimizations: 2~3x batched decoding, ~2x small-prompt prefill
awnihannun · x · 2026-10-29
zcbenz summarized a batch of recent optimizations merged into Apple's MLX framework: sliding window attention built into fused SDPA (with array-mask support), mma/nax QMV making batched decoding 23x faster, direct fp4-to-fp16 dequantization for 1.4x faster prefill on older Macs, a 2-stage nax QMM kernel for 2x faster small-prompt prefill, 1-bit affine quantization, thin GEMM kernels (1.5x), and 1.4x faster fused SDPA via K/V prefetching. He recommends browsing MLX's commit history for aspiring open source contributors.
More from Infra
- Qwen3.8 Flash Next at IQ3_S hits 275 tok/s on a single RTX 5080 via Basalt fork — yehyakar · 2026-10-29
- Google Cloud CEO: Apple Scaling Foundation Model Pretraining on Latest TPUs — shayan_ · 2026-10-29
- FlashAttention vs standard attention in one chart: the bottleneck is data movement, not compute — cneuralnetwork · 2026-10-29
- Particle-1 inference goes live on OpenRouter, claiming 28-52% fewer output tokens — ycombinator · 2026-10-29
- Qwen3.8-27B EXL3 quant hits lowest KLD of 0.0002 in just 23 GB — suprjami · 2026-10-29
- MLX-Serve 26.11.1 rewrite: plugin architecture adds models and quants without recompiling — TheMoonMidas · 2026-10-29