MLX-Serve v26.9.6: M5 Ultra optimizations push Qwen3.8 decode to 300+ tok/s locally
TheMoonMidas · x · 2026-09-26
ddalcu released mlx-serve v26.9.6 (renamed from MLX Core), with major local inference optimizations targeting M5 Ultra.
Key updates:
- M5 Ultra: Qwen3.8 Flash Next reaches 243-300+ tok/s decode and 3,362 tok/s prefill; fused MoE kernel adds +14% on verification, and fused multi-stream verification adds +13% at 4 streams
- Laya typed decisions: new POST /v1/decisions endpoint with local-JEV support
- MTP drafts from the conversation itself when replies copy it, speeding file rewrites/edits by 1.3-1.5x (on by default; disable with MLXSERVEMTPLOOKUP=0)
- GDN layers now run as one GPU dispatch, helping every Mac a few percent
- Fixed media and preservethinking handling, UI updates
Peak on M5 Ultra with Flash Next mixed 4-8 bit and --mtp: 219 tok/s decode, 227 tok/s across 4 streams, 3,150 tok/s prefill on an 8k prompt.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11