MLX MTP head silently ignored: a 3-line fix boosts Mac local decode speed by up to 79%
Micha0827 · reddit · 2026-09-23
A Reddit user benchmarking mlx-serve vs LM Studio on an M2 Max found a silent gotcha: mlx-serve expects MTP speculative decoding head tensors prefixed with mtp., but the HF-published mlx-community/Qwen3.8-27B-MTP-4bit weights ship without the prefix — and the server falls back to Prompt Lookup Decoding with no error or warning.
How to check: look at the spec-stats line, not the startup flag. mode=pld means the MTP head isn't loaded; you want mode=mtp attempts=31 accepts=67 perdraftpct=69.8%. The weights also must sit at mtp/weights.safetensors in the model dir (a symlink works).
The fix is three lines: re-save the safetensors with an mtp. prefix added to every key.
Measured gains (Qwen3.8-27B 4-bit, median of 5 runs, unique prefix per prompt): with MTP loaded, decode hits 13.0/20.1 tok/s (short/long) vs 8.0/11.2 without speculation; LM Studio scores 11.8/11.6 — that's +63%/+79% decode. Notably, LM Studio is faster without speculation, so silently losing MTP is expensive.
Limitation: MoE MTP heads fail to load entirely (MlxError even with correct prefix) — Qwen3.6-35B-A3B doesn't work while dense 27B does; issue filed. Environment: mlx-serve 26.9.5, MLX 0.32.2, macOS 26.6.2, M2 Max 96 GB.
More from Infra
- Leaked Alibaba roadmap: Qwen 5 to scale to 5-10 trillion parameters — burny_tech · 2026-09-23
- Goldman: US to Outspend China $806B to $110B on AI Infra in 2026, China Fights Back on Efficiency — FinanceYF5 · 2026-09-23
- Nokia Signs Four Cloud Data Deals in 90 Days, But Its AI-RAN Bet Rests on One Chipmaker — shashib · 2026-09-23
- China Weighs Curbs on Broadcom Switches Behind Up to 90% of State Data Centers — rohanpaul_ai · 2026-09-23
- tinygrad hits ~200 tok/s MiMo-V2.6-Pro on MI300X, brought up via GLM-5.3 — AIFlow_ML · 2026-09-23
- Rumors: 64GB+ VRAM RTX 5090 in R&D but not coming anytime soon — AIFlow_ML · 2026-09-23