Qwen3.8-Next hits 150 tps prefill on M5 Air via 3bit MoE streaming
maddie-lovelace · reddit · 2026-08-29
The author adapted their DeepSeek v4 MoE streaming stack to Qwen3.8-Next with better-than-expected results: on an M5 Air in low power mode, a 3bit quant (Qwen3.8-Flash-Next-MLX-oQ3-MTP) achieves 150 tps prefill and 3.6 tps decode on a 2k-token prompt.
For comparison, a dense 27b 4bit build on the same machine gets 70 tps prefill and 3 tps decode — a notable edge for MoE on-device, though quantization levels differ so it isn't apples-to-apples.
More from Infra
- Intel said to have secured foundry commitments from Broadcom and Google, Apple wins on 14A — BenBajarin · 2026-10-15
- Magnitude open-source inference engine beats Ollama by over 2x with auto-tuned kernels — nickbaumann_ · 2026-10-15
- Nvidia to announce $28B in AI investments in Europe this week, report says — georgemillo · 2026-10-15
- A $10,811 Cloudflare Bill From an Unused AI-Written Test Project: Refunds Aren't a Safety Net — sujingshen · 2026-10-15
- Retraining-Free Method Turns Transformers Into Elastic Models Adapting to Any Compute Budget — VISLab-Amsterdam · 2026-10-15
- vLLM's MTP3 validation: AIME25 up to 0.9667, prefill kernels up to 84% lower latency — vllm_project · 2026-10-15