Dev runs 125B Qwen model locally on M3 Max at 70 tok/s via MLX
mayfer · x · 2026-08-28
A developer verified the native performance of Qwen 3.8 Flash Next (125B-A6B) on Apple Silicon using mlx-serve. On an M3 Max (128GB), the setup achieved 70 tok/s serial decoding and 300 tok/s prefill, successfully handling 32k token tasks with 20GB memory remaining.
The implementation uses Zig + Metal without Python. Key optimizations include hyper-connection residual streams, n-gram embeddings gathered directly from disk, sparse attention past 2k tokens, and 5 new Metal kernels. The model also supports image and video input.
Related event: M3 Max Runs 125B Qwen Model Locally at 70 tok/s(2 posts)→
More from Infra
- Debunking AI data center myths: Water usage and grid impact — robleclerc · 2026-08-28
- DFlash 2 Introduces Block-Diffusion Speculative Decoding to Speed Up GLM-5.3-Flash — songhan_mit · 2026-08-28
- NVIDIA launches Mesh open compute network to aggregate idle GPUs for AI — nvidia · 2026-08-28
- AI automated research finds numerical bug in vLLM/SGLang backend — josh_tobin_ · 2026-08-28
- Hot Chips 2026: inference chips enter an "era of ferment" with divergent bets — BenBajarin · 2026-08-28
- Testing Muon Optimizer: Smoother Gradients and Stable Residual Maxima — stochasticchasm · 2026-08-28