Qwen3.8-Flash-Next runs 1M context locally on M5 Max at 40 tok/s via MLX-serve
Beamsters · reddit · 2026-09-09
The co-creator of the new Qwen3.8-Flash-Next engine support in MLX-serve shares a local deployment achieving 1M-token context on an M5 Max 128GB:
- Uses 8-bit KV cache with mixed quantization (8-bit dense layers, 4-bit expert layers) to preserve quality
- Sustained generation through full 1M context: 40 tok/s on prose, 75 tok/s on coding
- Requires iogpu.wiredlimitmb=120000 for full 1M context; peak memory 117GB
- Author stresses real temp-1.0 sampling over deep context, not short-context benchmarks; the model also built the MLX Serve Monitor plugin shown in Opencode2's app
All resources open-sourced: the mlx-serve engine, model weights on HuggingFace, and the Opencode2 plugin. Bugs expected — reports welcome.
More from Infra
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26