Qwen3.8-Flash-Next runs 1M context locally on M5 Max at 40 tok/s via MLX-serve
Beamsters · reddit · 2026-09-09
The co-creator of the new Qwen3.8-Flash-Next engine support in MLX-serve shares a local deployment achieving 1M-token context on an M5 Max 128GB:
- Uses 8-bit KV cache with mixed quantization (8-bit dense layers, 4-bit expert layers) to preserve quality
- Sustained generation through full 1M context: 40 tok/s on prose, 75 tok/s on coding
- Requires iogpu.wiredlimitmb=120000 for full 1M context; peak memory 117GB
- Author stresses real temp-1.0 sampling over deep context, not short-context benchmarks; the model also built the MLX Serve Monitor plugin shown in Opencode2's app
All resources open-sourced: the mlx-serve engine, model weights on HuggingFace, and the Opencode2 plugin. Bugs expected — reports welcome.
More from Infra
- Cerebras paper: layer dropout saves up to 25% training FLOPs and yields 1.55x faster decoding — burny_tech · 2026-09-09
- Viettel unifies GPU fleet into Token-as-a-Service platform with three open source layers — PyTorch · 2026-09-09
- Put per-turn action schemas in the last user message to preserve prompt caching — Low_Bad_6585 · 2026-09-09
- minnow: An Open-Source Fast Inference Server for LLaDA2.2 Diffusion LMs — coder543 · 2026-09-09
- PyTorch launches Accelerator Integration WG to fix hardware fragmentation — PyTorch · 2026-09-09
- HiSparse hybrid sparse attention lands in vLLM: 8x H200 concurrency jumps from 5 to 25 at 1M context — eliebakouch · 2026-09-09