Qwen3.8-Flash-Next runs 1M context locally on M5 Max at 40 tok/s via MLX-serve

Beamsters · reddit · 2026-09-09

The co-creator of the new Qwen3.8-Flash-Next engine support in MLX-serve shares a local deployment achieving 1M-token context on an M5 Max 128GB:

• Uses 8-bit KV cache with mixed quantization (8-bit dense layers, 4-bit expert layers) to preserve quality
• Sustained generation through full 1M context: 40 tok/s on prose, 75 tok/s on coding
• Requires iogpu.wiredlimitmb=120000 for full 1M context; peak memory 117GB
• Author stresses real temp-1.0 sampling over deep context, not short-context benchmarks; the model also built the MLX Serve Monitor plugin shown in Opencode2's app

All resources open-sourced: the mlx-serve engine, model weights on HuggingFace, and the Opencode2 plugin. Bugs expected — reports welcome.

Original post →

More from Infra

• Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
• Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
• Anthropic S-1: CSP marketplace revenue jumped from 11% to 48% in three years — AccBalanced · 2026-10-27
• Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
• Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
• Alchemy Cuts Cloudflare Bindings CPU Time by Up to 166× Via Profiling — samgoodwin89 · 2026-10-27

Infra channel →