MoEspresso runs Qwen3.8-Flash-Next (125B) at 12-15 tok/s on a 2021 32GB M1 Max
marcobaldo · reddit · 2026-09-28
The author of MoEspresso, a hobby inference engine, released its third iteration: it runs the 125B Qwen3.8-Flash-Next locally on a 2021 32GB M1 Max at 12-15 decode tokens per second.
Key techniques:
- Cache-Prior expert biasing: during decode, experts already resident in memory get a bias while the model's two strongest experts are always kept; prefill stays unbiased. Inspired by Apple's AFM 3 pruning work and a Cache-Prior paper.
- IQ2K quantization for routed projections via the custom mlx-iqk library on Metal — unsupported by standard MLX or mainline llama.cpp. KVarN K4/V4 preserves memory for resident experts as context grows.
- Automatic SSD streaming and Cache-Prior defaults when experts can't all fit.
On a reproducible 48-question benchmark, local Qwen3.8 Flash scored 84.3%, beating GPT-6 Luna xhigh (81.4) and Claude Opus 4.8 xhigh (80.3), trailing hosted Qwen3.8 Flash (89.7) and GPT-6 Sol medium (89.2).
Install via Homebrew; code is open source, with Linux and AMD Strix Halo support planned.
More from Infra
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16