MoEspresso runs Qwen3.8-Flash-Next (125B) at 12-15 tok/s on a 2021 32GB M1 Max
marcobaldo · reddit · 2026-09-28
The author of MoEspresso, a hobby inference engine, released its third iteration: it runs the 125B Qwen3.8-Flash-Next locally on a 2021 32GB M1 Max at 12-15 decode tokens per second.
Key techniques:
- Cache-Prior expert biasing: during decode, experts already resident in memory get a bias while the model's two strongest experts are always kept; prefill stays unbiased. Inspired by Apple's AFM 3 pruning work and a Cache-Prior paper.
- IQ2K quantization for routed projections via the custom mlx-iqk library on Metal — unsupported by standard MLX or mainline llama.cpp. KVarN K4/V4 preserves memory for resident experts as context grows.
- Automatic SSD streaming and Cache-Prior defaults when experts can't all fit.
On a reproducible 48-question benchmark, local Qwen3.8 Flash scored 84.3%, beating GPT-6 Luna xhigh (81.4) and Claude Opus 4.8 xhigh (80.3), trailing hosted Qwen3.8 Flash (89.7) and GPT-6 Sol medium (89.2).
Install via Homebrew; code is open source, with Linux and AMD Strix Halo support planned.
More from Infra
- Nvidia targets 500,000 RTX Pro 5500 chips per quarter for China; ByteDance order alone takes two quarters — pstAsiatech · 2026-09-28
- Nvidia says China's chip makers saw 'unprecedented growth' since 2022 amid outdated US export controls — pstAsiatech · 2026-09-28
- From 16GB to 128GB: A Local AI Hardware Journey — TheOyinbooke · 2026-09-28
- Liquid AI makes the case for edge agents: 350M-2.6B models with no per-token bill — JosephJacks_ · 2026-09-28
- Dev runs Bonsai 2 locally on Mac Studio to diff medication lists in 5.75 seconds — MaziyarPanahi · 2026-09-28
- $100 used Tesla P100 beats gaming GPU nearly 2x for local LLM inference — Mrinohk · 2026-09-28