Running a 95.5GB Qwen Model on a 64GB Mac at 27 tok/s via Expert Streaming
SnooPredictions515 · reddit · 2026-09-19
A developer published a checkpoint and llama.cpp fork that runs Qwen3.8-Flash-Next (95.5 GiB) on a 64GB Mac by streaming MoE experts from SSD. Key tricks: direct file reads instead of mmap (prefill 181→401 tok/s), draft-head speculative decoding (18→27.6 tok/s), depth-3 + 0.3 confidence floor beating depth 4 by 10%, and a quant mix lifting perplexity to 4.31. Full setup commands in the repo.
More from Infra
- Google engineers: LLM benchmark harnesses silently drop requests — 200 QPS in, 38 out — AI Engineer · 2026-09-20
- The rig built to run Emacs and doomscroll X is now worth more than its owner's car — tetsuoai · 2026-09-19
- Apple M4 sustains 10 instructions per cycle, beating most rivals; M5 speedup explained — lemire · 2026-09-19
- Apple M6 bumps cores to 12 with two super cores; CPUs keep improving fast — lemire · 2026-09-19
- Apple M-series chips gained ~50% Geekbench 6 performance over three years — lemire · 2026-09-19
- Inside OpenAI's inference routing: why the proportional controller had to go — AI Engineer · 2026-09-19