Running a 95.5GB Qwen Model on a 64GB Mac at 27 tok/s via Expert Streaming
SnooPredictions515 · reddit · 2026-09-19
A developer published a checkpoint and llama.cpp fork that runs Qwen3.8-Flash-Next (95.5 GiB) on a 64GB Mac by streaming MoE experts from SSD. Key tricks: direct file reads instead of mmap (prefill 181→401 tok/s), draft-head speculative decoding (18→27.6 tok/s), depth-3 + 0.3 confidence floor beating depth 4 by 10%, and a quant mix lifting perplexity to 4.31. Full setup commands in the repo.
More from Infra
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04