Expert-lookahead delivers 10%+ speedup for MoE inference on low-memory Macs
carloslfu · reddit · 2026-09-15
Developer carloslfu implemented expert-lookahead for MoE models running via expert-offloading/SSD-streaming (slotstream) on low-memory Macs, gaining 10%+ performance on top of prior optimizations in Qwen 3.8 flash.
The initial idea was training a small model to predict experts N layers ahead; experiments showed the model itself is the best predictor. The final setup uses N=2: while current computation runs, slotstream runs the router 2 layers ahead on the current hidden state to prefetch the next experts into memory.
A small correction model trained on top adds another 3-4%. Docs are open-sourced in the slotstream GitHub repo.
More from Infra
- Helion closes oversubscribed $500M Series G for fusion energy — ycombinator · 2026-09-16
- nginx 1.31.6 Patches Heap Buffer Overflow in HTTP/3 (CVE-2026-90439) — jedisct1 · 2026-09-16
- AWS Trainium Runs PyTorch Natively, PyTorch Con Keynote Revealed — PyTorch · 2026-09-16
- A long-form explainer on why local inference matters — and why you need uncensored models — HankYeomans · 2026-09-15
- DGX Spark + TRELLIS.2 turns pencil sketches into 3D GLBs fully locally — jasonkneen · 2026-09-15
- Skipping a second RTX 5090 for two more Sparks: local inference user explains why — ideamaker321 · 2026-09-15