35B MoE at 20.5 tok/s on an iPhone: OpenBMB's Lookahead Expert Scheduling skips no experts
GasSmooth7439 · reddit · 2026-10-21
OpenBMB published an engineering write-up on running its 35B-A3B MoE model (MiniCPM-V 4.7) fully on-device — without shrinking the model or skipping experts — by attacking the real bottleneck: getting expert weights from flash storage to compute fast enough. Lookahead Expert Scheduling (LES) predicts which experts the next layer needs, prefetches their weights during current computation, and executes ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, it achieves:
- 20.51 tok/s on iPhone 18 Pro Max, 6.07× the 3.38 tok/s baseline
- 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks
- 3 GB peak memory in a low-memory configuration
The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni. Notably, LES hides memory bottlenecks without changing which experts execute, so the approach may generalize to other MoE models.
Related event: OpenBMB's LES Lets iPhones Run 35B MoE Models in Real Time(2 posts)→
More from Infra
- CoreWeave launches Physical AI Launchables: run robotics and simulation workloads with one command — ayushthakur0 · 2026-10-21
- NVIDIA locks up massive ABF substrate capacity again, cementing its win in the AI compute cycle — BenBajarin · 2026-10-21
- php2shell Flaws Hit ~70% of Websites; Cloudflare Shipped WAF Rules 9 Hours Early — Cloudflare Blog · 2026-10-21
- SemiAnalysis: 1,730 solar pullers yield ~30 GWp, only 4-7 GW of firm datacenter power — BenBajarin · 2026-10-21
- Frozen 16 MB vocabulary slashes small LLM size to 46 MB with no quality loss — pmttyji · 2026-10-21
- llampart v2: open-source local LLM chat UI with charts, Mermaid and MCP support — mossy_troll_84 · 2026-10-21