ModelBest's LES runs a 35B MoE on-device at 20 tok/s, 6x the baseline
面壁智能 · wechat · 2026-10-20
ModelBest details LES (Lookahead Expert Scheduling), the inference system behind MiniCPM-V 4.7 that runs a 35B-A3B MoE losslessly on-device: over 20 tok/s on iPhone 18 Pro Max, 18 tok/s on M4 iPad.
- Expert weights live on flash and load on demand; LES reuses the current layer's Router input to predict the next layer's experts (80% recall) and prefetches, keeping prediction separate from real routing; ready experts compute first.
- Storage: per-expert contiguous GGUF layout plus mixed-precision quantization (experts at Q3K) cuts weights 26.7% (19.7→14.44GB) with only a 0.16 benchmark drop.
- Compute: DSpark speculative decoding adapted to llama.cpp (PR merged upstream) batches token verification.
Ablation: 3.38 tok/s baseline → 6.57 (scheduling) → 11.34 (storage) → 17.16 (quant) → 20.51 tok/s (DSpark), 6.07x total. Open-sourced at GitHub tc-mb/llama.cpp-omni; the approach generalizes to other MoE models.
Related event: OpenBMB's LES Lets iPhones Run 35B MoE Models in Real Time(2 posts)→
More from Infra
- NVIDIA pivots to local AI: DGX Spark clusters to 8 units with 1TB unified memory — ryanshrout · 2026-10-21
- Daytona raises Series B to be the execution layer for AI agents; Meta ran Llama post-training on it — mattturck · 2026-10-21
- Anthropic cofounder: cut token costs 1000x and enterprises will just use 1000x more tokens — gabriel1 · 2026-10-21
- China shipped 1,730 crystal pullers to US in a quarter as Musk plots 20 GW buildout — rickasaurus · 2026-10-21
- Nvidia unveils plug-in DSX Pod data center at GTC Berlin to fix Europe's AI infrastructure lag — Wired AI · 2026-10-21
- Google Cloud now serves OpenAI, Anthropic, and Google frontier LLMs with hot-swapping — rseroter · 2026-10-21