Open-source framework streams any MoE model on any device: 35B at 25 tok/s on a 16GB Mac
dai_app · reddit · 2026-10-24
A new open-source (Apache 2.0) framework built on llama.cpp runs large MoE models in any quantization via streaming across Android, iOS, macOS, Windows and Linux, with automatic CPU/GPU/NPU resource allocation to optimize tok/s. Measured results: Qwen 35B (Q4KM) at 25+ tok/s decode on a 16GB Mac, 6 tok/s on a 12GB Android phone, 3 tok/s on iOS with 300+ tok/s prefill on GPU/NPU.
More from Infra
- Muon optimizer may nearly halve LLM training compute, challenging a decade of AdamW — bycloud · 2026-10-24
- Infographic: why AI is an energy problem, from computation and cooling to the grid — CurieuxExplorer · 2026-10-24
- Engineer shares how he runs infra ops with Codex agents — kuza55 · 2026-10-24
- Running a 115B MoE with 262K context on an 8GB GPU at 70 T/s — jonathanx37 · 2026-10-24
- Physical AI Weekly: force-sensing gloves, Isaac Lab 3.0, humanoids in Audi plants — AIFlow_ML · 2026-10-24
- Chart: Hyperscalers' combined free cash flow projected to collapse near zero by 2026 — SumitGup · 2026-10-24