FreeToken: Open-Source Engine Runs 290B+ MoE Models Locally on Gaming PCs, 13.8k Stars
solyarisoftware · x · 2026-09-26
FlashML-org released FreeToken, an open-source edge-native MoE serving engine (13.8k GitHub stars) that brings datacenter-scale model serving to desktops. It treats GPUs, CPUs, host memory, and interconnects as one elastic inference platform to run 290B+ open-weight frontier MoE models on consumer hardware at interactive speeds, using bandwidth-adaptive CPU–GPU co-execution (q policy), full-layer double-buffering and other edge optimizations. Ships with one-line install, benchmarks, docs, and a linked paper.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11