UC Berkeley and MIT Open-Source FreeToken: Frontier MoE Models on Consumer PCs
UC Berkeley Sky Lab, together with MIT researchers, open-sourced FreeToken, an edge-oriented MoE inference engine (GitHub org FlashML-org, paper arXiv:2608.16157) that lets consumer PCs run frontier open-source MoE models locally without large VRAM. Official and early third-party results point in the same direction, and community developers have already begun contributing platform ports such as Apple Silicon/MLX support.
Confirmed
- The launch post came from @YuchenjUW (UC Berkeley Sky Lab); @gnukeith, @solyarisoftware and Synced report the project was jointly open-sourced by UC Berkeley and MIT researchers.
- Core techniques: bandwidth-adaptive CPU-GPU execution, treating CPU/GPU/memory as one elastic system with dynamic expert migration, and semantic caching across agent turns; @markk's paper walkthrough adds that the full-stack co-design targets two realities of local AI—constantly shifting agent workload execution patterns and heterogeneous, imbalanced edge hardware.
- Official numbers: roughly 3–4x decode and 6–30x prefill speedups over Ollama; a single RTX PRO 6000 workstation runs the 753B GLM-5.2 at 14.9 tok/s; an 8GB RTX 4060 laptop runs Qwen3.6 35B at 39 tok/s (39.3 per Synced); a single RTX 5090 runs DeepSeek-V4-Flash 284B at 25 tok/s, 1.46x faster than llama.cpp.
- Community tests: @ViRROOO's early local test (RTX 5080 16GB + 64GB DDR6) reports 100 tok/s for a 35B model on a 4090-class GPU; @jasonkneen submitted an Apple Silicon/Metal/MLX support PR, showing third-party platform contributions have begun.
- @gnukeith adds that an ordinary gaming PC (e.g. 5090 + large system RAM, no big VRAM needed) can run 290B+ parameter MoE models at interactive speeds.
Unconfirmed
- Benchmarks mostly come from official and a few preliminary third-party tests; @ViRROOO notes results are early, and independent replication across hardware configs (especially non-NVIDIA) and long-term stability remain to be verified.
Why it matters
- FreeToken lowers the barrier to running frontier open MoE models locally to a budget 4060 laptop or a gaming PC; @berkeleyai even claims optimized frontend model serving costs can drop to zero.
- It matters directly for privacy-sensitive, offline and cost-constrained use cases (personal agents, edge deployment); aligned third-party tests and voluntary platform contributions add credibility, though broader independent replication is still needed.
2026-08-22 ~ 2026-08-23 · 11 related posts
- Episode 1: UC Berkeley and MIT Open-Source FreeToken: Frontier MoE Models on Consumer PCs(2026-08-22, 11 posts)
- Episode 2: FreeToken Lets 8GB GPUs Run 35B MoE Models On-Device(2026-08-29, 2 posts)
Primary sources
- [source] Berkeley open-sources FreeToken: run 35B models on a $1,000 RTX 4060 laptop — Yuchenj_UW · 2026-08-22
- Berkeley/MIT Open Source Inference Engine: RTX 5090 Runs 284B Models at 25 tok/s — gnukeith · 2026-08-22
- FreeToken: run 290B+ MoE models locally on a gaming PC, no big VRAM needed — gnukeith · 2026-08-22
- FreeToken: 4x Faster Decode, Enables 284B Models on Gaming Desktops — airesearch12 · 2026-08-22
- [source] FreeToken Tested: 100 tok/s Running a 35B Model on Consumer Hardware — ViRROOO · 2026-08-22
- Frontier Models on Consumer GPUs: Qwen3.6 on 8GB VRAM at 39 tok/s — berkeley_ai · 2026-08-22
- FreeToken: Run 284B frontier models on consumer GPUs at interactive speeds — solyarisoftware · 2026-08-22
- FreeToken System Enables RTX4060 Laptops to Run 35B Models Smoothly — 机器之心 · 2026-08-22
- FreeToken Enables 35B Models on 8GB Laptops via Unified CPU-GPU Memory — mark_k · 2026-08-23
- FreeToken Paper: Edge-Native MoE Serving with Bandwidth-Adaptive Execution — mark_k · 2026-08-23
- [source] FreeToken benchmarks: a single RTX PRO 6000 runs a 753B model at 14.9 tok/s — jasonkneen · 2026-08-23