FreeToken: Running 35B Models on Consumer GPUs

Episode 1 · UC Berkeley and MIT Open-Source FreeToken: Frontier MoE Models on Consumer PCs (2026-08-22, 11 posts)

UC Berkeley Sky Lab, together with MIT researchers, open-sourced FreeToken, an edge-oriented MoE inference engine (GitHub org FlashML-org, paper arXiv:2608.16157) that lets consumer PCs run frontier open-source MoE models locally without large VRAM. Official and early third-party results point in the same direction, and community developers have already begun contributing platform ports such as Apple Silicon/MLX support.

Confirmed

• The launch post came from @YuchenjUW (UC Berkeley Sky Lab); @gnukeith, @solyarisoftware and Synced report the project was jointly open-sourced by UC Berkeley and MIT researchers.
• Core techniques: bandwidth-adaptive CPU-GPU execution, treating CPU/GPU/memory as one elastic system with dynamic expert migration, and semantic caching across agent turns; @markk's paper walkthrough adds that the full-stack co-design targets two realities of local AI—constantly shifting agent workload execution patterns and heterogeneous, imbalanced edge hardware.
• Official numbers: roughly 3–4x decode and 6–30x prefill speedups over Ollama; a single RTX PRO 6000 workstation runs the 753B GLM-5.2 at 14.9 tok/s; an 8GB RTX 4060 laptop runs Qwen3.6 35B at 39 tok/s (39.3 per Synced); a single RTX 5090 runs DeepSeek-V4-Flash 284B at 25 tok/s, 1.46x faster than llama.cpp.
• Community tests: @ViRROOO's early local test (RTX 5080 16GB + 64GB DDR6) reports 100 tok/s for a 35B model on a 4090-class GPU; @jasonkneen submitted an Apple Silicon/Metal/MLX support PR, showing third-party platform contributions have begun.
• @gnukeith adds that an ordinary gaming PC (e.g. 5090 + large system RAM, no big VRAM needed) can run 290B+ parameter MoE models at interactive speeds.

Unconfirmed

• Benchmarks mostly come from official and a few preliminary third-party tests; @ViRROOO notes results are early, and independent replication across hardware configs (especially non-NVIDIA) and long-term stability remain to be verified.

Why it matters

• FreeToken lowers the barrier to running frontier open MoE models locally to a budget 4060 laptop or a gaming PC; @berkeleyai even claims optimized frontend model serving costs can drop to zero.
• It matters directly for privacy-sensitive, offline and cost-constrained use cases (personal agents, edge deployment); aligned third-party tests and voluntary platform contributions add credibility, though broader independent replication is still needed.

• Berkeley open-sources FreeToken: run 35B models on a $1,000 RTX 4060 laptop — Yuchenj_UW · 2026-08-22
• Berkeley/MIT Open Source Inference Engine: RTX 5090 Runs 284B Models at 25 tok/s — gnukeith · 2026-08-22
• FreeToken: run 290B+ MoE models locally on a gaming PC, no big VRAM needed — gnukeith · 2026-08-22
• FreeToken: 4x Faster Decode, Enables 284B Models on Gaming Desktops — airesearch12 · 2026-08-22
• FreeToken Tested: 100 tok/s Running a 35B Model on Consumer Hardware — ViRROOO · 2026-08-22
• Frontier Models on Consumer GPUs: Qwen3.6 on 8GB VRAM at 39 tok/s — berkeley_ai · 2026-08-22
• FreeToken: Run 284B frontier models on consumer GPUs at interactive speeds — solyarisoftware · 2026-08-22
• FreeToken System Enables RTX4060 Laptops to Run 35B Models Smoothly — 机器之心 · 2026-08-22
• FreeToken Enables 35B Models on 8GB Laptops via Unified CPU-GPU Memory — mark_k · 2026-08-23
• FreeToken Paper: Edge-Native MoE Serving with Bandwidth-Adaptive Execution — mark_k · 2026-08-23
• FreeToken benchmarks: a single RTX PRO 6000 runs a 753B model at 14.9 tok/s — jasonkneen · 2026-08-23

Episode 2 · FreeToken Lets 8GB GPUs Run 35B MoE Models On-Device (2026-08-29, 2 posts)

UC Berkeley and UT researchers released FreeToken, an inference engine that dynamically adapts GPU caches and bandwidth for MoE models, enabling an 8GB laptop GPU to run a 35B model at 39.3 tokens per second.

• FreeToken Engine: Run 35B Models at 39.3 t/s on 8GB GPUs — rohanpaul_ai · 2026-08-29
• FreeToken: 8GB Laptop Runs 35B Model at 39.3 t/s via Edge-Native MoE Serving — rohanpaul_ai · 2026-08-29