UC Berkeley open-sources FreeToken: MoE inference 2-4x faster, runs 35B model on 8GB GPU
Roger_M_Taylor · x · 2026-08-25
UC Berkeley has open-sourced FreeToken, a local inference engine that runs 2-4x faster than Ollama by exploiting Mixture-of-Experts (MoE) architectures. Initial benchmarks show Qwen3.6-35B at 39.3 tokens/s on an 8GB GPU, DeepSeek-V4-Flash 284B at 22 tokens/s on a 32GB GPU, and GLM-5.2 753B at 14.9 tokens/s on a 96GB GPU. A 35B model normally requires 70GB for weights, but FreeToken serves it on 8GB by targeting the MoE router and dynamically splitting memory misses between CPU and PCIe. This architecture also benefits agents: coding agents constantly rewrite history, forcing thousands of tokens through prefill; FreeToken saves checkpoints at agent framework boundaries to reduce redundant computation.
Related event: Berkeley Open-Sources FreeToken: Run 290B MoE Models on 8GB GPUs(4 posts)→
More from Infra
- What is the next frontier for AI memory? — boneMechBoy69420 · 2026-08-25
- Robotic demos stuttering? FlashRT engine offers Hz-level latency benchmarks — Xianbao_QIAN · 2026-08-25
- AI Chipmaker Enflame Sets Subscription Date for Near $900M Shanghai IPO — pstAsiatech · 2026-08-25
- Taiwan indicts 9 as 74 Nvidia B300 AI servers allegedly smuggled into China — kimmonismus · 2026-08-25
- Qwench: Fully automated local fine-tuning pipeline for Qwen3.8-27B — raiyanyahya · 2026-08-25
- Keploy MCP connector: Generate API tests from traffic or specs — modelcontextprotocol · 2026-08-25