UC Berkeley open-sources FreeToken: MoE inference 2-4x faster, runs 35B model on 8GB GPU
Roger_M_Taylor · x · 2026-08-25
UC Berkeley has open-sourced FreeToken, a local inference engine that runs 2-4x faster than Ollama by exploiting Mixture-of-Experts (MoE) architectures. Initial benchmarks show Qwen3.6-35B at 39.3 tokens/s on an 8GB GPU, DeepSeek-V4-Flash 284B at 22 tokens/s on a 32GB GPU, and GLM-5.2 753B at 14.9 tokens/s on a 96GB GPU. A 35B model normally requires 70GB for weights, but FreeToken serves it on 8GB by targeting the MoE router and dynamically splitting memory misses between CPU and PCIe. This architecture also benefits agents: coding agents constantly rewrite history, forcing thousands of tokens through prefill; FreeToken saves checkpoints at agent framework boundaries to reduce redundant computation.
More from Infra
- SK Hynix ties new US plant to looser chip equipment exports to China, sources say — pstAsiatech · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- vLLM-Omni ships Day-0 serving for FLUX 3 Image Klein with BF16, FP8, NVFP4 support — vllm_project · 2026-10-15
- NVIDIA's Nemotron 3.5 Super VL lands training and inference support on Applied Compute — ypatil125 · 2026-10-15
- CoreWeave bond yield tops 16% as loans outlast customer contracts by ~2 years — tengyanAI · 2026-10-15