FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
jiqizhixin · x · 2026-08-26
Researchers from UC Berkeley, MIT, and UT Austin released FreeToken, a full-stack co-designed serving system for MoE models on consumer hardware. By using bandwidth-adaptive execution that dynamically splits computation between CPU and GPU based on real-time bus conditions, and employing lightweight token-boundary checkpoints for state reuse, FreeToken achieves significant performance gains. It delivers 39.3 tokens/s on a laptop RTX 4060, cuts TTFT by 42-58%, and reduces multi-turn agent latency by 65-80%, enabling full-speed DeepSeek V4-Flash execution on a single RTX 5090 without data center clusters.
More from Infra
- SK Hynix ties new US plant to looser chip equipment exports to China, sources say — pstAsiatech · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- vLLM-Omni ships Day-0 serving for FLUX 3 Image Klein with BF16, FP8, NVFP4 support — vllm_project · 2026-10-15
- NVIDIA's Nemotron 3.5 Super VL lands training and inference support on Applied Compute — ypatil125 · 2026-10-15
- CoreWeave bond yield tops 16% as loans outlast customer contracts by ~2 years — tengyanAI · 2026-10-15