FreeToken from UC Berkeley and MIT brings datacenter-scale MoE serving to your desktop
nilukush · reddit · 2026-09-04
Researchers from UC Berkeley and MIT (co-authored by Databricks co-founders Matei Zaharia and Ion Stoica, plus Song Han, Kurt Keutzer and others) open-sourced FreeToken, an inference engine bringing frontier MoE models to consumer hardware.
- Problem: Sparse MoE decoding still routes across hundreds of billions of inactive weights; PCIe bandwidth (16–64 GB/s) and host RAM latency bottleneck decode on desktops, and existing edge runtimes stall on cache misses with static expert offloading.
- q dynamic co-scheduling: Instead of halting the GPU on cache misses, FreeToken splits token computation between CPU cores and GPU tensor cores based on real-time interconnect throughput.
- FTW fast weight format + full-layer double buffering: Weight streaming over PCIe fully overlaps with active computation layers.
- Elastic memory manager: Dynamically reallocates VRAM between KV cache entries and resident expert slots at runtime without reloading the model.
The project reframes personal machines as elastic, heterogeneous computing fabrics rather than constrained datacenter nodes.
More from Infra
- Lightning AI ships 200x faster Drive persistence, starts swapping H100s for H200e — LightningAI · 2026-09-04
- LLM inference metrics explained: what TTFT, TPS, TPOT actually measure — abhijithneil · 2026-09-04
- LLM decoding explained: prefill reads in parallel, decode writes token by token — abhijithneil · 2026-09-04
- GPU Inference Explained: Memory Bandwidth, Not Compute, Caps Tokens Per Second — abhijithneil · 2026-09-04
- LLM inference 101: bandwidth ÷ weight bytes gives your throughput ceiling before any code — abhijithneil · 2026-09-04
- Red-Teaming Semantic Cache Verifiers: 84% of Adversarial Pairs Slip Through — Reasonable_Royal_621 · 2026-09-04