FreeToken open-sources engine running 284B models on a gaming PC
alex_verem · x · 2026-11-03
Berkeley and MIT researchers open-sourced FreeToken, an edge-native MoE serving engine that treats GPUs, CPUs and host memory as one elastic inference platform. Paper results: 35B on a laptop with 8GB VRAM, 284B on a gaming desktop, 753B GLM-5.2 on one workstation card. Features bandwidth-adaptive CPU-GPU co-execution, semantic-aware caching for agent contexts, and runtime VRAM reallocation. Supports DeepSeek, Qwen, GLM, Gemma, gpt-oss with OpenAI/Anthropic-compatible APIs. 15.9k GitHub stars, Apache-2.0.
Related event: FreeToken Open-Sources Datacenter-Scale MoE Inference for Desktops(2 posts)→
More from Infra
- Sampler Pause Stays Under a Second Even Across Distant Datacenters — gharik · 2026-11-03
- Google's Suncatcher satellite runs Gemma 270mi n space—power limits mean only one TPU lit at a time — demian_ai · 2026-11-03
- Mistral VP: Le Chonk trained on just 10 MW (~4,000 H100s), begs investors for 1 gigawatt — _AndrewZhao · 2026-11-03
- UCSB awards IEE seed funding to three ECE projects on energy-efficient AI and quantum hardware — ninamiolane · 2026-11-03
- Europe's Physical AI Agenda: Gigafactories, Sovereign Compute and AI Act Rules — ingliguori · 2026-11-03
- Weight teleportation open-sourced: cross-datacenter RL weight updates under 0.23s — gharik · 2026-11-03