FreeToken Paper: Edge-Native MoE Serving with Bandwidth-Adaptive Execution
mark_k · x · 2026-08-23
FreeToken is an edge-native MoE serving system designed to treat personal machines as unified, elastic inference platforms. The system co-designs the full serving stack around two realities of local AI: agent workloads continuously change execution patterns, and edge hardware exposes heterogeneous resources.
Rather than using a fixed offloading strategy, FreeToken dynamically maps computation and model state to available resources. It supports over 20 MoE models and real coding/tool-using agents across hardware from 8GB laptop GPUs to single workstation GPUs. Key results include running 35B models on laptops, 284B models on gaming desktops, and the 753B GLM-5.2 on a single workstation GPU.
More from Infra
- NVIDIA releases Nemotron 3.5 Bolt 74B MoE that fits in 24GB VRAM at 2-bit — NerdyRodent · 2026-10-16
- Analyst: 80% of neoclouds may not survive, but the demand for their GPU capacity is real — BenBajarin · 2026-10-16
- Apple and Meta partner to run Muse Spark model on linked Mac Studios — pdamodaran · 2026-10-16
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16