FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
cs.DC
2026-08-17
A consumer PC as one elastic MoE serving platform: cache misses split between PCIe and CPU execution by measured bandwidth; 35B on an 8GB laptop, 753B on one GPU.
Open weights closed the access gap, not the running gap. DeepSeek-V4-Flash activates only 13B of its 284B parameters per token, yet its full expert pool is roughly 140GB in FP4, far beyond consumer VRAM, so local engines stream experts over PCIe on demand. llama.cpp, Ollama, and KTransformers each fix a fragment. Three holes remain: prefill on a long agent context routes through nearly every expert, pushing the whole pool across PCIe (about 2s on PCIe 5.0 x16, 10s or more on laptop x8 links); decode has no principled policy for dividing expert cache misses between PCIe transfer and CPU execution, and static placement cannot follow token-level routing; and edge VRAM budgets shift mid-session as browsers and games claim memory. The authors are a systems group from Berkeley and UT Austin (Song Han, Matei Zaharia, and Ion Stoica among them), and the stance is explicit: schedule the whole machine as one elastic inference platform instead of treating a consumer GPU as a small datacenter GPU.
The skeleton is a two-level expert memory: the complete expert pool stays resident in host memory, non-expert weights stay on the GPU, and all leftover VRAM becomes one elastic expert cache shared across MoE layers, managed by logical (layer, expert) IDs. Three mechanisms sit on top:
Execution is captured into CUDA Graphs: dedup, hit classification, q-star derivation, and LRU victim selection happen in one GPU kernel, and the CPU branch is captured as a graph node with its synchronization, avoiding per-layer host sync. When the VRAM budget changes, the cache is rebuilt at a scheduler safe point without restarting. Weights ship in an FTW format pre-merged into the runtime layout, so loading reads straight into the final host layout before pinning.
| Setup | Metric | Result |
| RTX 5090, Qwen3.6-35B-A3B | decode | 77-83 tok/s, 1.8-2.3x the strongest baseline |
| RTX 5090, DeepSeek-V4-Flash 284B | decode | 22-25 tok/s, 1.5-1.9x |
| RTX PRO 6000, GLM-5.2 753B | decode | 14.9 tok/s vs llama.cpp at 7.3 |
| 8GB RTX 4060 laptop, 35B | decode | 39.3 tok/s, above the 33 tok/s Codex production median |
| Worst-turn TTFT, all workloads | tail latency | under 44s; every baseline crosses 179-946s somewhere |
Stability under agentic load is the headline: decode stays within 12% of the single-turn setting, while KTransformers loses 31% on its first agent workload. Attribution experiments back the mechanisms: at equal cache capacity the global LRU misses 16%/39% of decode-time expert reads (two models) versus 41%/59% for KTransformers' prefill-time placement and 62%/89% for llama.cpp's static split; removing double buffering costs 26% of prefill throughput at 16k tokens; across five consumer machines the lead over the strongest baseline is 1.3-2.1x.
For anyone running frontier open models locally, this moves the boundary of what hardware serves what model: an 8GB laptop runs 35B interactively, one 96GB workstation GPU runs 753B, and that frontier-tier comparison uses bit-identical expert weights, so the win comes from scheduling rather than quantization. The q-star closed-form split by measured bandwidth is directly reusable in any hybrid CPU-GPU inference stack. The system is open-sourced at flashml.ai, supports more than 20 MoE models, and was validated on four real agent workloads including Claude Code's native protocol.
The full expert pool must fit in host memory: GLM-5.2's 433GB checkpoint restricts that tier to workstation-class RAM (the test box has 512GB). The rented 3090/4090/5090 servers cap CPU at 6 threads to emulate consumer hosts, an emulation rather than fully native hardware, though two real machines measured bandwidth in the same range. The evaluation is single-stream and single-user, with no concurrent-serving comparison; whether every baseline was tuned to its best is not argued case by case, and MoE-Infinity only completes the single-turn workload. The CPU branch depends on pinned memory and DMA registration; where those are unavailable the system falls back to a pure-CPU MoE backend at reduced peak bandwidth.