UC Berkeley open-sources FreeToken, an edge-native engine running frontier MoE models on consumer hardware
petrusenko_max · x · 2026-11-10
UC Berkeley open-sourced FreeToken, an edge-native inference engine that runs frontier-scale MoE models on consumer hardware by treating GPU, CPU, host memory, and PCIe interconnects as one unified inference platform.
- Why it works: MoE models like DeepSeek-V4-Flash have 284B total parameters but activate only 13B per token (6 of 256 routed experts per layer); the bottleneck is keeping the full expert pool accessible in memory, far exceeding VRAM
- Two-level hierarchy: non-expert weights stay resident on GPU; all expert weights live in host memory and are fetched on demand
- In practice: the poster runs 93 scheduled inference jobs on a single Mac mini; when two MoE jobs share one PCIe link, the scheduling policy tunes dynamically around shifting bandwidth
More from coding & agent
- Eric S. Raymond shares his AI coding workflow: Threadripper rig, Codex, no autonomous commits — esrtweet · 2026-11-10
- Chollet backs 'AI CEO' Steve that runs acquired SaaS firms with P&L as objective function — fchollet · 2026-11-10
- The most important dev skill in 2026: telling your agent "no, not like this" — dbreunig · 2026-11-10
- Univer 1.4 launches with Spaces: build agent-native workspaces with docs, sheets, and slides — FellMentKE · 2026-11-10
- PCBRouteAgent: multimodal LLM agent routes 84/86 boards vs baseline 10/86 — Yue Zhang · 2026-11-10
- Forgetful AI Agent Derails Enterprise Pilot: Rereading Context Ate 7% of Daily Usage — thedealdirector · 2026-11-10