FreeToken: run 290B+ frontier MoE models locally on your gaming PC
Saboo_Shubham_ · x · 2026-08-25
FreeToken is an edge-native MoE serving engine (4.8k stars, 422 forks on GitHub) built to run frontier-scale open-weight models on consumer hardware — claiming interactive-speed inference of 290B+ parameter MoE models on a gaming PC.
Key design points:
- Treats heterogeneous edge resources (GPU, CPU, host memory, interconnects) as one elastic inference platform
- Bandwidth-adaptive CPU–GPU co-execution with a q policy
- Full-layer double-buffered prefill streaming, global LRU expert caching, and the FTW fast weight format
- Semantic-aware caching
It speaks the Anthropic/OpenAI APIs so Claude Code and Codex plug straight in; fully open source with a paper and community channels. The thread also highlights the Awesome LLM Apps repo (134k+ stars) with 100+ AI agent, skills and RAG templates.
More from coding & agent
- Devin Oncall tops SREGym with 100% end-to-end success, auto-resolving production incidents — tianyin_xu · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Tool calls are the hidden bottleneck killing AI coding speed — Crabbox and 5 fixes — steipete · 2026-10-16
- Agentic sandbox WorldBox doubles GPT-6 Astra's MazeBench score, 23 to 61 gems — ZhitingHu · 2026-10-16
- Two prompts: Gemini 4 Argon turns a single-player game into full multiplayer — chongdashu · 2026-10-16