FreeToken: run 290B+ MoE models locally on a gaming PC, no big VRAM needed
gnukeith · x · 2026-08-22
Open-source project FreeToken (FlashML-org) is an edge-native MoE serving engine for running frontier-scale 290B+ open-weight MoE models on consumer hardware like a 5090 with large system RAM, at interactive speeds.
It treats heterogeneous edge resources—GPU, CPU, host memory, interconnects—as one elastic inference platform, with bandwidth-adaptive CPU–GPU co-execution (q policy), full-layer double-buffered prefill streaming, global LRU expert caching, the FTW fast weight format, and semantic-anchor checkpointing for KV cache. Nearly 300 stars on GitHub so far.
More from Infra
- x402 Protocol Replaces Thousands of APIs with One Line of Code for Agentic Payments — 0xJeff · 2026-08-22
- Running Phi-4 Mini on Intel NPU/iGPU: Real-world Tests — Ecstatic-Wash-7667 · 2026-08-22
- SiPho Loss Reality Check: The Trick Behind Low-Loss SiN — jwt0625 · 2026-08-22
- llama.cpp Error: Ling 3.0 Flash Model Shows 'Unused Tensor' Warnings — Debianreiser69 · 2026-08-22
- AMD 7900 XTX Bug: Model Unloads After 5 Seconds of Idle — deathcom65 · 2026-08-22
- Cloud Providers Will Drive the Blend of Open and Closed Source — iamtrask · 2026-08-22