FreeToken: run 290B+ MoE models locally on a gaming PC, no big VRAM needed
gnukeith · x · 2026-08-22
Open-source project FreeToken (FlashML-org) is an edge-native MoE serving engine for running frontier-scale 290B+ open-weight MoE models on consumer hardware like a 5090 with large system RAM, at interactive speeds.
It treats heterogeneous edge resources—GPU, CPU, host memory, interconnects—as one elastic inference platform, with bandwidth-adaptive CPU–GPU co-execution (q policy), full-layer double-buffered prefill streaming, global LRU expert caching, the FTW fast weight format, and semantic-anchor checkpointing for KV cache. Nearly 300 stars on GitHub so far.
More from Infra
- Fluidstack hires Electrical Engineer for Modular R&D to build frontier AI infrastructure — MxMnr · 2026-10-16
- Jensen Huang joins BG2 podcast: does AI revenue justify capex, '27 bottlenecks — elonmusk · 2026-10-16
- Mistral's on-device infra rack revealed: DGX Spark, Mac Studio and more — gregmushen · 2026-10-16
- NVIDIA and Sweden's Wallenberg launch Sferical One, a shared AI supercomputer — nvidia · 2026-10-16
- Next infra frontier: self-evolving systems that adapt and self-verify — _shreya_s · 2026-10-16
- Xiaomi's MiMo RL dashboard reveals $3.47M spent on just 30 training steps — bycloud · 2026-10-16