FreeToken doubles local MoE speed: Qwen on 16GB RTX 5070 Ti hits 90+ tps
solyarisoftware · x · 2026-09-06
- A llama.cpp GitHub discussion calls for adopting FreeToken after wild local MoE benchmarks: Qwen3.6-35B-A3B-NVFP4 (23GB) runs on a 16GB RTX 5070 Ti at 90+ tps (100+ on subsequent calls); on an RTX 5090 with Qwen3.8-Flash-Next Q4, FreeToken hits 53 tps vs llama.cpp's 22 — more than 2x.
- FreeToken treats the whole PC as an inference system: experts live in system RAM, hot experts are cached in VRAM, cache misses stream over PCIe, the CPU can execute some misses, and CPU work overlaps with PCIe transfers — the key is deciding which experts must be in VRAM right now.
More from Infra
- llama.cpp team launches Llama: one-click local AI on your Mac menu bar — Hugging Face · 2026-10-23
- China's daily token use hits 500T, up 4x as average price doubles to ¥13.46 — teortaxesTex · 2026-10-23
- New all-reduce kernels deliver 12% decode speedup for DeepSeek-V4.1 on PCIe-connected GPUs — AlpinDale · 2026-10-23
- Offshore oil rigs could be converted into AI data centers powered by stranded gas — clemnt · 2026-10-23
- Same native UI, two stacks: Swift build is ~80MB Mac-only, Zig build is ~10MB cross-platform — jedisct1 · 2026-10-23
- SurfSense: an open-source local NotebookLM with MCP agents and a prompt-caching lesson — FurtiveMirth · 2026-10-23