Qwen3.8-Flash-Next hits 68.3 tok/s on a single RTX 5090 with new FreeToken inference stack
matei_zaharia · x · 2026-09-05
UC Berkeley Sky Lab's Shuo demonstrated FreeToken, a local inference setup running Qwen3.8-Flash-Next on a single RTX 5090 at 68.3 tok/s — no extreme quantization, no speculative decoding. It uses a GB300-validated NVFP4 production checkpoint from RadixArk, needs only 63GB host RAM (less than 1-bit quants), and keeps the 51GB n-gram table on NVMe at 0.5% throughput cost. A demo shows generating a playable Minecraft-style world from one prompt and fixing a real FreeToken bug in 10.5 minutes. Matei Zaharia highlighted it as a sign powerful local AI is coming.
More from Infra
- llama.cpp team launches Llama: one-click local AI on your Mac menu bar — Hugging Face · 2026-10-23
- China's daily token use hits 500T, up 4x as average price doubles to ¥13.46 — teortaxesTex · 2026-10-23
- New all-reduce kernels deliver 12% decode speedup for DeepSeek-V4.1 on PCIe-connected GPUs — AlpinDale · 2026-10-23
- Offshore oil rigs could be converted into AI data centers powered by stranded gas — clemnt · 2026-10-23
- Same native UI, two stacks: Swift build is ~80MB Mac-only, Zig build is ~10MB cross-platform — jedisct1 · 2026-10-23
- SurfSense: an open-source local NotebookLM with MCP agents and a prompt-caching lesson — FurtiveMirth · 2026-10-23