Qwen3.8 Flash Next on one RTX 5090 hits 50 t/s decode via FreeToken expert caching
dir3ctly · reddit · 2026-09-20
A Reddit user shares benchmarks running Qwen3.8 Flash Next locally on a single RTX 5090 with FreeToken: 50 t/s token generation and 2300 t/s prefill, stable over long contexts.
- Motivation: llama.cpp still lacks MoE Expert Caching (several unmerged PRs), making default performance disappointing.
- FreeToken's expert caching is the key; the post includes the full launch config (--moe-strategy auto --moe-cache-auto --expert-load parallel, 229376-token sequence override, etc.) using the nvidia/Qwen3.8-Flash-Next-NVFP4 model (convert to FreeToken format or startup is slow).
- Hardware: RTX 5090, 128GB DDR5 6000 (96GB works fine).
- The approach works with other MoE models too; the project is early — vision just landed, no KV quantization or MTP yet.
More from Infra
- Activist evicted in 2024 puts $5B Maryland AI data center on hold — ATTlKA · 2026-11-12
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12