Running Qwen3.8-Flash-Next with 256K context at 16 tok/s on DDR4 and a Tesla T4
BusTiny207 · reddit · 2026-09-04
A Reddit user runs Unsloth's Qwen3.8-Flash-Next UD-Q4KXL (180B total/6B active, 111GB) on a refurbished Dell R740 with ikllama.cpp, achieving 256K context at 16 tok/s generation on a single Tesla T4 plus 384GB DDR4.
Key details:
- All 512 experts kept in host RAM (-cmoe); non-expert weights (4606 MiB) on the T4; full 256K context fits in 13.0GB
- Quantizing -ctv and -ictk from f16 to q80 and dropping -ub to 1024 were the critical changes to fit 256K
- Performance: 159.6 t/s prompt processing on a cold 12.5K prompt, 16-17.6 tok/s generation; going 128K→256K costs only 2.5% generation speed
- Real-world use: it completed a solid code refactor and handled obscure Nim coding questions, with notably less verbose output than Opus
The post includes the full launch command and tuning flags — a valuable blueprint for cheap large-context MoE inference on old servers.
More from coding & agent
- GPT Sol plays real-time table tennis via latency-aware agent research loop — wzenus · 2026-10-21
- Daytona raises $60M Series B at $1B valuation to build the agent cloud — tsi_org · 2026-10-21
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- 9x Vision open-sources video perception for agents: timestamped evidence via MCP and CLI — koto9x · 2026-10-21
- LangChain adds Human-in-the-Loop beta to Managed Deep Agents — LangChain · 2026-10-21
- Voxtral 3 ships a LiveKit-based voice agent SDK streaming listen-think-speak — MistralAI · 2026-10-21