Running Qwen3.8-Flash-Next with 256K context at 16 tok/s on DDR4 and a Tesla T4
BusTiny207 · reddit · 2026-09-04
A Reddit user runs Unsloth's Qwen3.8-Flash-Next UD-Q4KXL (180B total/6B active, 111GB) on a refurbished Dell R740 with ikllama.cpp, achieving 256K context at 16 tok/s generation on a single Tesla T4 plus 384GB DDR4.
Key details:
- All 512 experts kept in host RAM (-cmoe); non-expert weights (4606 MiB) on the T4; full 256K context fits in 13.0GB
- Quantizing -ctv and -ictk from f16 to q80 and dropping -ub to 1024 were the critical changes to fit 256K
- Performance: 159.6 t/s prompt processing on a cold 12.5K prompt, 16-17.6 tok/s generation; going 128K→256K costs only 2.5% generation speed
- Real-world use: it completed a solid code refactor and handled obscure Nim coding questions, with notably less verbose output than Opus
The post includes the full launch command and tuning flags — a valuable blueprint for cheap large-context MoE inference on old servers.
More from coding & agent
- Stalkr lands first customers in 10 minutes: social listening with AI-sorted mentions and MCP support — marclou · 2026-09-04
- Using MCP for CRM data cleanup: auditing 14k stale HubSpot accounts instead of prospecting — snowingbol · 2026-09-04
- GPT-6 Astra reviews: half the per-task cost, but CoT monitoring breaks down — vista8 · 2026-09-04
- 1360 runs: Edinburgh researcher benchmarks local LLMs against Aider, Claude Code, OpenCode and more — PMinervini · 2026-09-04
- HKUST Guangzhou and Tencent open-source VibeWorlding, an agent framework for building 3D worlds by chat — jiqizhixin · 2026-09-04
- Editing an Old Message Is ChatGPT's Closest Thing to /compact for Long Project Chats — Ok_Negotiation_2587 · 2026-09-04