Running Qwen3.8 Flash Next on 128GB RAM + one 5080: 136pp/19tg at Q5_K_XL
whatyathinkk · reddit · 2026-09-15
A detailed local-inference setup for Qwen3.8 Flash Next on 128GB RAM + a single RTX 5080 (16GB): Unsloth's Q5KXL hits 136pp/19tg, Q4KXL 152pp/23tg, Q3KXL 195pp/23tg. Slow but impressive quality, the poster says.
Config highlights: 6-part GGUF (UD-Q5KXL-ncmoe48), lazy-mode on, ngl=999 with n-cpu-moe=48 keeping MoE layers on CPU, perlayertokenembd forced to CPU, 262K context, q80 KV cache, flash-attn on; sampling at temp=1.0/top-k=20/top-p=0.95 with thinking preserved and reasoningeffort=medium. Full llama.cpp config included; the poster asks about MTP or other speedups.
More from Infra
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30