Running Qwen3.8-Flash-Next on a 5090 with llama.cpp: 40 tok/s and barely any RAM used
nirurin · reddit · 2026-09-25
A first-hand report of running Qwen3.8-Flash-Next (Atomic Q4KM quant, 33 GGUF shards) via llama.cpp on a 5090 + 64GB RAM. Using mmap/lazy-mode flags with --n-cpu-moe 32, the setup uses 27GB VRAM and only 8GB system RAM at 40 tok/s decode and 50 tok/s prompt processing — surprisingly fast, with parts apparently streaming from NVMe rather than being buffered in RAM.
More from Infra
- Anthropic paid $5B for short-term compute in Q3 as spot prices run ~2x long-term deals — tengyanAI · 2026-11-11
- UK Sovereign AI invests in Fractile to build next-gen frontier AI chips — charlieharris01 · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- AI chip startup Fractile raises $720M Series C for DRAM-on-processor design — Tom_Westgarth15 · 2026-11-11
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Vivo's Starlink direct-to-cell service hits 10M users in two weeks across Brazil — XFreeze · 2026-11-11