Running a 180B MoE (5B active) on one RTX 3090 + 128GB RAM: full config and 15.5 tok/s benchmarks
cezarducatti · reddit · 2026-09-05
A developer shares a complete llama.cpp setup and real benchmarks for running Qwen3.8-Flash-Next (UD-Q4KXL, 180B total / 5B active MoE) on a single RTX 3090 24GB plus 128GB DDR4, and asks for tuning advice:
- Key flags: -ngl 99 to offload all layers possible, -ncmoe 42 to keep 42 expert layers on CPU/RAM, 200k context, f16 K/V cache, full xhigh reasoning with no truncation
- After loading, VRAM is essentially maxed (0.6GB free) with 52GB system RAM headroom for offloaded experts and context
- Benchmarks: prompt processing holds 185-190 tok/s at 12k context; generation stays remarkably stable at 15.5-16 tok/s even as context grows past 12k
- He notes recent llama.cpp master builds shipped Qwen4Exp-specific changes (MTP speculative decoding rollback support, a Vulkan top-k radix sort shader, fewer graph splits) — upgrade before blaming your config for MTP instability
- Open questions: optimal -ncmoe value, squeezing prompt-processing throughput, and whether MTP speculative decoding is reliable on mainline builds
More from Infra
- GeneralistAI collects 500,000+ hours of real robot data, launches onchain data bounties via Robinhood — broodsugar · 2026-09-05
- NInfer vs llama.cpp vs vLLM: rigorous quality and speed test of Qwen3.8-27B NVFP4 on one RTX 5090 — bengizmoed · 2026-09-05
- Omagrid P2P compute network goes live, sharing DeepSeek and Qwen across Omarchy machines — dee_hw · 2026-09-05
- Hugging Face surveys 16 open-source RL libraries: async disaggregation is the consensus — Thom_Wolf · 2026-09-05
- Gave up local video rendering on an RTX 3060 for cloud — but lost the node-based modularity — Suspicious_Pizza9529 · 2026-09-05
- Anthropic IPO investors press for revenue per token and per gigawatt metrics — Hesamation · 2026-09-05