Running Qwen3.8-Flash-Next on 2x5080: should I move from llama.cpp to vLLM?
whatyathinkk · reddit · 2026-09-17
A user runs Qwen3.8-Flash-Next Q5KXL on 2x RTX 5080 + 128GB DDR5 via llama.cpp with MTP speculative decoding and CPU MoE offload (ncmoe42), hitting 100 tps prefill and 18-20 tps decode. With few NVFP4/FP8 .gguf files available, they ask whether switching to vLLM/SGLang would yield better performance for this model on their setup. Full config included.
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02