QEngine 1.3.1 hits 504 tok/s decode on 2x RTX 5090 with Qwen3.8-Flash-Next
Relative-Ant-9249 · reddit · 2026-10-31
A Reddit user benchmarked four inference engines running Qwen3.8-Flash-Next on 2x RTX 5090 (Threadripper PRO 3975WX, 262K context). QEngine 1.3.1 (INT4 AutoRound) wins single-stream decode by a clear margin.
Key numbers:
- Decode: 370/439 tok/s default, 406/504 with KV-in-RAM — roughly 1.4-1.9x Strata and 3x tuned vLLM (with the caveat that QEngine's expert cache favors repeated prompts; 300-400 tok/s on varied traffic)
- Only change needed for 5090: CMAKECUDAARCHITECTURES 86→120; serving 53s after launch
- Agentic latency: 1.75s turn-12 prompt read, level with Strata, 2x faster than vLLM
- FLASHKV=ram keeps 1M tokens of KV in pinned host RAM: 8/8 idle conversations returned warm in 1.56-2.01s vs 2/8 in VRAM mode (others re-prefilled in 9s)
- Quality holds: 44/44 on the omp agent suite; aider-polyglot 26/30, within Strata INT4's repeat spread
More from Infra
- Wafer hits #3 on OpenRouter by daily token volume at 1.9T — ycombinator · 2026-10-31
- Blogger argues $30K local Kimi K3.1 setup is worth it, despite taking 4.76 years to match GPT-6.1 Terra output — bookwormengr · 2026-10-31
- You can fingerprint your exact GPU with a single torch.index_add_ call — CatAstro_Piyush · 2026-10-31
- SGLang boosts DeepSeek V4.1 Flash throughput 67% to 1.09M tok/s per GPU in 4 days — BanghuaZ · 2026-10-31
- llama.cpp ROCm regression: prompt processing drops from ~1200 to ~460 t/s on RX 9070XT — uber-linny · 2026-10-31
- Sub-scale neocloud warning: VC-backed GPU cloud startups reportedly struggling — AccBalanced · 2026-10-31