Qwen3.8-27B on 2× RTX 5070 Ti: One vLLM Flag Kills Speed 11×, Full Benchmarks Inside
puthre · reddit · 2026-09-05
The author benchmarks Qwen3.8-27B across llama.cpp, vLLM and NInfer on dual RTX 5070 Ti 16GB (Blackwell, no P2P), with full server configs.
- Results: llama.cpp (Q4KXL) leads at 115 tok/s with MTP3 speculative decoding and 220k context; vLLM (NVFP4) 110 tok/s at 122k; NInfer 100 tok/s but the only one with full native 262k context.
- Biggest lesson: --enforce-eager tanks vLLM on Blackwell from 110 to 9.6 tok/s — an 11× difference from one flag.
- Key flags: tiny CUDA graphs (--max-cudagraph-capture-size 4), --kv-cache-memory, --disable-custom-all-reduce for 16GB dual-GPU; llama.cpp needs q80 KV cache for 220k, MTP3 acceptance 65%, stable 110 tok/s from 8k to 131k context.
- NInfer officially lists only 5090 but works flawlessly on 5070 Ti without modification.
- Verdict: speeds are close (100–115 tok/s); context window and setup complexity are the real differentiators.
More from Infra
- Anthropic IPO investors press for revenue per token and per gigawatt metrics — Hesamation · 2026-09-05
- Tauri-Based Wan2GP Desktop Launcher Brings One-Click Open-Source Video Gen to Windows — Extension_Affect_483 · 2026-09-05
- MLX-DLSS: Hacker runs NVIDIA's DLSS 5 neural renderer on a Mac, within 0.005 MAE of the original — WaveCut · 2026-09-05
- ESP32 voice assistant: AIMET AdaRound cuts wake-word model 8x without the 3.7% accuracy hit of naive 4-bit — carrycooldude · 2026-09-05
- Dev boosts GLM 5.2 TPS on a B300 and swaps it into Claude Code in place of Anthropic models — abhijithneil · 2026-09-05
- What forces LLM teams to optimize inference when going from MVP to production? — Ok_Philosophy_4031 · 2026-09-05