Qwen3.8 27B on one RTX 5090: six inference engines benchmarked across TTFT, decode and tool tasks
fuzhongkai · reddit · 2026-10-27
The author of TensorSharp (disclosed) benchmarked six engines—TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer, NInfer—running Qwen3.8 27B on a single RTX 5090 (32GB), covering TTFT, prefill, decode, concurrency, and tool tasks. Greedy decoding, thinking disabled, 40,960-token context. Key findings:
- Short-prompt TTFT: llama.cpp fastest at 27ms plain
- Long lookups: NInfer lowest 16K/32K TTFT (3.02s at 32K)
- Speculation overhead: llama.cpp's 32K TTFT jumped 5.01s→8.16s; others added <0.2s
- Screenshot input: TensorSharp 125ms vs vLLM 127ms roughly tied; with DFlash2, TensorSharp led at 142ms
- Configs differ: quantization (NVFP4 GGUF / safetensors / .ninfer) and KV-cache formats vary, so results compare complete configurations
Rankings shift by workload—no overall winner. Full methodology and configs in the linked report.
Related event: Six Inference Engines Benchmarked on a Single RTX 5090 Running Qwen 27B(2 posts)→
More from Infra
- Nvidia pushing NVHBM to de-risk base die after Hynix and Micron fumble, says report — basedjensen · 2026-10-28
- Intel CEO leads $30M Series A1 in Hyperbolic to unite fragmented neocloud GPUs — YiMaTweets · 2026-10-28
- Opera introduces WebAI standard to run AI models locally in the browser, no cloud or API keys — emmanuelvivier · 2026-10-28
- OpenRouter announces first APAC partnership with Korea's KT Telecom for sovereign models — AccBalanced · 2026-10-28
- Porcupine-like heatsink found inside Calnex Rb/GPS frequency reference during Noctua fan swap — Pytlicek · 2026-10-28
- Unverified: sub-1GB quantized text models reportedly run on phones, VLM rivals Flash Lite class — JosephJacks_ · 2026-10-28