Six inference engines benchmarked on one RTX 5090 running Qwen 27B with DFlash2
fuzhongkai · reddit · 2026-10-27
The TensorSharp author benchmarked llama.cpp, vLLM, SGLang, NInfer, velox-inferer and TensorSharp on a single RTX 5090 32GB running a Qwen 27B model: plain decode peaked at 84.2 tok/s (llama.cpp), while DFlash2 speculative decoding lifted NInfer to 280.5 tok/s single-request and 488 tok/s aggregate across four concurrent requests (TensorSharp: 445). NInfer also had the best 32K-prompt TTFT (3.0s); llama.cpp degraded to 8.16s TTFT with speculation. Caveat: engines used different quantized checkpoints and drafters, and sub-5% differences are inconclusive. velox failed JSON-schema checks in all runs.
Related event: Six Inference Engines Benchmarked on a Single RTX 5090 Running Qwen 27B(2 posts)→
More from Infra
- Critical pre-boot RCE in EDK II PXE driver: one crafted packet hijacks servers before Secure Boot — cyb3rops · 2026-10-28
- Screenless M1 Airs at $30 per 1,000 multicore points spark cheap Mac cluster idea — vincent_koc · 2026-10-28
- RTX 3080 Tensor Core unlock claims 80-87% TFLOPS boost, RTX 50 series blocked — Revolutionalredstone · 2026-10-28
- CXMT to start small-scale HBM3E production within two months, yield at 20% — zephyr_z9 · 2026-10-28
- Video Transcription Costs Under 1 Yuan per Hour, CPU Free but Error-Prone — huangyun_122 · 2026-10-28
- A forgotten supercomputer design: bare dies, gold post interconnects, 3D stacking, immersion cooling — jwt0625 · 2026-10-28