Qwen 3.6 27B VLLM Inference Benchmarks: BF16/FP8/NVFP4 Compared
live4evrr · reddit · 2026-07-05
The author tested the VLLM inference performance of three quantization formats (BF16, FP8, NVFP4) for Qwen 3.6 27B on an RTX 6000 Pro Blackwell 96GB GPU. NVFP4 is the fastest but suffers from looping output issues in agent mode. BF16 offers the highest response quality, while FP8 provides the best overall cost-performance ratio. The tests also compared VLLM with llama.cpp: VLLM is faster due to its paged attention mechanism and more stable than llama.cpp (which suffers from random errors requiring prompt resets). The test system used Ubuntu 26.04, vLLM 0.24.0, and CUDA 13.2.
More from Infra
- WSJ: Nvidia is in talks to backstop about $250 billion of OpenAI's data center plan — KateClarkTweets · 2026-07-27
- YC talk on BCI x AI says infrastructure is what really determines speed — garrytan · 2026-07-27
- A 13B model ran on a no-GPU PC by paging weights from SSD via llama.cpp — ID_R_McGregor · 2026-07-27
- llama.cpp warns that GGUFs made before a recent change must be regenerated — EconomySerious · 2026-07-27
- RTX 5090 local tests show Qwen Q6 can drop to 15 tok/s at 80k context — LFAdvice7984 · 2026-07-27
- Surprising Ubuntu Setup: NVIDIA 5090 PC Becomes the Easiest AI Rig — _xjdr · 2026-07-27