Qwen 3.6 27B VLLM Inference Benchmarks: BF16/FP8/NVFP4 Compared
live4evrr · reddit · 2026-07-05
The author tested the VLLM inference performance of three quantization formats (BF16, FP8, NVFP4) for Qwen 3.6 27B on an RTX 6000 Pro Blackwell 96GB GPU. NVFP4 is the fastest but suffers from looping output issues in agent mode. BF16 offers the highest response quality, while FP8 provides the best overall cost-performance ratio. The tests also compared VLLM with llama.cpp: VLLM is faster due to its paged attention mechanism and more stable than llama.cpp (which suffers from random errors requiring prompt resets). The test system used Ubuntu 26.04, vLLM 0.24.0, and CUDA 13.2.
More from Infra
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- PyTorch Day Korea 2026 launches first offline conf, CFP closes Sept 13 — PyTorch · 2026-09-11
- Local LLM server dilemma: 4x CMP-170HX (price up 53% in 20 days) vs Mac Studio M5 Ultra — rumboll · 2026-09-11
- llama.cpp lands Flash Attention tuning for RDNA4, big prefill gains on AMD — pmttyji · 2026-09-11
- Your p99 latency benchmark may be lying: a deep dive into coordinated omission — Franc0Fernand0 · 2026-09-11