Qwen 3.6 27B VLLM Inference Benchmarks: BF16/FP8/NVFP4 Compared
live4evrr · reddit · 2026-07-05
The author tested the VLLM inference performance of three quantization formats (BF16, FP8, NVFP4) for Qwen 3.6 27B on an RTX 6000 Pro Blackwell 96GB GPU. NVFP4 is the fastest but suffers from looping output issues in agent mode. BF16 offers the highest response quality, while FP8 provides the best overall cost-performance ratio. The tests also compared VLLM with llama.cpp: VLLM is faster due to its paged attention mechanism and more stable than llama.cpp (which suffers from random errors requiring prompt resets). The test system used Ubuntu 26.04, vLLM 0.24.0, and CUDA 13.2.
More from Infra
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- Anthropic S-1: CSP marketplace revenue jumped from 11% to 48% in three years — AccBalanced · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Cuts Cloudflare Bindings CPU Time by Up to 166× Via Profiling — samgoodwin89 · 2026-10-27