Qwen3.8-27B hits >70 tok/s and full 262k context on 2x3090 with vanilla vLLM
maqifrnswa · reddit · 2026-09-22
On 2x3090s, a user benchmarks Qwen3.8-27B-INT4 (RedHatAI quant) with vanilla vLLM: >70 tok/s single-stream decode, 200 tok/s at 3-4 concurrency, >10k tok/s prefill, and full 262k context. Key settings: INT4 quant ideal for Ampere, weighted fp8 KV cache (near-identical to bf16), MTP speculative decoding (3 tokens), tensor parallel 2, prefix caching. Tuned over a month using vLLM metrics + Prometheus and vllm bench serve; full serve command posted. The author argues the overlooked RedHat quant is ideal for Ampere.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12