RTX 3090 runs Qwen3.8-27B at 672 tps via extreme quantization
iamMess · reddit · 2026-08-17
The author achieved peak speeds of 672 tps for Qwen3.6-28B on an RTX 3090 through extensive optimization. Key techniques include W4A16 quantization, FP8 KV cache, and int8 conversions for lmhead and embedtokens, reducing VRAM usage to 14.2GB. The setup achieves 82 tps for single requests and sustains 417 tps with 64 concurrent requests, outperforming ninfer by 17% to 149%. The solution runs on vLLM with a reported quality loss of approximately 0.6%.
Related event: Qwen3.8-27B local benchmarks: from RTX 4080 Super to RTX 6000 Pro(7 posts)→
More from Infra
- Musk says he wants 'maybe more than 10GW' of compute; Nvidia now counts SpaceX as a CSP and co-designs a space server — elonmusk · 2026-10-16
- Jensen Huang: Musk's AI buildout moved from K's to M's, now a CSP-tier NVIDIA customer — elonmusk · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16
- Meta's Muse Spark Open scores 39 vs 48 for closed Spark 1.3 on AA Intelligence Index — EAccelerate_42 · 2026-10-16
- LLM spots useless 95GB-per-frame memcpy in ffmpeg dblur, cutting runtime from 2452ms to 49ms — IgorCarron · 2026-10-16
- DeepSeek-V4-Flash on 4x Radeon AI PRO R9700: the 'refused' 4-card config hits ~79 tok/s — brainchillzZ · 2026-10-16