Qwen3.8-27B on RTX 5090: NVFP4 Quantization Achieves 256 t/s Code Gen with 175k Context
pennyonaire · reddit · 2026-08-30
A detailed guide on running Qwen3.8-27B on a single RTX 5090 using a Blackwell-optimized recipe. The setup utilizes NVFP4 quantization (Q6 equivalent), the sglang engine, and DFLASH2 speculative decoding. Benchmarks show code generation speeds of 256 t/s (single slot) and 451 t/s (dual parallel slots), with prose generation at 144 t/s. The configuration supports a 175k context pool, leveraging a Host-RAM KV tier to resume 100k token conversations in roughly 1 second. Performance gains were achieved by quantizing the lmhead, optimizing KV caches with fp8, and using a GPU-managed RAM backend. The post also includes specific tuning parameters and evaluation scores (e.g., GPQA 84.8%). Additionally, the author shares a pro tip on using an admission proxy to manage request queues and prevent large requests from blocking parallel slots.
More from Infra
- llama.cpp NUMA mirroring boosts dual-EPYC inference by up to 137% — mattescala · 2026-08-30
- Autonomous Launches Personal AI Datacenter Hardware Starting at $26,100 — dee_hw · 2026-08-30
- Seeking the current best LLM inference setup for dual A100 GPUs — Theio666 · 2026-08-30
- Analysis: Meta's AI Infrastructure is Mispriced and Massive — RihardJarc · 2026-08-30
- FlashAccel: Leveraging High-Bandwidth Flash for LLM Inference — 9r4n4y · 2026-08-30
- Maximizing throughput: running parallel LLM instances on 2x V100s — Kike328 · 2026-08-30