Qwen3.8-27B on RTX 5090: NVFP4 Quantization Achieves 256 t/s Code Gen with 175k Context
pennyonaire · reddit · 2026-08-30
A detailed guide on running Qwen3.8-27B on a single RTX 5090 using a Blackwell-optimized recipe. The setup utilizes NVFP4 quantization (Q6 equivalent), the sglang engine, and DFLASH2 speculative decoding. Benchmarks show code generation speeds of 256 t/s (single slot) and 451 t/s (dual parallel slots), with prose generation at 144 t/s. The configuration supports a 175k context pool, leveraging a Host-RAM KV tier to resume 100k token conversations in roughly 1 second. Performance gains were achieved by quantizing the lmhead, optimizing KV caches with fp8, and using a GPU-managed RAM backend. The post also includes specific tuning parameters and evaluation scores (e.g., GPQA 84.8%). Additionally, the author shares a pro tip on using an admission proxy to manage request queues and prevent large requests from blocking parallel slots.
More from Infra
- Same 1,000 Cloudflare Durable Objects: $4,050/month cut to $22 with one trick — threepointone · 2026-10-16
- NVIDIA ships FP4-quantized Qwen3.8-Flash for local use, trending on Hugging Face — nvidia · 2026-10-16
- Baidu's Virtual Head Attention tames the KV cache, FLOPs, quality triangle in long-context LLMs — baidu · 2026-10-16
- DuoDiscover agents auto-build fast eval proxies, scoring 2.68x TPU kernel speedup — berkeley_ai · 2026-10-16
- Google Raises 2026 Capital Spending to Up to $205 Billion — emmanuelvivier · 2026-10-16
- Google open-sources AX, a Kubernetes-style orchestrator for billions of AI agent tasks — thisdudelikesAI · 2026-10-16