Pushing Qwen3.8-27B to 124 tps on a single RTX 3090
iamMess · reddit · 2026-08-19
A developer demonstrated pushing Qwen3.8-27B inference to 124 tps (greedy) on a single RTX 3090 through extreme optimization. Key techniques include:
- Draft Vocabulary Optimization: Built from model's own outputs, increasing coverage from 92% to 97.5%, significantly reducing rejection sampling.
- Operator Quantization & Customization: GPTQ-int4 quantization for lmhead and MTP (+0.6% PPL); custom Split-KV attention kernel addressing the SM bottleneck in verify step (5-10x speedup).
- Sampler Patch: Sort-free top-k/top-p and multi-block softmax improved default sampling performance.
- KVarN KV Cache: Enables 262k context (at the cost of 20% slower long-context decode).
All optimizations maintain exact output distribution. The code is open-sourced.
Related event: Qwen3.8-27B Inference Optimized for RTX 3090(2 posts)→
More from Infra
- NVIDIA Details Qwen3.8-2.4T Deployment on GB300, Achieving >4K Tokens/s per GPU — PyTorch · 2026-08-21
- SpaceX launch cadence could enable 12-50 GW of space compute — teortaxesTex · 2026-08-21
- Local AI Coding Hardware Tiers: $1k Gets You the Smartest Model — nickbaumann_ · 2026-08-21
- Cerebras officer Sean Lie files to sell $153M in shares as IPO retail buyers get crushed — firstadopter · 2026-08-21
- Fable launches enterprise safeguards running on your infrastructure for data control — trq212 · 2026-08-21
- IOTA SN9 tests decentralized training of 16B model on mixed GPU clusters — bittingthembits · 2026-08-21