Pushing Qwen3.8-27B to 124 tps on a single RTX 3090
iamMess · reddit · 2026-08-19
A developer demonstrated pushing Qwen3.8-27B inference to 124 tps (greedy) on a single RTX 3090 through extreme optimization. Key techniques include:
- Draft Vocabulary Optimization: Built from model's own outputs, increasing coverage from 92% to 97.5%, significantly reducing rejection sampling.
- Operator Quantization & Customization: GPTQ-int4 quantization for lmhead and MTP (+0.6% PPL); custom Split-KV attention kernel addressing the SM bottleneck in verify step (5-10x speedup).
- Sampler Patch: Sort-free top-k/top-p and multi-block softmax improved default sampling performance.
- KVarN KV Cache: Enables 262k context (at the cost of 20% slower long-context decode).
All optimizations maintain exact output distribution. The code is open-sourced.
Related event: Qwen3.8-27B Inference Optimized for RTX 3090(2 posts)→
More from Infra
- Phison CEO calls CXMT memory scare a 'ghost story', warns of materials shortage — BenBajarin · 2026-10-16
- Ben Bajarin: The Future of Compute Is Fungible as Hyperscalers Design for Workload Flexibility — BenBajarin · 2026-10-16
- AirLLM runs frontier-size models layer-by-layer: 671B on 12GB VRAM — Aiden_Tech_Ai · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Q4-quantized coder model on 8-core CPU builds interactive solar system demo in one shot — IntrepidMindExplorer · 2026-10-16
- TokenSpeed MLA 2 ships for Vera Rubin: up to 3.44× faster Kimi K3 decode — zhyncs42 · 2026-10-16