Qwen3.8-27B hits 134 TPS on RTX 3090 with DFlash2 and custom optimizations
iamMess · reddit · 2026-08-20
The author pushed Qwen3.8-27B inference on an RTX 3090 to 134 TPS (default sampling) and drastically reduced long-context turn latency from 23s to 1s.
Key Optimizations:
- DFlash2 Drafting: Integrated a 5-layer block drafter predicting 7 tokens per pass. Requantized to Int4 (1.19GB) to fit within 24GB VRAM without quality loss.
- Lookup-augmented Drafting: A custom Triton kernel scans token history to reproduce repeated content (e.g., quotes, code), boosting tokens/step by 29% on reproduction tasks.
- Prefix Caching: Enabled caching for the Mamba/GDN hybrid model, slashing 24k-token follow-up latency from 23s to 1s.
- Memory Efficiency: Fixed allocator logic to reduce KV cache per token from 105KB to 78KB for 64k context.
Performance:
- Single User: 134 TPS.
- 64 Concurrent: 942 TPS.
- Quality: Perplexity 8.09, GSM8K 96.5%, with exact speculative decoding.
Related event: Qwen3.8-27B Inference Hits 381 TPS on RTX 3090(2 posts)→
More from Infra
- Electric Flight Cost Calculated at $40, Signaling End of Fossil Fuels — examachine · 2026-08-21
- Swiss town of 2,000 quietly builds a 400MW data center — ZeroStateReflex · 2026-08-21
- Debunking Misleading Claims About Data Center Water Usage — AndyMasley · 2026-08-21
- OpenAI is hiring for "reducing risk of opposition" to its data centers — dinabass · 2026-08-21
- Ornith 1.5 35B Q5 runs locally inside GitHub Copilot on a Mac M3 Max — DanWahlin · 2026-08-21
- Economic Computing Cloud for AI Emerges: Training, Inference, Routing in Focus — rhythmrg · 2026-08-21