Qwen3.8-27B hits 134 TPS on RTX 3090 with DFlash2 and custom optimizations
iamMess · reddit · 2026-08-20
The author pushed Qwen3.8-27B inference on an RTX 3090 to 134 TPS (default sampling) and drastically reduced long-context turn latency from 23s to 1s.
Key Optimizations:
- DFlash2 Drafting: Integrated a 5-layer block drafter predicting 7 tokens per pass. Requantized to Int4 (1.19GB) to fit within 24GB VRAM without quality loss.
- Lookup-augmented Drafting: A custom Triton kernel scans token history to reproduce repeated content (e.g., quotes, code), boosting tokens/step by 29% on reproduction tasks.
- Prefix Caching: Enabled caching for the Mamba/GDN hybrid model, slashing 24k-token follow-up latency from 23s to 1s.
- Memory Efficiency: Fixed allocator logic to reduce KV cache per token from 105KB to 78KB for 64k context.
Performance:
- Single User: 134 TPS.
- 64 Concurrent: 942 TPS.
- Quality: Perplexity 8.09, GSM8K 96.5%, with exact speculative decoding.
Related event: Qwen3.8-27B Inference Hits 381 TPS on RTX 3090(2 posts)→
More from Infra
- Nvidia MGX Power Rack session packed to standing room on final OCP day — firstadopter · 2026-10-16
- Intel hosts 816B-param Muse Spark Open air-cooled on four 160GB D-series GPUs — BenBajarin · 2026-10-16
- Running an 816B-parameter model on two Mac Studios: $40k instead of $500k of compute — charles_irl · 2026-10-16
- Filecoin marks 6 years of mainnet, pivots storage stack to AI workloads with new S3 offering — juanbenet · 2026-10-16
- PrismML ships Bonsai 2 ternary models: 9B fits in 2GB for local AI — gajesh · 2026-10-16
- Artificial Analysis to add 4-bit formats and Splash Metal engine to agent benchmark — ArtificialAnlys · 2026-10-16