Qwen3.8-27B hits 381 tok/s on a single RTX 3090 via inference software alone
Redo · x · 2026-11-06
A developer reports running Qwen3.8-27B at 381 tokens/sec on a single RTX 3090 (24GB), up from an 82 tokens/sec baseline — a 4.6x gain with no hardware or model changes, achieved purely through inference software.
The optimization ladder:
- Baseline: 82 tokens/sec
- Optimized MTP (speculative decoding): 114 tokens/sec
- DFlash2 + context lookup: 138 tokens/sec
- Optimized lookup verification: 381 tokens/sec
Each stage attacks a different bottleneck: speculative decoding breaks sequential token generation, while context lookup and optimized verification further cut per-token compute.
A separate inference setup is in development to run Qwen3.8-27B on 16GB NVIDIA cards (RTX 5060 Ti/5070 Ti/5080 and older) while keeping memory-hungry features like large context windows and MTP speculative decoding — instead of forcing the model in via aggressive Q2 quantization. Linux and Windows support planned. The key insight: model weights are only part of VRAM usage; the KV cache grows with context length.
More from Infra
- Kimi K3.1 Pro gets Day-0 TPU support: serving a 2.8T-param MoE with llm-d on TPUs and GPUs — vllm_project · 2026-11-06
- Satirical pitch: cool data centers with fluorinert and market them as 'water-free' — typedfemale · 2026-11-06
- US stocks lost $3.8T in a day; chipmakers took $1.15T of the hit — CharlieMuirhead · 2026-11-06
- 'This is why we need at least a datacenter/home': the case for local compute — tech__unicorn · 2026-11-06
- Reddit hopes for cheap RAM by Christmas as memory prices stay in focus — keepthepace · 2026-11-06
- J.B. Huang's new video explains how LLM serving actually works, from fast to cheap — jbhuang0604 · 2026-11-06