Pushing Qwen3.8-27B to 381 tps on a single RTX 3090
iamMess · reddit · 2026-08-21
The author shares updates on their hyper-optimized Qwen3.8-27B inference engine, achieving 382 tps for document reproduction on an RTX 3090. Key optimizations include:
- Longer verify blocks: Increasing verification to 16 tokens per step using a lookup drafter, achieving a 15/16 acceptance rate in document-quoting workloads.
- Breaking the 64K context limit: Using int8 KV cache and adjusting sliding window block sizes to expand context support from 69k to 138k tokens.
The author provides a honest correction, noting that quantized KV caches reduce MTP acceptance rates (dropping from 2.56 to 2.38), resulting in a 2.13x decode tax at 112k context. This mode is optimized for RAG and coding assistants that frequently quote context, rather than general chat.
Related event: Qwen3.8-27B Inference Hits 381 TPS on RTX 3090(2 posts)→
More from Infra
- Haight Street Compute Market: a one-day trading floor for GPU deals in SF — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- LMA brings day-0 model support to TPU for the first time, spanning NVIDIA, AMD, Intel via SGLang — ying11231 · 2026-10-16
- Semtech's analyst-day slide cleanly explains the four signal-integrity impairments — BenBajarin · 2026-10-16
- Fortytwo Engine boosts Qwen3.8-27B accuracy 7% and cuts tokens 25% via Swarm Inference — rohanpaul_ai · 2026-10-16
- Berkeley scholars: concentrated chip supply chain makes a hardwired frontier AI pause feasible — wfithian · 2026-10-16