Qwen 3.8 27B Decoding Speed Tripled to 92.9 tok/s via Optimization
TheMoonMidas · x · 2026-08-23
Demonstrated a performance optimization result for the Qwen 3.8 27B model, where decoding speed increased significantly from 26 tok/s to 92.9 tok/s.
- Impact: Decoding throughput improved by approximately 3.5x.
- Requests: Users are calling for further optimizations on other parts of the model, including KV cache quantization/kernels, Metal verify kernels, adaptive draft schedule, and the prefill path.
This indicates that local LLM inference experience can be significantly enhanced through low-level kernel optimization.
Related event: Qwen3.8-27B Long-Context Decoding Speeds Up Over 3x(2 posts)→
More from Infra
- Nvidia MGX Power Rack session packed to standing room on final OCP day — firstadopter · 2026-10-16
- Intel hosts 816B-param Muse Spark Open air-cooled on four 160GB D-series GPUs — BenBajarin · 2026-10-16
- Filecoin marks 6 years of mainnet, pivots storage stack to AI workloads with new S3 offering — juanbenet · 2026-10-16
- PrismML ships Bonsai 2 ternary models: 9B fits in 2GB for local AI — gajesh · 2026-10-16
- Artificial Analysis to add 4-bit formats and Splash Metal engine to agent benchmark — ArtificialAnlys · 2026-10-16
- NVIDIA RTX Spark laptop runs local agent workload 2x faster than MacBook Pro M5 Pro — ArtificialAnlys · 2026-10-16