Optimizing Qwen3.8-27B: From 9.5 to 153 Tokens/Second
TrifleHopeful5418 · reddit · 2026-08-21
Through 159 experiments, the author optimized Qwen3.8-27B on a heterogeneous setup (AMD Strix Halo + RTX 3090 Ti), boosting generation speed from 9.5 to 153 tok/s (32K code context) and outperforming a dual-3090 vLLM cluster on HumanEval.
Key Optimizations:
- Chat Template Swap: Switching to the terser Qwen-Sharp template reduced wall time by 44% and output tokens by 51% by stopping model narration.
- KV Cache Format: Downgrading K and V to q40 freed 2GB VRAM, allowing all full-attention layers to sit on the fast GPU.
- Speculative Decoding: The flag --spec-type draft-mtp,ngram-mod delivered +72% to +140% speedup on code-shaped output at no VRAM cost.
- llama.cpp Patch: Adjusting --spec-draft-ubatch freed 1GB VRAM for better layer placement.
- Vision Encoder Offload: Using MTMDBACKENDDEVICE=Vulkan1 moved the vision encoder to the idle iGPU to prevent OOM on the 3090 Ti.
Failed Approaches:
- External draft models were slower due to PCIe latency.
- Q6K quantification increased wall time and slightly reduced accuracy.
Benchmarks:
- HumanEval: Local 159/164 (29.7 min) vs Remote vLLM 157/164 (42.4 min).
- Long Context Retrieval: 15/15 passed locally with faster execution.
More from Infra
- Kubernetes CPU Limits Make Apps Slow and Costly: Proof and Experiments — JeremyCMorgan · 2026-08-21
- Productionizing AI Apps: OpenTelemetry, On-Call Agents, and Full Observability Workflow — Al_Grigor · 2026-08-21
- LLMRouter 2.0: Unified Infrastructure for LLM Routing Dev and Eval — youjiaxuan · 2026-08-21
- Run MiniMax H3 locally on 12GB GPUs: 15-second multi-shot ComfyUI template — vortis23 · 2026-08-21
- llama.cpp adds tensor split for LFM2/MoE, boosting inference performance significantly — pmttyji · 2026-08-21
- Used RTX 3090 purchase review: Local AI performance crushes 3060, Qwen 35B 20x faster — Yanzihko · 2026-08-21