Optimizing Qwen3.8-27B: From 9.5 to 153 Tokens/Second
TrifleHopeful5418 · reddit · 2026-08-21
Through 159 experiments, the author optimized Qwen3.8-27B on a heterogeneous setup (AMD Strix Halo + RTX 3090 Ti), boosting generation speed from 9.5 to 153 tok/s (32K code context) and outperforming a dual-3090 vLLM cluster on HumanEval.
Key Optimizations:
- Chat Template Swap: Switching to the terser Qwen-Sharp template reduced wall time by 44% and output tokens by 51% by stopping model narration.
- KV Cache Format: Downgrading K and V to q40 freed 2GB VRAM, allowing all full-attention layers to sit on the fast GPU.
- Speculative Decoding: The flag --spec-type draft-mtp,ngram-mod delivered +72% to +140% speedup on code-shaped output at no VRAM cost.
- llama.cpp Patch: Adjusting --spec-draft-ubatch freed 1GB VRAM for better layer placement.
- Vision Encoder Offload: Using MTMDBACKENDDEVICE=Vulkan1 moved the vision encoder to the idle iGPU to prevent OOM on the 3090 Ti.
Failed Approaches:
- External draft models were slower due to PCIe latency.
- Q6K quantification increased wall time and slightly reduced accuracy.
Benchmarks:
- HumanEval: Local 159/164 (29.7 min) vs Remote vLLM 157/164 (42.4 min).
- Long Context Retrieval: 15/15 passed locally with faster execution.
More from Infra
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16
- vLLM ships Day-0 LoRA RL for Muse Spark Open, lifting AIME from 0.56 to 0.85 — vllm_project · 2026-10-16
- DSpark draft model for Muse Spark Open proposes 7 tokens per pass with 23-42% acceptance — vllm_project · 2026-10-16