Qwen3.8-27B on Strix Halo: MTP gives 3.1x speedup, ROCm FP4 fork hits 29.9 tok/s
deepu105 · reddit · 2026-08-23
A rigorous local inference benchmark of Qwen3.8-27B on a Strix Halo laptop (Radeon 8060S, 128GB unified memory), using llama.cpp's own timing block, 3 reps per config, with 85W boost verified:
Headline results:
- MTP (multi-token prediction) works: Q80 jumps from 7.3 to 22.4 tok/s (3.1x) at 73% draft acceptance
- The julianmb/q38rocm FP4 fork reaches 29.9 tok/s (4.1x over stock baseline) at 84% acceptance, using only 14.6GB vs 29GB for Q80
- ROCm vs Vulkan decode within 1%, but Vulkan prefill lags (209 vs 277 tok/s), so ROCm stays the default
Key observations:
- Draft length 5 (Q80) / 6 (FP4) beats the backend default of 3, and the optimum doesn't shift with context length
- MTP's payoff shrinks as context fills: 3.1x on empty window → 2.2x at 64k → only 1.15x at a full 256k (KV cache makes verification costlier; acceptance holds)
- Prefill roughly quarters from empty to 256k (277→75 tok/s); the FP4 fork falls off faster and drops below stock Q80 by 128k (70 vs 119) — its edge is decode, not prompt processing
- Didn't help: KV cache q80 (slightly worse at draft 5), flash attention off (-5%)
- Caveat: the FP4 fork's completion lengths drift across draft settings (152-160 tokens for the same prompt)
Runs driven via the author's LlamaStash tool with reproducible flags.
More from Infra
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16
- Shadeform to Announce Six New Features on Oct 19: Lower GPU Costs, More Suppliers — _jaydeepkarale · 2026-10-16
- Fastino launches Training API for SFT/RL, claiming 25-85% cost cut vs Tinker — kalyan_kpl · 2026-10-16
- Dallas Fed: AI data center debt issuance equals ~1/8 of Treasury duration supply — _onionesque · 2026-10-16
- Running Wan2.1 14B FP16 on a 12GB laptop GPU: VRAM swap stress test finishes in 4m11s — RYuu-1120 · 2026-10-16