Custom llama.cpp build pushes 7900XTX to 1600tk/s prefill on Qwen 27B Q8
nasone32 · reddit · 2026-09-08
A Reddit user released a custom llama.cpp build deeply optimized for AMD 7900XTX (single or dual card), targeting usable speeds for Qwen models on consumer ROCm hardware.
Headline results:
- Qwen 3.8 Next Q3KXL: 920tk/s pp8192 on two cards, 24-27tk/s prose, 40+tk/s code
- Qwen 3.8 27B Q80: 1600tk/s pp8192, 60-65tk/s prose, 100+tk/s code with tensor parallel — even with one card behind the chipset on PCIe x4
- Qwen 3.6 27B Q4KM single card: 1020tk/s pp8192
Key techniques include a custom HIP allreduce path enabling tensor parallel for chipset-behind cards, optional Q80 compression of inter-card PCIe traffic, --adaptive-mtp, DFLASH2 on tensor parallel, and unmerged upstream PRs (lazy PLE load path +58.88% Flash, GPU MoE expert cache +19.95% decode, etc.).
Tested on Ubuntu 24 with ROCm 7.14. The author offers no support, hoping the patches get merged upstream so the "frankenstein" build can die peacefully.
More from Infra
- Yacine urges building sovereign AI infrastructure or losing privacy and IP — yacineMTB · 2026-09-08
- Crusoe raises $3B at $30B valuation as it builds Stargate, delivered 200MW in 11 months — 快鲤鱼 · 2026-09-08
- CPU-Only LLM Tests: 35B MoE at Q2 Beats a 2B Model Despite Half the Speed — ML-Future · 2026-09-08
- HydraDB: a Rust graph database that lives entirely in S3, with nothing on disk — thisdudelikesAI · 2026-09-08
- Running Qwen 27B and Gemma 31B locally on one RTX 4090: quantization and context tradeoffs — MooseEfficient2151 · 2026-09-08
- Could distributed iPhones form an inference network? A 2AM open question — gajesh · 2026-09-08