Custom llama.cpp build pushes 7900XTX to 1600tk/s prefill on Qwen 27B Q8
nasone32 · reddit · 2026-09-08
A Reddit user released a custom llama.cpp build deeply optimized for AMD 7900XTX (single or dual card), targeting usable speeds for Qwen models on consumer ROCm hardware.
Headline results:
- Qwen 3.8 Next Q3KXL: 920tk/s pp8192 on two cards, 24-27tk/s prose, 40+tk/s code
- Qwen 3.8 27B Q80: 1600tk/s pp8192, 60-65tk/s prose, 100+tk/s code with tensor parallel — even with one card behind the chipset on PCIe x4
- Qwen 3.6 27B Q4KM single card: 1020tk/s pp8192
Key techniques include a custom HIP allreduce path enabling tensor parallel for chipset-behind cards, optional Q80 compression of inter-card PCIe traffic, --adaptive-mtp, DFLASH2 on tensor parallel, and unmerged upstream PRs (lazy PLE load path +58.88% Flash, GPU MoE expert cache +19.95% decode, etc.).
Tested on Ubuntu 24 with ROCm 7.14. The author offers no support, hoping the patches get merged upstream so the "frankenstein" build can die peacefully.
More from Infra
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- NVIDIA Plans ~$2,000 Per GPU Per Year Software Fee for Running AI Agents — LinusEkenstam · 2026-10-26
- 12 DevOps concepts to learn in order, from Linux to SRE — _jaydeepkarale · 2026-10-26
- Killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: lessons from multi-agent loops — OpenComplaint524 · 2026-10-26
- Gemma 4 26B-A4B runs offline on a 16 GB Mac: 33/40 on TypeScript tasks at 31 tok/s — Mission-Sir-1614 · 2026-10-26
- How thousands of GPUs train one AI: a breakdown of 5D parallelism — jbhuang0604 · 2026-10-26