GLM-5.2 hits 12.2 tok/s on 16 AMD MI50s with llama.cpp RPC

Legal-Ad-3901 · reddit · 2026-07-23

A user reports GLM-5.2 UD-Q4KXL GGUF running on a 16× AMD MI50 32GB setup via llama.cpp RPC.
On a real 10.7k-token document, it reached 12.2 tok/s output and 30.9 tok/s input; with two parallel requests, aggregate throughput was 14.5 tok/s.
At 10.7k context, output stayed at 10.2 tok/s with coherent long-form generation.
The cluster uses 2× 8-GPU nodes, 512 GB total VRAM, 100W cap per card, and 10 GbE DAC interconnect.
The author says earlier attempts with a custom vLLM-gfx906 build hit decent speed but degraded past 10k context, and they hope someone can get the Moby Dick repo working better with it.

More from Infra