A GLM 5.2 serving rig reaches 47 tok/s/user after async pipeline tuning
TheZachMueller · x · 2026-07-21
A cursed GLM 5.2 rig hits 47 tok/s/user after async pipeline tuning
The author describes a highly customized serving setup for GLM 5.2 NVFP4 that ran through 6.4M tokens over 10h47m of optimization.
- Starting point: a conventional 6-GPU multi-node vLLM baseline at about 20 tok/s/user.
- They moved to an asymmetric TP2→TP4 topology with a minimal inter-node pipeline boundary.
- A 30/48 layer split plus async PP/NCCL work improved throughput while preserving exact outputs.
- Final result: 47.1 tok/s/user at T=1 and 42.7 tok/s/user at T=4, with 169 tok/s aggregate at four users.
The attached chart shows the step-by-step gains from baseline to final peak.
Related event: Extreme Topology Tuning Boosts LLM Inference Throughput to 47 tok/s(2 posts)→
More from Infra
- A100 rental prices still rising in 2026 per a16z data, Jevons paradox hits compute — VraserX · 2026-10-27
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27