A GLM 5.2 serving rig reaches 47 tok/s/user after async pipeline tuning
TheZachMueller · x · 2026-07-21
A cursed GLM 5.2 rig hits 47 tok/s/user after async pipeline tuning
The author describes a highly customized serving setup for GLM 5.2 NVFP4 that ran through 6.4M tokens over 10h47m of optimization.
- Starting point: a conventional 6-GPU multi-node vLLM baseline at about 20 tok/s/user.
- They moved to an asymmetric TP2→TP4 topology with a minimal inter-node pipeline boundary.
- A 30/48 layer split plus async PP/NCCL work improved throughput while preserving exact outputs.
- Final result: 47.1 tok/s/user at T=1 and 42.7 tok/s/user at T=4, with 169 tok/s aggregate at four users.
The attached chart shows the step-by-step gains from baseline to final peak.
Related event: Extreme Topology Tuning Boosts LLM Inference Throughput to 47 tok/s(2 posts)→
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11