GLM agent built its own inference infra in two weeks, tripling end-to-end throughput
jietang · x · 2026-09-17
Zhipu engineer jietang shares how GLM-5.3-Flash went from first run on domestic accelerators to serving all production traffic in two weeks, with 3.2× end-to-end throughput — with most optimization done by an Infra Agent powered by GLM-5.3 itself.
The constraints were tough: limited memory and interconnect bandwidth, 1M-token context, multimodal requests, and an immature stack with missing kernels. Key trade-offs: compute for memory (ReplaySSM), communication for memory (intra-node tensor parallelism), precision for capacity (mixed INT8/FP8/BF16 caching), and disaggregation for scheduling freedom (Encode–Prefill–Decode).
The most important lesson wasn't any single optimization but dense feedback: local correctness tests, execution traces, microbenchmarks, and end-to-end measurements enabling targeted hypothesis testing instead of relying on aggregate metrics alone.
More from coding & agent
- Indie dev uses Devin to build a Lemon Squeezy-Patreon hybrid with multi-payment support — CtrlAltDwayne · 2026-09-17
- Devin coded autonomously for 10.5 hours, using just 2% of weekly quota — CtrlAltDwayne · 2026-09-17
- Multica launches open-source Kanban for agent teams, supporting 20+ coding tools — jiayuan_jy · 2026-09-17
- Hands-on with Jev: a classifier model to replace LLM-as-a-judge and route agents — doesdatmaksense · 2026-09-17
- Dev Inspired by AgentConf Keynote to Run All Agents Locally on Home Blackwell GPU Stack — TejasKumar_ · 2026-09-17
- 102 inventory photos to 22 live eBay listings: why right answers for wrong reasons fail validation — Ok_Appearance_7559 · 2026-09-17