Zhipu says a GLM-5.3 agent optimized its own inference stack, 3.2x throughput on 100k domestic chips
SinclairWang1 · x · 2026-09-17
Zhipu revealed that all production inference for GLM-5.3-Flash now runs on over 100,000 domestic AI accelerators, with 3.2x end-to-end throughput gains achieved in under two weeks — and much of the optimization was done by a GLM-5.3-powered Infra Agent, a model optimizing the very system that serves it.
Hard constraints, constant trades
- Domestic accelerators have limited memory/interconnect bandwidth and an immature software stack with missing kernels and guesswork documentation
- The model must serve 1M-token context and multimodal requests
- Every optimization was a trade: compute for memory (ReplaySSM), communication for memory (intra-node tensor parallelism), precision for capacity (mixed INT8/FP8/BF16 caching), architecture separation for scheduling freedom (encode-prefill-decode disagg)
- Per-token cost on domestic chips now matches mainstream NVIDIA GPUs
The key method: dense feedback
CEO Jie Tang notes agents rarely get stuck because they can't write code — they get stuck because "throughput dropped 20%" doesn't say which layer broke, what to verify, or what to test next: a sparse-reward credit-assignment problem, with each end-to-end benchmark run taking hours. Zhipu made senior engineers' implicit "process rewards" explicit via layered verification interfaces: correctness (is it right), system behavior (where time goes), and performance (which approach wins under what conditions) — each signal local, cheap to obtain, and objectively verifiable.
With this, the agent found and fixed three real engineering bugs, including TF32 rounding-error drift accumulating through chained state-matrix merges in the KDA kernel's context-parallel path; the fix is merged into the open-source Flash Linear Attention project.
More from coding & agent
- Dev builds 3D universe explorer with 20 galaxies, 767 planets using Codex GPT-6.1 Sol — OpenAIDevs · 2026-11-02
- ChatGPT, Claude and Perplexity's fetchers don't run JavaScript, leaving client-rendered sites invisible to AI — gaganghotra_ · 2026-11-02
- SGLang boosts multimodal inference >10% by replacing GPU memory bookkeeping with a dict lookup — charles_irl · 2026-11-02
- kivi: hold Fn, talk, and it types polished text anywhere on your Mac — itsOmSarraf_ · 2026-11-02
- A base LLM plus 11 lines of code is all you need for a decision model — helloiamleonie · 2026-11-02
- Fix codebase structure before asking AI agents to write tests, says engineer — arpit_bhayani · 2026-11-02