IBM runs 753B MoE on 544 H100s with llm-d, serving 3,000 concurrent coding agents at 5-10x lower cost
dl_weekly · x · 2026-09-16
IBM Research and Red Hat benchmarked llm-d serving GLM-5.2, a 753B-parameter open-weight MoE (39B active), on 544 NVIDIA H100s for 3,000 concurrent coding agents, with 85.2% of input tokens served from cache — at 5-10x lower cost than commercial APIs. The post details why agentic workloads (massive context reuse, bursty parallel sub-agents) stress inference infra, and shows enterprises can run today's agentic traffic on existing H100 fleets.
More from coding & agent
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- "Coding isn't the bottleneck anymore — usage windows are," says founder Kyle Gawley — kylegawley · 2026-11-02
- Demo passed ≠ production-ready: 5 questions to answer before letting agents run your business — _kellhus · 2026-11-02
- An AI agent passed 100 test chats—still not ready to run the business — djgoel123 · 2026-11-02
- Agent given total creative freedom produces a 1:44 code-drawn animated short film — ivegotnoidea1 · 2026-11-02
- Meme: Wake Up, Use Claude Code, Repeat All Day — tekbog · 2026-11-02