New Benchmark AA-AgentPerf: GB300 Hits 20x H200 Concurrent Agents per MW
新智元 · wechat · 2026-07-04
Independent evaluation agency ArtificialAnalysis released AA-AgentPerf, the industry's first inference benchmark designed specifically for AI agents, with "concurrent agents per megawatt" as its primary metric. Instead of feeding fixed-length synthetic requests, it replays real coding agent trajectories covering 12+ languages, up to 200 turns, and contexts reaching 130,000 tokens. It enables production optimizations like KV cache reuse and speculative decoding, locking in SLO service standards before testing maximum concurrency.
Nvidia's published AA-AgentPerf results show that under service standards of 20 and 60 tokens per second, the GB300NVL72's concurrent agents per megawatt are about 20 times those of the H200—for the same one megawatt of electricity, GB300NVL72 can handle roughly 61,400 concurrent agents, while H200 handles only about 2,600.
The article notes that agent workloads are relay-style long-chain calls; single-request benchmarks can no longer measure these chained invocations, tool waiting, and context bloat. For those spending real money on GPUs to build data centers, the real concern is how many working agents can be sustained per kilowatt-hour per GPU.
More from Infra
- MiniMax H3 runs fully on a Galaxy S26: 5-second video with audio in ~7 minutes — BidFunny4862 · 2026-10-27
- Unconventional AI acquires inference chip firm NeuReality in 1,000x efficiency bid — NaveenGRao · 2026-10-27
- Musk: robot RAM demand will dwarf everything else, and 100% of superintelligence compute goes to deep space — elonmusk · 2026-10-27
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27
- Wan2GP lands on Pinokio: 1-click AI video on 6GB GPUs — cocktailpeanut · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27