Lambda benchmark: RoCE delivers ~70% lower throughput than InfiniBand on B200 nodes
TheZachMueller · x · 2026-11-21
TheZachMueller rented two same-region on-demand Lambda nodes and benchmarked interconnect impact, citing Lambda's official comparison: at 250 concurrent users on 2x HGX B200 nodes, a RoCE deployment delivered about 70% lower throughput and 2.5x slower time-to-first-token than NVIDIA Quantum InfiniBand. Controlled variables: same model (nvidia/GLM-5.2-NVFP4), same serving stack (SGLang TP16/DP16/EP16), same benchmark (AIPerf, 8192 in / 1024 out tokens) — the interconnect was the variable, though he notes it wasn't the only factor behind the gap.
Related event: Cloud B200 Test: RoCE Throughput Falls Far Short of InfiniBand(2 posts)→
More from Infra
- DayOne data center IPO on hold as investors doubt AI infrastructure valuations — SumitGup · 2026-11-21
- Cathie Wood predicts orbital compute will undercut terrestrial data centers within two years — XFreeze · 2026-11-21
- Space data centers face a financing problem: lenders don't want collateral in orbit — TechCrunch AI · 2026-11-21
- Self-hosted cloud GPUs beat Higgsfield and Runway APIs on cost and flexibility — Al_Grigor · 2026-11-21
- NVIDIA GTC DC lineup: Ian Buck, Catanzaro and Lebaredian sessions on infra, agents, physical AI — nvidia · 2026-11-21
- KV cache economics: why a $200 Claude plan can hand out $14,000 of API-equivalent tokens and still profit — rachittshah · 2026-11-21