InfiniBand vs RoCE on two cloud B200 nodes: 4x throughput gap measured
TheZachMueller · x · 2026-11-20
- Zach Mueller benchmarked two on-demand Lambda HGX B200 nodes in the same region (RoCE Ethernet, no InfiniBand), running an NVFP4-quantized GLM model with TP16 across 2 nodes, 8k in / 1k out.
- Results: InfiniBand peaks above 1,000 output tokens/s vs 250-300 for RoCE; p95 TTFT at high concurrency is 700s+ on RoCE vs 200s on IB; p95 ITL exceeds 1,400ms on RoCE vs 600ms on IB.
- Takeaway: plain Ethernet-connected on-demand nodes can run multi-node inference, but with a 4x throughput penalty—fine for cost-sensitive, latency-tolerant workloads.
Related event: Cloud B200 Test: RoCE Throughput Falls Far Short of InfiniBand(2 posts)→
More from Infra
- Self-hosted cloud GPUs beat Higgsfield and Runway APIs on cost and flexibility — Al_Grigor · 2026-11-21
- NVIDIA GTC DC lineup: Ian Buck, Catanzaro and Lebaredian sessions on infra, agents, physical AI — nvidia · 2026-11-21
- KV cache economics: why a $200 Claude plan can hand out $14,000 of API-equivalent tokens and still profit — rachittshah · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Nvidia's rivals are vying to be the 'Central Bank' of AI, says investor — GavinSBaker · 2026-11-21
- Benchmarking 11 open models' post-GQA attention: up to 139x less memory at 128K — bariskasikci · 2026-11-21