Hugging Face engineer: interconnect overhead means one big GPU beats multiple networked ones
TheZachMueller · x · 2026-10-29
TheZachMueller, an engineer at Hugging Face, argues that communication is the bane of both inference and training: RDMA, PCIe, even top-tier InfiniBand — splitting work across multiple GPUs or machines yields only a fraction of the throughput of a single beefier machine. The lone exception is NVLink, which he jokes he's "not speaking to these people right now" about. Key takeaway: prefer scaling up on one machine before scaling out over the network.
More from Infra
- Map tracks data centers building their own gas plants, four near Dallas ozone zone — AINowInstitute · 2026-10-29
- GitHub commits hit 7.38B in September, 5x a year ago as agents flood in — rohanpaul_ai · 2026-10-29
- Why Macs lose to DGX Spark at batched inference: bandwidth wins solo, FP4 compute wins crowds — QuixiAI · 2026-10-29
- ComfyUI reproducibility: the workflow JSON alone isn't enough — Im_bored_rn69 · 2026-10-29
- Akamai launches Akamai Inference and Agent Sandbox for enterprise AI deployment — lucaolivari · 2026-10-29
- Cloudflare Q3 DDoS report: 8,200 attacks/hour as short bursts surge 208%, mega attacks drop 69% — Cloudflare Blog · 2026-10-29