Meta loses a node every 5 hours; topology-aware scheduling recovers 27% more GPUs
skypilot_org · x · 2026-10-30
SkyPilot's blog analyzes NVL72 (Grace Blackwell) cluster scheduling: on Meta's 2,304-node Prometheus cluster, a node goes down roughly every 5 hours and each rack fails about every 27 days.
Key points:
- NVL72 puts all 72 GPUs of a rack on one NVLink fabric; only rack-local jobs get the bandwidth. Multi-node jobs spread across racks fall back to InfiniBand and run up to 43% slower.
- Hardware failures and staggered job completions scatter free capacity across racks, eroding rack-locality over time.
- SkyPilot's approach — topology-aware placement, in-place recovery from GPU failures, and automatic defragmentation — yields about 27% more usable GPUs from the same racks.
Takeaway: Grace Blackwell clusters need a new generation of NVLink-topology-aware schedulers.
More from Infra
- Native 3K MiniMax H3 video on a 4090: one ComfyUI node kills the VRAM crash — friedlc · 2026-10-30
- Baseten partners with Postman to serve enterprise agents for 40M developers — baseten · 2026-10-30
- AI bubble datapoint: debt financing for AI datacenters is getting harder — Terminator857 · 2026-10-30
- Ollama says enterprise teams worldwide are moving workflows to open models — ollama · 2026-10-30
- Rogers Expands Starlink Satellite Roaming to Japan, Australia and New Zealand — XFreeze · 2026-10-30
- Corey Quinn flags macOS update silently restoring the removed eMudhra root cert — kingslyj · 2026-10-30