AWS shows NVRx fault-tolerant FSDP training on EKS: sync checkpointing ate up to 40% of wall time
AWS ML Blog · rss · 2026-09-17
AWS ML Blog details a full recipe for fault-tolerant distributed training with NVIDIA Resiliency Extension (NVRx) on Amazon EKS.
The problem: large jobs run for days across dozens of nodes, so faults are statistically inevitable. A single GPU fault cascades through NCCL timeouts and kills healthy workers, while synchronous checkpointing blocks all ranks on I/O — up to 40% of total wall time at the cluster sizes tested.
Three independently adoptable NVRx layers (pip-installable, no changes to model or training code):
- Async checkpointing: TorchAsyncCheckpoint hands the state dict to a background process; each rank writes its own FSDP shard directly, removing the rank-0 bottleneck
- In-process restart: inprocess.Wrapper survives soft faults — aborts the process group, runs per-rank health checks (GPU/NVLink/NIC), re-rendezvouses survivors and resumes from the latest checkpoint without killing the interpreter or CUDA allocator
- ftlauncher in-job restart: handles hard faults (SIGKILL, OOM) via heartbeat timeouts, then respawns fresh workers in the same job
Infrastructure: self-managed p5.48xlarge node groups (8× H100 each), EFA at 3,200 Gbps for NCCL, and FSx for Lustre co-located in the same Availability Zone to cut checkpoint-read latency during recovery. Includes benchmarks at 2- to 8-node H100 scale with reproducible code.
More from Infra
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02