AWS shows NVRx fault-tolerant FSDP training on EKS: sync checkpointing ate up to 40% of wall time

AWS ML Blog · rss · 2026-09-17

AWS ML Blog details a full recipe for fault-tolerant distributed training with NVIDIA Resiliency Extension (NVRx) on Amazon EKS.

The problem: large jobs run for days across dozens of nodes, so faults are statistically inevitable. A single GPU fault cascades through NCCL timeouts and kills healthy workers, while synchronous checkpointing blocks all ranks on I/O — up to 40% of total wall time at the cluster sizes tested.

Three independently adoptable NVRx layers (pip-installable, no changes to model or training code):

Async checkpointing: TorchAsyncCheckpoint hands the state dict to a background process; each rank writes its own FSDP shard directly, removing the rank-0 bottleneck
In-process restart: inprocess.Wrapper survives soft faults — aborts the process group, runs per-rank health checks (GPU/NVLink/NIC), re-rendezvouses survivors and resumes from the latest checkpoint without killing the interpreter or CUDA allocator
ftlauncher in-job restart: handles hard faults (SIGKILL, OOM) via heartbeat timeouts, then respawns fresh workers in the same job

Infrastructure: self-managed p5.48xlarge node groups (8× H100 each), EFA at 3,200 Gbps for NCCL, and FSx for Lustre co-located in the same Availability Zone to cut checkpoint-read latency during recovery. Includes benchmarks at 2- to 8-node H100 scale with reproducible code.

Original post →

AWS shows NVRx fault-tolerant FSDP training on EKS: sync checkpointing ate up to 40% of wall time

More from Infra