AWS HyperPod adds workload recovery and infra checks to boost training goodput

AWS ML Blog · rss · 2026-11-19

AWS introduced three new SageMaker HyperPod capabilities targeting failures where "every node looks healthy but training silently breaks," raising training goodput.

Why: Existing checks inspect one node's hardware at a time and miss three failure classes — numerical faults (GPUs passing hardware tests but computing wrong results; Meta attributed 6 of 419 unexpected interruptions in its 54-day Llama 3 405B pre-training run to silent data corruption), hung jobs (NCCL timeout or deadlock leaves GPUs occupied while Slurm still shows "running"), and cluster-level failures (e.g., crash-looping K8s device plugins, with errors that never mention the culprit).

New capabilities:

Economics: a 32-instance p5.48xlarge (256 H100) pre-training job hanging at 1 AM and noticed at 9 AM wastes 2,048 GPU-hours ($16,900 at $66/instance-hour); a 15-minute hang-detection rule cuts the cost to $530, saving $16,400.

Original post →

More from Infra

Infra channel →