AWS HyperPod adds workload recovery and infra checks to boost training goodput
AWS ML Blog · rss · 2026-11-19
AWS introduced three new SageMaker HyperPod capabilities targeting failures where "every node looks healthy but training silently breaks," raising training goodput.
Why: Existing checks inspect one node's hardware at a time and miss three failure classes — numerical faults (GPUs passing hardware tests but computing wrong results; Meta attributed 6 of 419 unexpected interruptions in its 54-day Llama 3 405B pre-training run to silent data corruption), hung jobs (NCCL timeout or deadlock leaves GPUs occupied while Slurm still shows "running"), and cluster-level failures (e.g., crash-looping K8s device plugins, with errors that never mention the culprit).
New capabilities:
- MultiNodeTrainingValidation: a deep health check that trains a small Megatron-LM workload per-instance then cross-instance at provisioning (or on demand); NaN/infinite loss or timeouts fail the instance, which is labeled and can be auto-replaced.
- HyperPod Kubernetes Resource Monitoring Agent: continuously watches EKS clusters for crash-looping device plugins, missing DaemonSets, and taints, reporting involved pods/nodes on job failure.
- Hung job detection (new for Slurm): a per-node agent watches job output, GPU activity, or user-defined log patterns; it flags stalled jobs to CloudWatch and can cancel them for Slurm requeue.
Economics: a 32-instance p5.48xlarge (256 H100) pre-training job hanging at 1 AM and noticed at 9 AM wastes 2,048 GPU-hours ($16,900 at $66/instance-hour); a 15-minute hang-detection rule cuts the cost to $530, saving $16,400.
More from Infra
- Liquid models run locally in Mercedes-Benz cars; Shopify needs 20ms query rewriting — agihouse_org · 2026-11-19
- Liquid AI previews on-device continual learning within a fixed memory budget — agihouse_org · 2026-11-19
- Liquid AI keynote: bringing capable AI to the devices around us — agihouse_org · 2026-11-19
- Top 10% OpenAI researchers reportedly burn $25,000 of tokens a day — ivan_bezdomny · 2026-11-19
- Fchollet: latent token demand is near-infinite, cost is the only brake as RoI keeps improving — fchollet · 2026-11-19
- Running Hy4-Flash locally on 2x RTX 3090: Q3 quant hits PP 63 / TG 22 tokens per second — EmilPi · 2026-11-19