PyTorch Monarch on AMD GPUs Enables Fault-Tolerant Distributed Training
PyTorch · x · 2026-07-08
A PyTorch Foundation blog post details how contributors from AMD and Meta ported PyTorch Monarch to AMD Instinct GPUs (based on ROCm) to support large-scale, fault-tolerant distributed training.
The article outlines the ROCm porting of the Monarch GPU runtime and distributed communication stack. It demonstrates how Monarch, TorchFT, and TorchTitan allow healthy replicas to continue training while failed nodes recover and rejoin, eliminating the need for a full checkpoint restart.
Validation included training Llama 3 8B on a 128-GPU AMD Instinct MI300 cluster (SLURM) and a 256-GPU MI355 cluster (Kubernetes).
More from Infra
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- NVIDIA Claims HGX Rubin NVL8 Delivers 10x More Tokens per Megawatt Than B200 — thursdai_pod · 2026-10-27
- Together lets multiple LoRA post-training experiments share the same GPU capacity — togethercompute · 2026-10-27
- Commoditization dilemma: either human labor or GPUs become the commodity — _AustinCalvert_ · 2026-10-27
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27