Together GPU Clusters adds auto repair: from GPU fault to healthy node with no human in the loop
togethercompute · x · 2026-10-28
Together AI detailed its fully automatic node remediation for GPU clusters, taking nodes from fault to healthy without human intervention.
- Passive health checks continuously monitor every node, inside the VM and on the underlying host
- Detects Xid errors, double-bit ECC errors, GPUs falling off the PCIe bus, and flapping InfiniBand links
- On fault detection, the system recommends the lightest fix with supporting evidence
- Users configure exactly which repairs run unattended, down to individual Xid codes
The capability launched earlier this year and is already used by cluster customers and Together's internal inference platform to minimize downtime for distributed training and inference.
More from Infra
- SK hynix Q3 revenue hits ~$78.8B, DRAM prices up high-20% QoQ, crushing estimates — BenBajarin · 2026-10-28
- SK Hynix and Micron both expect memory demand to outpace supply through 2030 — BenBajarin · 2026-10-28
- Liquid Inference serves 10B tokens/day, 88.7% of inputs are cache reads — AccBalanced · 2026-10-28
- Poll: UK Net Support for a Nearby Data Centre Sits at -13%, Only North East Positive — SamuelAlbanie · 2026-10-28
- Axionic Labs' RHFM mega kernel trains a model, predicts, and runs Doom on one GPU at once — tw_killian · 2026-10-28
- 8x P100 llama.cpp: Qwen3.8 Flash Q4 Throughput Swings Wildly With Slot Count — ILoveDangerousStuff2 · 2026-10-28