A GPU running 5% slow is fine for inference but catastrophic for training: why health checks invert
AccBalanced · x · 2026-09-06
A widely shared ops insight: the same GPU running 5% slow is fine for inference but catastrophic for training, so provisioning and health checks are nearly inverted across the two workloads.
- Training is synchronous and tightly coupled: thousands of GPUs step through all-reduce together, so the slowest card sets the pace. One straggler, flapping NVLink, or memory-error GPU taxes the whole job. Health work happens before launch — burn-in, collective bandwidth tests, thermal margin and link checks, pulling marginal nodes. A mid-run node death rolls back to the last checkpoint, costing every GPU hours of progress.
- Inference flips this: replicas are loosely coupled and mostly stateless, so ops becomes elastic scaling rather than pre-launch vetting.
Takeaway: admission-style health checks for training clusters, elastic scheduling for inference — don't mix the two.
More from Infra
- KV cache often spills out of HBM in the agentic era, tanking effective bandwidth — AccBalanced · 2026-09-06
- Hybrid bonded HBM hypothetical market: over 3 billion D2D applications per year — zephyr_z9 · 2026-09-06
- Ollama CEO: open models will carry 80-90% of enterprise tokens at just 10-20% of cost — victor_explore · 2026-09-06
- Nvidia de-specced Rubin Ultra HBM from 12-Hi to 8-Hi: $/bandwidth is the bottleneck — AccBalanced · 2026-09-06
- Hot Chips 2026: Irrational Analysis publishes investment-driven recap — jwt0625 · 2026-09-06
- New method predicts transformer training divergence before the run starts — burkov · 2026-09-06