Meta traced 6 of 419 Llama 3 405B training interruptions to silent data corruption — here's how to detect it
AWS ML Blog · rss · 2026-11-12
AWS ML Blog details detecting and recovering from silent data corruption (SDC): GPUs return wrong-but-plausible numbers with no error, which flow into every replica's gradients and weights while loss looks normal for hours. Meta attributed 6 of 419 unexpected interruptions in Llama 3 405B pretraining (up to 16K H100s, 54 days) to SDC — and those were only the caught ones.
Why existing checks miss it: ECC guards memory, not the ECC-less tensor-core math path; Xid errors require the GPU to self-report; deep health checks run on idle nodes; transport and checkpoint checksums verify delivery, not computation.
Three in-loop checks (the only layers that catch truly silent SDC):
- Non-finite gradient check: catches NaN/Inf/overflow at the step it happens;
- Ring replay audit: every GPU recomputes another node's micro-batch, comparing loss bit-for-bit and gradient fingerprints;
- Weight fingerprint comparison: flags replicas whose post-update weights differ from the majority.
Localization and recovery: map suspect rank to a Kubernetes node, cordon it, corroborate with DCGM metrics and a deep health check, replace the node, resume from a pre-window checkpoint. Validated with fault injection on 4 ml.p5.48xlarge instances (32 H100s); code is open source.
More from Infra
- Gratis launches: a local Mac AI assistant with no internet or API fees, built on Bittensor SN65 — markjeffrey · 2026-11-12
- ComfyUI update doubles Qwen LLM token rates on consumer GPUs, up to +251% on RTX 5090 — Lexius2129 · 2026-11-12
- 2.9bpw quantized DeepSeek V4.1 Flash on 6x3090 scores 58.3% on APEX 2025, beating GPT-5.4 — hp1337 · 2026-11-12
- HipKittens turns one: breaking AMD's 'CUDA moat' for a multi-silicon future — simran_s_arora · 2026-11-12
- 'Dear MacKenzie Scott, please send $8M for GPUs' becomes an AI meme — AIFlow_ML · 2026-11-12
- Uno uses discrete diffusion for up to 3x lossless LLM inference speedup, talk Nov 13 — burny_tech · 2026-11-12