Skipping failed pipeline stages during training: near-baseline loss at 1% failure rate
covenant_ai · reddit · 2026-09-22
Templar's latest work on their Crucible distributed pre-training platform targets fault tolerance: data-parallel replicas plus pipeline parallelism, with SparseLoCo exchanging compressed updates across replicas and pipeline compression across stage boundaries. The key idea is stage skipping — when an inner stage goes offline, activations and gradients bypass it for multiple steps so healthy stages keep processing tokens instead of waiting.
Simulation setup: a 178M model, 8 replicas, 4 stages each, 1% per-replica failure probability per global step, each outage removing a stage for 6 steps. Validation loss stayed close to the no-failure baseline. Fixed projections shared across layers further improve robustness with pipeline compression, hypothesized to align representations across stage boundaries and make bypasses less disruptive.
The authors argue this points toward training on broader, less reliable compute pools (e.g., spot instances), while noting this simulates learning effects, not physical worker replacement or production savings.
More from Infra
- DigitalOcean Managed Agents enters public preview: idle pausing, 75+ models, one bill — HeyAmit_ · 2026-09-23
- Reka EdgeQ VLM Runs Natively on Snapdragon 8 Elite NPU With 0.73s First Token — RekaAILabs · 2026-09-23
- Cache-Friendly Context Compacting Plugin for OpenCode Cuts Local Model Time from 10min to 1-2min — schennardo · 2026-09-23
- RTX 3060 Test: ByteShape Ultra-Low-Bit Quant Far Behind GSQ Despite Paper Similarity — zyxciss · 2026-09-23
- CXL Disaggregated Memory Nears Commercial Deployment, Says Ben Bajarin After FMS Meetings — BenBajarin · 2026-09-23
- Sandboxing ComfyUI for AI agents in a VM with a single shared GPU remains unsolved on Windows — johnshedletsky · 2026-09-23