Cross-Datacenter RL: Modal Shrinks 500GB Weight Syncs to 500MB
AI Engineer · youtube · 2026-08-11
Nan Jiang from Modal detailed an engineering optimization for cross-datacenter Reinforcement Learning (RL). In RL training, shipping a 500GB checkpoint to rollout fleets in other regions takes minutes to hours, stalling weight updates.
Core Finding: Fewer than 1% of served weights actually change between consecutive versions. This isn't due to gradient sparsity (99% get non-zero gradients), but because of the gap between the precision floor of low-precision serving and the tiny Adam step size, an effect called Adam absorption.
Implementation:
- The trainer ships a roughly 500MB lossless patch, allowing the rollout engine to reconstruct bitwise identical weights.
- Lower precision serving like FP8 sharpens this absorption (e.g., GLM 4.7 Air saw only 0.15% weight change on the first step).
- By making a lossless patch the unit of sync instead of a full checkpoint, rollout fleets can decouple from the trainer's cluster and utilize scattered GPUs across regions. Modal's implementation is called Stitch.
More from Infra
- Microsoft Plans to 'Significantly' Increase Production of Next-Gen AI Chips — thoefler · 2026-08-11
- fal Signs 3 Hot GenAI Model Companies, Expands H200 and B300 Capacity — gorkem · 2026-08-11
- Alphabet Aims to Raise $25B in Bonds to Fund AI Infrastructure Buildout — Beth_Kindig · 2026-08-11
- Nvidia reportedly partners with financial firms to finance $500B in AI chips and data centers — Polymarket · 2026-08-11
- Fixing Intel Arc VRAM Issues: Developer Releases Stability Patch for ComfyUI — Valuable-Subject-274 · 2026-08-11
- Oz-FP4: Emulating FP64 DGEMM on Low-Precision FP4 Tensor Cores — teortaxesTex · 2026-08-11