NVIDIA's NeMo-DCR Cuts Trillion-Parameter RL Weight Sync from 87.5 min to 150s
nvidia · hf · 2026-10-07
NVIDIA presents NeMo-DCR (Delta-Compressed Refit), tackling the refit bottleneck in disaggregated agentic RL: transferring a full 1T checkpoint between AWS regions takes 87.5 minutes, yet only 1% of BF16 weights change per step.
Key designs: send only deltas while remaining bit-exact; fixed affine mappings project changes from training shards into checkpoint coordinates, with residual conversion and the serving runtime's native loader handling placement; compressible XOR masks and overwrites guarantee exact stored bits; in-place application with retries and a joint commit bind the policy to the next delta baseline; object storage or a relay tree streams payloads without cross-cluster collectives.
Even at 3-5% change rates, refits of 30B-1T models are 12-40x faster than full-checkpoint transfer; a 1T relay-tree refit at 3% takes 150 seconds instead of 87.5 minutes.
More from Infra
- Solo dev hits Together AI rate limits running parallel agents, seeks generous API credits — Correct_Positive_108 · 2026-10-07
- Ramjet: an open-source local alternative to NVIDIA Dynamo for multi-GPU inference — DoggoProfessor959 · 2026-10-07
- Qwen3.8 Flash Next GGUF benchmark: IQ3_S the sweet spot, 42.7M tokens tested — lxfater · 2026-10-07
- Used PS5 Pro hits $1,399 at GameStop as AI datacenters squeeze memory supply — aakashgupta · 2026-10-07
- Musk: xAI will build and run its Terafab itself, TSMC may only sublease part of it — MickeySteamboat · 2026-10-07
- "72% of the intelligence with 3.8% of the GPUs": Mistral's compute-efficiency ratio sparks debate — cyb3rops · 2026-10-07