Meta paper: only 50-60% of recommendation training time actually trained before optimizations
_reachsumit · x · 2026-10-02
A Meta team has published an arXiv paper on fleet-scale inefficiency in recommendation system training, where their largest workloads process tens of billions of examples daily on thousands of GPUs. Before the work, only 50-60% of end-to-end wall time advanced training on new data.
Key contributions:
- Effective Training Time (ETT%): an operational framework to instrument lost time, attribute it to independently owned infrastructure components, and expose work repeated across job restarts
Optimizations span the full training stack: communication elimination and pipeline overlap during trainer initialization, dynamic-shape handling and autotuning pruning, reusable PyTorch 2 compilation caches, asynchronous checkpointing, standalone model publishing, and reduced recovery cost. The team evaluated each optimization on representative models and measured fleet-wide ETT% improvements.
More from Infra
- Investor: market hates uncertainty more than money after Nvidia earnings — MartinGTobias · 2026-11-19
- Microsoft and Intel deepen Azure collaboration: EMIB packaging, Diamond Rapids Xeons, 2.3x CXL gains — ryanshrout · 2026-11-19
- SGLang ships Day-0 serving support for new model DaisyRose-27B — ying11231 · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- DaisyRose-27B's in-house sandbox: ~80% cheaper than cloud providers, 99.95% success rate — BanghuaZ · 2026-11-19
- Tencent's 220B Hy4-flash quantized to 49 GB weights, runnable on 64 GB VRAM — gladkos · 2026-11-19