Why sequence padding bloats training systems: pipeline parallelism is to blame
青稞AI · wechat · 2026-09-12
A deep-dive post-mortem on why post-training frameworks become slow and inelegant, tracing the rot to pipeline parallelism and sequence padding. Covers the bubble-fighting lineage (GPipe to DualPipe), why RL post-training's dynamic inputs break fixed schedules (RLHFuse, PipelineRL, StreamRL), why padding to [B,S] is unnecessary given FlashAttention varlen and Orca-style packing, torchtune's FlexAttention workaround and its complexity/memory costs, and real bugs like Unsloth's gradient-accumulation loss error and Megatron-Core's FLOPs overestimation caused by padding.
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28