torch.compile with FSDP Dynamic Shapes Degrades MFU

Developers training the Flux.2 klein image model found that torch.compile skips installing guards on FSDP-wrapped modules under dynamic shapes, as FSDP1 legacy logic still applies to FSDP2. This generates extra symbolic parameters, hindering fusion of elementwise ops like LayerNorm and RoPE and lowering MFU.

2026-08-24 ~ 2026-08-24 · 3 related posts

• PyTorch Compilation Pitfall: FSDP Guard Skipping Hurts MFU — SeunghyunSEO7 · 2026-08-24
• torch.compile skips guards for FSDP, reducing MFU in dynamic shape training — SeunghyunSEO7 · 2026-08-24
• Legacy FSDP logic hinders operator fusion and slows training — SeunghyunSEO7 · 2026-08-24