PyTorch's ezyang: SAC should target norm+attn/mlp, not just the block
ezyang · x · 2026-09-26
PyTorch core developer ezyang discusses Saved Activation Checkpointing (SAC) granularity in transformers.
- SAC is conventionally applied at the transformer block level.
- He observes many SAC policies end up always saving the residual-add outputs, which are the sole inputs into attn/mlp, so SAC-ing them individually works.
- Follow-up correction: since norms are usually recomputed, it's more accurate to say you SAC the norm + attn/mlp rather than attn/mlp alone.
More from Research
- Anthropic: Claude solves nine-loop scattering amplitudes, breaking the eight-loop record — AnthropicAI · 2026-09-26
- Greenblatt Warns Latent Reasoning Architectures Would Sharply Raise Misalignment Risk — anmarasovic · 2026-09-26
- Bilevel optimization densifies scarce labels to fix OOD molecular property prediction (ICML WS 2025) — CatAstro_Piyush · 2026-09-26
- Blender-generated datasets teach world models object permanence at 10k samples per task — _akhaliq · 2026-09-26
- Benchmark maker says no Ascend version — models would hill-climb it; TPU/Trainium/AMD better — xeophon · 2026-09-26
- RBC Borealis Details ATOM, Its Large Transaction Model Wired to LLMs via Agentic Workflows — VectorInst · 2026-09-26