PyTorch Maintainer: SAC Can Be Applied at Finer Granularity Than Whole Blocks

PyTorch maintainer ezyang observed that sparse activation checkpointing is conventionally applied at transformer block granularity, but can be refined down to individual residual additions, applying to norm+attn/mlp instead of whole blocks.

2026-09-26 ~ 2026-09-26 · 2 related posts

• ezyang: sparse autoencoders can be applied per residual add, not per block — ezyang · 2026-09-26
• PyTorch's ezyang: SAC should target norm+attn/mlp, not just the block — ezyang · 2026-09-26