Hugging Face Transformers Adds Activation Checkpoint Offload to Reduce GPU Memory
StasBekman · x · 2026-09-01
Hugging Face Transformers has merged a new feature adding offload capabilities to gradient checkpointing.
Key Functionality:
- Moves checkpointed activations, which usually reside in GPU memory, to pinned host memory (CPU).
- This is extremely useful for enabling long contexts or running larger models with fewer GPUs without using parallelism, freeing up a significant amount of VRAM.
- Can be enabled via gradientcheckpointingkwargs={"offload": True} in TrainingArguments.
Technical Details:
- It wraps PyTorch's saveoncpu(pinmemory=True), following the model's existing path without requiring per-model support.
- The author notes that if implemented with full overlap, this could be a zero-cost performance feature and questions why PyTorch hasn't implemented it natively.
More from Infra
- AWS Launches AgentCore, Accepting Multi-Cloud Reality — DavidLinthicum · 2026-09-01
- Shopify's Continual Learning Loop Cuts Serving Costs by 96% — yenkel · 2026-09-01
- Shopify Engineer Details Continual Learning for GraphQL Agent — Drewch · 2026-09-01
- Inference provider Wafer raises $40m at $200m+ valuation, rejecting offers — steph_palazzolo · 2026-09-01
- x402 Standard: Building a Native Payment Layer for the Agentic Economy — kleffew94 · 2026-09-01
- Custom HBM drives 5x LLM inference speed boost — BenBajarin · 2026-09-01