PyTorch DevLog: Why You Should Never Free Pinned Memory
ezyang · x · 2026-08-10
PyTorch core contributor Edward Yang shared a deep technical dive into the mechanics of CUDA pinned memory, a post drafted by Claude and edited by the author.
The article argues that despite traditional memory management practices, you should never actually free pinned memory in steady-state high-performance workflows. The key reasons include:
- CUDA Graphs constraints: A captured graph bakes the host buffer's address into its nodes, meaning any participating pinned buffer must stay alive at that exact address as long as the graph is replayed.
- High reallocation cost: Calling cudaHostAlloc costs milliseconds. Freeing memory in steady state just means paying this expensive overhead again on the next iteration.
- Fixed working sets: Workloads that heavily utilize pinned memory (like LLM training/inference) typically use the same tensor shapes every step, so releasing buffers between steps saves no actual memory.
The author notes that major LLM frameworks like Megatron-LM and vLLM already implement this by allocating dedicated pinned pools upfront and holding them until the process exits, with some even offering config flags specifically to prevent pinned buffers from being freed.
More from Infra
- 7 Engineering Bottlenecks Slowing Down Your AI Applications — goyalshaliniuk · 2026-08-10
- Report: Nvidia Qualifies 300mW Lasers, Buys Bulk of Supply — zephyr_z9 · 2026-08-10
- Dual-GPU Optimization Speeds Up MiniMax-H3 Video Generation 8x — multimodalart · 2026-08-10
- Rust Linear Algebra to Wasm Achieves 6x Browser AI Performance Boost — doodlestein · 2026-08-10
- France's 10GW Power Surplus Could Yield €350B Annually via AI Datacenters — emmanuelvivier · 2026-08-10
- Won 5th Place in GPU Mode with Coding Agents, No CUDA Background — tokenbender · 2026-08-10