Open-source governor avoids PyTorch OOMs on 8GB GPUs by dynamically downshifting batch size
uBazzyZ- · reddit · 2026-09-11
Frustrated by random CUDA OOM crashes while training small models locally on an 8GB RTX 5060 Ti, the author built a lightweight Python governor around PyTorch, MEM Orchestrator (open source):
- Monitors VRAM headroom during training, dynamically throttling micro-batch size and raising gradient accumulation steps before hitting OOM, then stepping back up when pressure drops.
- Saves atomic checkpoints with SHA-256 validation so crashes never corrupt weights.
Branches: main is the Linux core with DeepSpeed ZeRO integration; refactor/architecture-and-portability is a decoupled native PyTorch runner, cross-platform (tested on Windows).
The author ran a 1M-step endurance test on a 130M model and stress-tested a 255M model with injected memory spikes without crashing, with under 0.5% overhead. Code and 38 unit tests are on GitHub; feedback from PyTorch allocator/DeepSpeed folks welcome.
Related event: Open-Source Memory Orchestrator Stops CUDA OOM on 8GB GPUs(2 posts)→
More from coding & agent
- Ten lessons from three years building agents for real production work — garrytan · 2026-09-11
- Devin's New Model Verdict: Not a Benchmaxxer, a 'Killer Execution Model' at $20/Month — brandon_galang · 2026-09-11
- Shopify CEO Tobi Lütke hails single-dev open-source agent harness Pi — aakashgupta · 2026-09-11
- Shopify CEO Tobi Lütke Calls Pi, a Solo-Dev Open-Source Agent Harness, 'the Most Interesting' One — aakashgupta · 2026-09-11
- Dev explains why MCP won him over: organic UX beats telling agents to run CLI commands — zeeg · 2026-09-11
- Sentry Founder: MCP Won Because Agents Can Just 'Fix This URL', Not Call CLIs — zeeg · 2026-09-11