MEM v3 Open-Sourced: Dynamic Batch Scaling Against CUDA OOM, Survives +10GB VRAM Shocks
uBazzyZ- · reddit · 2026-09-27
The author open-sourced MEM v3, a "memory governor" for PyTorch that tackles CUDA out-of-memory crashes during long training and fine-tuning runs by watching VRAM and throughput in real time and dynamically adjusting batch size and gradient accumulation without stopping the process.
Key features:
- Dynamic lane switching: scales batch size up or down in milliseconds based on actual GPU memory pressure, instead of keeping batch sizes overly small out of caution
- Chaos resistance: tested against sudden +10GB VRAM allocation shocks without crashing
- Crash-proof checkpoints: atomic file replacement with SHA-256 checks across rotating slots, so power outages won't corrupt saved weights
- Live telemetry: built-in local web dashboard tracking loss, throughput, and lane switches
A zero-setup Colab demo is available; repo: nobazzy/mem-llm-orchestrator.
More from Infra
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16