Training engineer tip: save checkpoints to K8s PVC — and don't alert on that PVC
cto_junior · x · 2026-09-21
A hands-on ML infra tip replying to @arohan: always save training checkpoints to the PVC allotted in Kubernetes, but make sure you don't set capacity alerts on that PVC — if it fills up, it breaks the checkpointer itself and takes down training.
More from Infra
- Nautilo Guide: Stand Up a Self-Hosted AI Assistant Locally with Just Docker — Dan_Jeffries1 · 2026-09-21
- Qwen's new Flash model and Meituan LongCat both use N-gram to work around HBM shortage — bookwormengr · 2026-09-21
- How Meituan's LongCat lab trained a frontier model on Huawei 910C chips — bookwormengr · 2026-09-21
- Apple Approves MLX LLM Serving App in Under 10 Hours, Stunning 19-Year iOS Dev — alexcovo_eth · 2026-09-21
- Qwen papers suggest compute demand isn't going down anytime soon — TheZachMueller · 2026-09-21
- AI takes on additive manufacturing's hardest work: materials, parameters and defect analysis — burhop · 2026-09-21