Postmortem: Fake Timeouts and OOM in a Production Fine-tuning Job
Reasonable_Royal_621 · reddit · 2026-08-25
A detailed postmortem of a production fine-tuning job that ran for 10 hours despite a 1-hour timeout. The root causes were: 1. Redis Queue's signal-based timeout cannot preempt a process stuck inside a single uninterruptible C-level tensor op (like PyTorch's self-attention), so the timeout never fired; 2. Uncapped maxlength caused quadratic memory growth in self-attention, leading to OOM. The fix involved capping maxlength to 128 tokens and reducing batch size.
More from coding & agent
- AutoScientist extends to auto-configuring reinforcement learning recipes — sarahookr · 2026-08-25
- Clairvoyance turns icons into intelligent data objects for AI agents — draginol · 2026-08-25
- Hister: Your own personal semantic search engine for browser history — asciimoo · 2026-08-25
- Ponytail: Makes your AI agent think like the laziest senior dev — DietrichGebert · 2026-08-25
- Anthropic releases official directory for Claude Code Plugins — anthropics · 2026-08-25
- App Store Connect CLI 4.9.0 Released With a Long List of Changes — rudrank · 2026-08-25