Three silent PyTorch bugs in AI-generated code that run fine but quietly hurt results
lmoroney · x · 2026-10-10
Laurence Moroney points out that the worst ML bugs never throw an error, making AI-generated training code hard to audit. He lists three common PyTorch training-loop mistakes:
- Forgetting zerograd(), so gradients keep accumulating across batches
- Skipping model.eval() before evaluation, leaving dropout on and batch norm using batch statistics
- Applying softmax before nn.CrossEntropyLoss, which already expects raw logits
All three run fine while quietly giving worse numbers. His sanity check for any loop, human- or model-written: train on a handful of examples until loss drops near zero — if the model can't memorize ten examples, something in the loop is broken.
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23