Forking: AutoResearch's training tricks trigger sudden overfitting, new paper
ZimingLiu11 · x · 2026-10-27
Ziming Liu presents Forking: Sudden Overfitting Under Replay (arXiv:2610.00394) at MetaCircle AI's Research Exposition: a training trick found by AutoResearch causes a sharp train/validation loss separation at epoch boundaries under data replay — dubbed "forking" — reproduced in NanoGPT and a DeepSeek-style model with Engram.
- Mechanism: repeated updates sharpen seen continuations while suppressing unseen ones; n-gram modules create weakly interacting context-specific subspaces amplifying the effect; low-frequency contexts drive most of the gap.
- Takeaway: forking joins grokking and double descent as a curious deep-learning phenomenon, and is an unpleasant byproduct of autoresearch agents' tricks — their many seemingly useful results demand human scrutiny, not worship.
42 pages, 22 figures, code released.
More from coding & agent
- OpenAI launches GPT-6.1 Terra: faster, cheaper computer-use model for the whole workday — airesearch12 · 2026-10-27
- Coding Agent Deleted a Production Database During Code Freeze: Anaconda's Governance Fix — anacondainc · 2026-10-27
- GPT-6.1 Terra Model Slug Spotted in APIs and Codex Changelog Before Launch — testingcatalog · 2026-10-27
- Claude Projects now lets you drag threads into the message box or @ mention them for precise context — dfeinition · 2026-10-27
- Jane Street's Yinsky shares tips for making codebases agent-friendly: types, property tests, formal methods — sh_reya · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27