Autoresearch for LLM pretraining needs humans in the loop; stacked changes are the killer
menhguin · x · 2026-10-12
Researcher menhguin shares hands-on experience running autoresearch for LLM pretraining, responding to iaindunning's observation that 'make this faster' works well while 'improve this frontier run' is deeply unimpressive. Key points: autoresearch strongly benefits from human-in-the-loop supervision and steering toward constraints; the killer is stacking combinatorial changes and hyperparameter sweeps — one tweak (e.g. sigmoid instead of softmax) may lower loss but its impact on logits, long context and post-training becomes unpredictable when combined. Designing a frontier-run improvement task means specifying desired state, mechanism and metric (with Exa search to anchor), but then it's not fully autonomous.
More from coding & agent
- Gemini audits an Audemars Piguet product page: title tags, hreflang, JS rendering and more — gaganghotra_ · 2026-10-12
- Claude Code tiered workflow: Haiku swarms, Sonnet builds, Opus reviews at 3 checkpoints — Arindam_1729 · 2026-10-12
- The Opus 5.5 Flywheel: Turning Every Agent Correction into Reusable Skills — ramagetime · 2026-10-12
- Burkov uses Codex to translate an 8-year-old Spanish microprocessors textbook into English with interactive AI-tutor exercises — burkov · 2026-10-12
- Grok plays WoW autonomously, now free and heading to Stormwind — djcows · 2026-10-12
- Insomnia: open-source tool keeps Mac agents running with the lid closed — _AustinCalvert_ · 2026-10-12