NYU lab open-sources Outerloop: autonomous research agents that open PRs only when benchmarks improve
mengyer · x · 2026-11-12
NYU CDS assistant professor Mengye Ren's lab released Outerloop, open-source autoresearch software where multiple AI agents propose ideas, run experiments on your cluster, and open pull requests only when a benchmark measurably improves, with negative attempts also recorded.
- Workflow: propose → experiment (parallel Slurm jobs) → measure (same-seed comparison) → verify → record
- On the gpt-speedrun task: 9472 → 8640 steps (-8.8%), 82 attempts, 770 GPU-hours, 4 agents, 0 human edits
- The accompanying preprint reruns three published autoresearch systems on new seeds and finds their gains shrink—part came from lucky runs; self-graded gains shrink too, while audited gains hold up
Available on GitHub, PyPI, with a paper and community channels.
More from coding & agent
- Microsoft on agentic schedulers: LLM proposes, deterministic code validates — WirelessLife · 2026-11-12
- Visual block scripting drives an AI opponent in a classic arcade game — ezshine · 2026-11-12
- Gradient Labs' headless agents embed AI customer service inside banks' own stacks — neal_lathia · 2026-11-12
- Microsoft ships agent harness package for Agent Framework Go, easing agent setup — adnan_hashmi · 2026-11-12
- Dev releases open coding SFT dataset for Kimi K3.1 Pro after LFM3 1.7B tune's surprise popularity — alexcovo_eth · 2026-11-12
- fabric-planning-skill: open-source Copilot skill and Python CLI for Microsoft Fabric Plans — adnan_hashmi · 2026-11-12