RSIBench-Data benchmarks whether agents can do data-centric research
rohanpaul_ai · x · 2026-07-29
RSIBench-Data tests whether agents can do data-centric research
The paper introduces RSIBench-Data, a controlled benchmark for recursive self-improvement focused on data-centric post-training research.
- It fixes the post-training stack so agents are evaluated on research decisions rather than optimization, serving, or systems work.
- Agents must iteratively revise training-data strategies for a fixed target model using feedback from official evaluation runs.
- The benchmark covers six tasks across software engineering, terminal use, scientific QA, and mathematics.
- Four frontier agents improved on the first valid attempt in 58.33% of settings, but performance was inconsistent.
- Among searches that continued past their best score, 78.26% ended with a worse final attempt, suggesting good candidate strategies often appear early or mid-run.
More from coding & agent
- 10 open-source repos that let your AI agent reverse-engineer apps and games — victor_explore · 2026-10-16
- The new CTO job: agent management and QA loops, 'cable management' instead of coding — HankYeomans · 2026-10-16
- Why company AI agents go stale: adoption needs in-house knowledge, not just better agents — EdenEmarco177 · 2026-10-16
- Open-source MCP lets coding agents search Obsidian's 9,000-plugin directory before reinventing wheels — pitfa19 · 2026-10-16
- Cheng Lou open-sources Handwired: hand-written weights turn code into training-free neural nets — dotey · 2026-10-16
- Building a portable context vault in Notion so every AI agent knows you from day one — ivanhzhao · 2026-10-16