New RSIBench-Data benchmark shows agents improve model data pipelines, but often overshoot

imjustnewatai · x · 2026-07-29

A new benchmark finds frontier agents can improve data-centric research, but only if they stop at the right time

The post discusses RSIBench-Data, an arXiv paper on benchmarking LLM agents as data-centric researchers for recursive self-improvement.

Four frontier agents were tasked with improving training data for a fixed target model across six benchmarks.
In 14 of 24 settings, feedback helped agents beat their first valid attempt.
But continuing to search often hurt: among 23 searches that kept going after their best checkpoint, 18 ended worse at the final attempt, while the remaining 5 only matched the peak.
The paper’s scope is bounded: one fixed model, shared LoRA stack, and a controlled evaluation setup—not unrestricted self-rewriting.

The takeaway is that current agents can generate useful hypotheses, but they still struggle with scientific judgment: knowing when to preserve a good result and compound it instead of over-optimizing past it.

Original post →

New RSIBench-Data benchmark shows agents improve model data pipelines, but often overshoot

More from AGI Musings