RSIBench-Data: Benchmarking AI Agents for Automated Data Iteration
cwolferesearch · x · 2026-07-31
The author introduces RSIBench-Data, a novel benchmark focusing on the data aspect of Recursive Self-Improvement (RSI). It fixes the training setup (based on Tinker) and tasks a coding agent with iteratively refining the training data for a separate LLM.
The Agent Workflow:
- Submit a training manifest and configuration.
- Execute model training.
- Evaluate on a dev set and get feedback (e.g., token usage, verifier outcomes, errors).
- Iteratively refine data strategy based on feedback (sourcing, filtering, mixing, or creating curriculums).
Current Performance & Challenges:
Frontier models currently struggle with this benchmark. While they often show early gains, they fail to achieve monotonic improvements and frequently degrade performance. The author notes the current setup mostly involves brute-forcing data mixtures for simple wins. Preventing decontamination is also tough, as agents could learn to search ArXiv for optimal mixing strategies.
Despite this, the author is optimistic about agents in data-centric research, suggesting that providing better tools (clustering, inspection, external LLM judges) could significantly enhance long-horizon data research.
More from coding & agent
- Coding agent runs are getting long — one dev suggests breathing, not parallel sessions — teropa · 2026-10-16
- Dev builds MCP layer so Claude Code can work on servers without ever holding SSH keys — NETMESAFE · 2026-10-16
- As coding agents run longer autonomously, one dev's tip: don't parallel-session, just breathe — teropa · 2026-10-16
- 'Refusing to code with AI in 2026 is like refusing Google in 2005' — prasenx · 2026-10-16
- Claude Code adds idle compaction to save 1h prompt cache on idle sessions — EricBuess · 2026-10-16
- Handling agents that crash mid-task: the hard part is the unknown outcome across the network — Responsible-Dare5139 · 2026-10-16