Flex-RSI benchmark tests whether agents can self-improve across 18 frontier models
_vztu · x · 2026-11-15
Flex-RSI benchmarks whether AI agents can get better at their jobs by rewriting their own code, tools, prompts and workflows (weights frozen): 18 frontier models, 24 tasks spanning robot control, 3D reconstruction, reasoning and visual creation, 188 runs and nearly 3,800 rounds of self-improvement.
Key findings:
- Building reveals more than answering: gains were largest when agents built programs, robot policies or pipelines — models starting from identical solutions diverged dramatically
- Feedback quality is the bottleneck: open-weight models progressed with graded feedback but stalled on binary success/failure signals; on RoboCasa, closed models posted median validation gains of 51.2 points, well ahead of open ones
A rare systematic measurement of the ceiling of agent self-improvement.
More from coding & agent
- Claude Code builds a browser mech game in 2 days, with AI autopilot playtesting — vinishkapoor · 2026-11-15
- Codex user reports a full day of chat history vanished with no way to recover it — SecretEmployee7612 · 2026-11-15
- Dev builds an MCP server that lets Claude auto-generate DraftKings and FanDuel lineups — dfsstudio · 2026-11-15
- Open-source Cowork orchestrator falls back to local models when Claude rate-limits — Less_Release_1977 · 2026-11-15
- Claude wrote every line of an acid techno track; human ears were the fitness function — ColossalCosci · 2026-11-15
- Cursor runs unattended for 1+ hours: watching processes and auto-fixing failures — NickPassig · 2026-11-15