The simple self-testing loop behind every RL data company, per practitioners
teodorio · x · 2026-10-30
michaelrbock argues the "secret" behind today's RL env/data companies is a simple loop: generate synthetic benchmarks, let the model self-test, and iterate on failures — jokingly calling it the algorithm for RSI.
teodorio confirms he uses exactly this approach deploying agents for companies: have Claude test agents against a synthetic benchmark, then improve runtime data or instructions at failure points, adjusting models only if needed. A practitioner-backed signal that synthetic-eval-driven iteration is becoming standard agent engineering practice.
More from coding & agent
- 1,656 commits in a month mocked as slop — CodeSandbox CEO jokes his own image is ruined — zeeg · 2026-10-30
- Self-Direction turns agent mistakes into LoRA lessons, lifting GAIA success from 38.8% to 55.8% — rohanpaul_ai · 2026-10-30
- The best enterprise architecture today is knowing when not to use an LLM — kashifmanzoor · 2026-10-30
- Dev outsources competitor ad research to a phone call agent built on Cartesia Managed Agents — fekdaoui · 2026-10-30
- Every's 3 rules for running one Slack agent for the whole company — every · 2026-10-30
- As agents write nearly all new code, teams are shifting risk to customers — julianweisser · 2026-10-30