Flex-RSI preprint breaks down self-improving agents: editable harness, frozen weights, one-shot hidden test
krishnan · x · 2026-11-11
Krish Subramanian walks through the Nov 6 Flex-RSI preprint in an 8-post thread: 24 vision/robotics tasks, 18 models, all under fixed budgets.
- Two roles share one model: an executor improves the task solution; an improver rewrites the executor's prompts, tools and workflow plus its own instructions — the improvement procedure can edit itself
- What stays fixed matters: model weights never change; evaluator, data splits and budget accounting are locked; the agent cannot touch grading
- New solutions must beat the current one on held-out validation by a noise margin; final score comes from a hidden test split used once
- Editing the harness helped in 6 of 7 paired runs (gains of 2–40 points, one tie); evidence is thin — a repeat Opus run scored 40, not 80
- The commercial opening: services that improve at a recurring job without waiting for new models. Bottleneck is the loop: pass/fail feedback barely moved open-weight models; scored feedback helped every model
- Includes a pilot design: fixed vs editable harness, same budget, frozen then scored on hidden cases, accepted results per dollar, regression checks
More from coding & agent
- Anthropic's real bet is MCP connectors embedding HubSpot, Adobe and Atlassian inside Claude — petrusenko_max · 2026-11-11
- ZenML Pro launches always-on AI engineering team to keep agents in production — strickvl · 2026-11-11
- Latent Space tests NaiveAI's 2,100 tok/s coding model: fast, verbose, promising — Latent Space · 2026-11-11
- Anaconda's LLMOps guide: retrieval quality degrades independently of the model — anacondainc · 2026-11-11
- Dev demos agent with 'durable object brain' plus on-demand computer, local or remote — samgoodwin89 · 2026-11-11
- Right-sizing intelligence per role: frontier reasoning at the control plane, adversarial verifiers at the gate — clmt · 2026-11-11