AWS+HSBC paper co-evolves agent graders from just 10 labeled examples, matching ground truth
rohanpaul_ai · x · 2026-10-12
A paper from AWS and HSBC introduces Double Ratchet, which lets the verifier in a self-improving agent loop evolve alongside the agent's skills instead of relying on a hand-written rubric.
- How it works: the grader is an inspectable expression over small, mostly rule-based failure detectors, synthesized from clustered failures. New grader versions are selected only for agreement with a 10-item anchored reference set plus consensus on unlabeled outputs—never for raising the agent's score, guarding against reward hacking.
- Results: across code generation, enterprise text-to-SQL, and reference-free report generation, the loop retains 88–110% of the lift that ground truth or a rubric provides; on MBPP+ it gains +0.21 held-out agreement over the hand-authored seed.
- Key warning: removing the anchor guards collapses the verifier into a vacuous always-pass grader—yet it trains skills just as well. Downstream task scores cannot certify a self-evolved verifier.
Related event: AWS and HSBC Paper Proposes Co-evolving Verifiers for Self-evolving Agents(2 posts)→
More from coding & agent
- Work2Skill Turns Human GUI Demos into Reusable Agent Skills, Beating No-Skill Baselines — nex-agi · 2026-10-12
- UCSD paper bakes theory of mind into RL reward, letting a 7B agent model partners solo — rohanpaul_ai · 2026-10-12
- Indie maker shows Claude Code prompt that builds SEO free tools, driving 4M impressions — tibo_maker · 2026-10-12
- Codex and ChatGPT Work Hit 40M Weekly Active Users, On Track for 100M by March 2027 — FinanceYF5 · 2026-10-12
- Two misconceptions in the REA era: software isn't dead, coding is — AlchainHust · 2026-10-12
- Cloudflare open-sources Clef decision models, tops Jev Decision Index — emmanuelvivier · 2026-10-12