AWS+HSBC paper co-evolves agent graders from just 10 labeled examples, matching ground truth
rohanpaul_ai · x · 2026-10-12
A paper from AWS and HSBC introduces Double Ratchet, which lets the verifier in a self-improving agent loop evolve alongside the agent's skills instead of relying on a hand-written rubric.
- How it works: the grader is an inspectable expression over small, mostly rule-based failure detectors, synthesized from clustered failures. New grader versions are selected only for agreement with a 10-item anchored reference set plus consensus on unlabeled outputs—never for raising the agent's score.
- Results: across code generation, enterprise text-to-SQL, and reference-free report generation, the loop retains 88–110% of the lift that ground truth or a rubric provides; on MBPP+ it gains +0.21 held-out agreement.
- Key warning: removing the anchor guards collapses the verifier into a vacuous always-pass grader—yet it trains skills just as well. Downstream task scores cannot certify a self-evolved verifier.
Related event: AWS and HSBC Paper Proposes Co-evolving Verifiers for Self-evolving Agents(2 posts)→
More from coding & agent
- Developers take note of ML Intern: 6 models trained in a week at ~$100 compute — Gradio · 2026-10-12
- Hugging Face's ML Intern turns a prompt into a trained model: 6 models in a week for ~$100 — Gradio · 2026-10-12
- VoiceStudio hits 59K stars: 10 exploding GitHub repos, from no-code to technical — PrajwalTomar_ · 2026-10-12
- yoagent: a Rust agent runtime that reuses pi extensions and DeepSeek Harness plugins — liyuanhao · 2026-10-12
- xses: a unified CLI to sync and resume AI coding agent sessions across 5 tools — Choice_Concern_5387 · 2026-10-12
- 15 role-based prompts claim to replace a $114,500 ML education, from theory to MLOps — bigaiguy · 2026-10-12