AQuA paper: if an agent can edit its tool schema, you're not evaluating the same system
creditme7 · reddit · 2026-08-28
A discussion of arXiv preprint AQuA (v2, peer review unverified) asks: if an agent can change its tool descriptions, valid arguments, retry policy, or metric feedback, are you still comparing the same agent even with fixed weights? AQuA's leakage taxonomy separates generation leakage (agents using unavailable future information) from selection leakage (repeated metric access during adaptive search).
AQuA's sealed sandbox keeps data splits, features, labels, and evaluators outside the editable surface. Part II uses a config DSL where one diff changes architecture/loss/sampler/optimizer while data path and evaluator stay fixed; the 2021–2025 US-equity window is reserved for final evaluation. The preprint admits isolation is a governance property, not a cryptographic barrier.
The reusable takeaway is an evaluation-contract manifest: modelhash, promptandplannerhash, toolschemahash, retryandroutinghash, datasplithash, featureandlabelhash, evaluatorhash, allowedchangeset, metricreadlog. Rule of thumb: anything that can alter the action distribution gets a new harness revision; every metric read enters the selection log.
More from coding & agent
- Anthropic tests 'Hub' feature for task orchestration in Claude — testingcatalog · 2026-08-28
- Weak RL may trigger communication traits hidden in pretraining — xuanalogue · 2026-08-28
- Tool call dispositions may generalize to unsanctioned agent communication — xuanalogue · 2026-08-28
- OpenAI agent message-leaving may emerge from pretraining priors — xuanalogue · 2026-08-28
- TraceDiff prototype debugs voice agents by diffing success vs. failure traces — Responsible-Dot8405 · 2026-08-28
- After Trying Every Coding Agent Setup, What Workflow Actually Survived — and What Does It Cost Per Month? — philzxx · 2026-08-28