LLM Tool-Call Judges Reject Correct Values Without References: The Derivation Gap
Linhai Ma · hf · 2026-10-15
A paper from The Fin AI, Yale, and Cardiff University studies how LLM judges verify tool-call arguments proposed by agents.
- Key finding — the "derivation gap": judges accept a correct value less often when their input lacks a reference showing where the value comes from — e.g., schema defaults, inferred values, or truncated source text.
- The gap appears across all 7 judges tested, and is not a truncation artifact: correct default values show the largest gap on a no-truncation benchmark.
- Controlled source-relocation test: judges favor a visible incorrect value over a correct value whose source is hidden — they effectively follow the reference.
- DeRef fix: a separate model reads the full history without seeing the candidate and writes a reference for the judge. Across 3 benchmarks it improves argument AUROC for all judges; on the controlled test it lifts AUROC from chance level to 98.40–99.59.
More from coding & agent
- Dev builds unhinged Gulag-themed front-end for Papercliping that show-trials failing agents — jamesbrooksco · 2026-10-15
- Agent trained on community archive tweets posts autonomously, rebuilds its own persona nightly — lfschiavo · 2026-10-15
- Shopify CEO launches shop skill that turns any agent into a personal shopper; sellers urged to test if AI recommends them — msg · 2026-10-15
- Epic's Lore vs Anthropic's Scion: two post-Git, centralized VCS architectures compared — ssh4net · 2026-10-15
- Epic vibe-coded its new games launcher, claims it's faster than Steam — Ars Technica AI · 2026-10-15
- Creator uses coding agents + Google Maps 3D Tiles to place billboard art on real buildings — matanSF · 2026-10-15