Adobe turns eval reference answers into Python functions, lifting LLM-judge MCC from 0.331 to 0.427
dair_ai · x · 2026-09-17
Adobe researchers propose a new approach to agent evaluation: instead of storing a fixed reference answer per eval case — which goes stale when underlying data changes daily — they write each reference answer as a Python function.
- The function runs against the live system at eval time, so the expected answer follows the data, and upstream API changes make the test fail visibly
- An LLM judge then splits the agent's response and the computed answer into atomic facts and scores precision/recall regardless of output format
Results: agreement with expert labels rises from MCC 0.331 to 0.427, and token cost per case drops 16%. A judge with no ground truth scored MCC -0.379, worse than chance. The pipeline ships as a harness skill; one stated limitation is that the same model, Claude Sonnet 4.6, served as both judge and target.
More from coding & agent
- 68-Agent Build Cut $2K in API Costs by Keeping Long Context on One Orchestrator Only — TheMoonMidas · 2026-09-17
- Dev proposes predictive dynamic context caching for Claude Code — Sauers_ · 2026-09-17
- Devin Fusion called best intelligence-per-cost coding agent; Meta Muse as wildcard pick — brandon_galang · 2026-09-17
- Dev builds an RL environment around God of War for agents to train in — silver__tsuki · 2026-09-17
- Building explainable GraphRAG: why knowledge graphs beat vector search for auditable answers — camerongreen95 · 2026-09-17
- AI Agent Astra Helps Port a Beloved 386-Era Game From One's Childhood — banteg · 2026-09-17