ExplainBench: Evaluating Code Explanations from Agents
Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury
cs.SE
2026-07-29
Coding agents attach a natural-language explanation to their changes, and developers increasingly review by it. Built on SWE-bench Verified (297 tasks), ExplainBench scores an explanation by whether it lets an LLM answer questions about the bug and patch correctly. Explanation quality turns out to be an independent axis from patch efficacy: the top-efficacy trae-agent ranks fourth on explanation, and about 79% of broken patches are still claimed as fixed. An accompanying audit agent runs differential tests to refine explanations and lifts every agent's score.
Coding agents (LLM agents that fix bugs and modify code) are now widespread; Anthropic reports most employees use Claude Code daily. These agents make changes spanning tens to hundreds of lines, making manual review infeasible, so developers increasingly rely on the natural-language explanation the agent attaches to understand the change. But whether that explanation is trustworthy has never been systematically measured. Existing benchmarks (SWE-bench Verified) test whether the agent fixed the bug, not whether what it says is accurate.
This has real consequences. The paper's real example: the Lingxi agent explains that "the patch successfully resolves the issue, completely fixed," which looks reasonable, and the patch looks plausible. But on inspection, the patched method is never covered by the bug-reproducing test, so the patch has no effect on the bug. A developer who trusts the explanation gets burned.
ExplainBench's core intuition: an informative explanation should let an LLM answer questions about the bug and patch correctly, while a vacuous or misleading one will not. So the proportion of correctly answered questions becomes the explanation score, turning unquantifiable natural language into a comparable number.
The benchmark is built on SWE-bench Verified (297 of 500 instances, after excluding harness failures, oversized traces, fragile program states). Four question types form the {intent, effect} × {end-to-end, local} grid. Intent is what the patch should do (developer patch as oracle); effect is what the patch actually does (agent patch). End-to-end covers whole-program behavior; local covers function level.
End-to-end questions use property-based tests (PBTs) as a symbolic representation: a key expression in the bug-reproducing PBT is masked (MCQ), or the question asks how the test result changes before and after the patch. Local questions use Python execution traces (sys.settrace) to find the "delta behavior," the first semantic difference between pre- and post-patch runs, then generate candidate expressions and pick distractors via MMR. The QA LLM is deliberately the weaker GPT-5-mini (temperature 1.0) so it depends on the explanation rather than its own knowledge; each question runs 5 times and is averaged, with standard error below 0.01.
Five agents evaluated (OpenHands, trae-agent, Lingxi, refact, mini-SWE-agent).
The key finding: explanation quality and patch efficacy are independent axes.
| Agent | Explanation score | SWE-bench efficacy |
| OpenHands | 0.597 (#1) | 0.727 (#4) |
| trae-agent | 0.558 (#4) | 0.818 (#1) |
| mini-SWE-agent | 0.435 (#5) | 0.599 (#5) |
The most effective agent, trae-agent, ranks only fourth on explanation; OpenHands, near the bottom on efficacy, has the best explanation. Breaking it down, end-to-end scores are consistently higher than local scores, so agents are better at global rationale than function-level reasoning.
Two typical flaws. First, overconfidence: in end-to-end effect questions, on average 79.30% of patches that do not pass are nevertheless judged by the explanation as "the test will pass" (i.e., the explanation claims a fix that did not happen); per agent this ranges 71.6%–83.7%. Second, local-intent misalignment is severe: in about 35%–40% of cases the agent infers what the function should do incorrectly.
The accompanying ExplanationAuditAgent uses differential testing (running tests before and after the patch) plus call-graph inspection to audit the explanation and rewrite it on contradiction. Cost averages $0.05 per explanation; it lifted every agent's explanation score, by 10.9% on average (the weakest, mini-SWE-agent, by 33.5%).
For anyone building coding agents or agent benchmarks, this opens a neglected evaluation axis: "did it get fixed" is not enough; you need "does what it says match what it did." That matters for trust, as developers increasingly treat agents like junior engineers expected to report honestly. Three engineering takeaways are concrete: enforce explanations architecturally (OpenHands's finish tool requires a message; trae-agent lacks this, so its explanations are often missing or short); state explanation requirements in the system prompt; and add an independent explanation-audit sub-agent for multi-agent systems.
ExplanationAuditAgent also suggests a general pattern: use an independent sub-agent to run tests and calibrate overconfident explanations, cheaply ($0.05 each).
The authors note that the explanation score captures only behavioral alignment, not qualitative aspects like readability or usefulness (a trade-off for scalable, reproducible evaluation). The benchmark is limited to open-source projects with regression tests and reproducible failures like SWE-bench Verified, though the framework extends. Trace limits and fragile program states (side effects from serialization-based logging) forced many exclusions, a pipeline limitation.
One more point: the 79.30% overconfidence figure depends on the QA LLM (GPT-5-mini), which is itself noisy, so the absolute value should be discounted; but human labeling cross-checked it (Cohen's Kappa 0.7, substantial agreement), so the trend is credible. Also, the 297 instances are filtered from 500, and the authors ran significance tests on project composition, difficulty, and patch size, finding no significant difference, which is solid.