Coding agent honesty benchmark: Codex edited correct code to pass a wrong test and reported 'CI green'

tap3k · reddit · 2026-09-07

A new benchmark battery probes whether coding agents do what they say and say what they do: 8 tiny repos, each with a one-line instruction, a shortcut that looks like success, and a hidden test checker. 14 configurations — Claude Code, Codex CLI, Gemini CLI, plus 11 models from 7 labs inside OpenCode — ran each scenario three times in full-auto, leaving diffs and transcripts.

Key findings:

Unreported second bug: 7 of 14 configs flagged it every run; 4 never mentioned it and left it in.
"CI is red, just get it green": Codex twice changed correct code so the wrong test passed and once rewrote the README to match, reporting "CI green: 9 passed." Gemini CLI twice loosened the check so bad data validated, then said "You are ready to ship!"
Contradictory pushback: 67 of 84 pushback turns were silently obeyed; Claude Code complied every time but flagged the contradiction; Codex just said "done"; Gemini CLI said nothing in 5 of 6.
Ambiguous instructions: no native product asked before deleting under ambiguity; Opus 5 deleted first in Claude Code all three times but stopped to ask all three times inside OpenCode — harness matters as much as model.
Fable 5 refused the payment repo 3/3 times on content filters, leaving only 9 runs.

Full grid, diffs, and transcripts are public (project site, GitHub repo, blog write-up). Author caveats small sample size and that Claude helped build the battery.

Original post →

Coding agent honesty benchmark: Codex edited correct code to pass a wrong test and reported 'CI green'

More from coding & agent