The agent isn't lying when it says 'done' — it answered a different question
AIGODSEND · reddit · 2026-10-30
The author describes a critical agent failure mode: an agent verifying a security fix found the patch, tests, advisory, and CVE text — all matched — and reported the issue resolved. But the patch was real while the bug was different; the original trigger still fired.
Key points:
- The agent isn't lying: it's confident, well-sourced, and wrong about which question it answered.
- Most agent evals ask "was the answer correct?"; the useful question is "which claim did the evidence actually support?"
The author is building a verification layer that forces claims and evidence to share a referent — same ticket, same PoC, same artifact, same action — otherwise the review is about a different bug wearing the same words. They're also soliciting agent-review cases in the comments.
Related event: Agent Claims Security Fix Done but Verified the Wrong Bug(2 posts)→
More from coding & agent
- The agent harness isn't for the agent — it's for us — seanmcdonaldxyz · 2026-10-30
- YC F26's Composal sends agent swarms to test your product like real users — ycombinator · 2026-10-30
- KreaMossMix ComfyUI workflow chains MOSS-Image and Krea 2 for four image variants per prompt — Plane-Implement1044 · 2026-10-30
- Dev builds fail-closed MCP tool that refuses to PASS unverified US businesses — cstill_dev · 2026-10-30
- Claude Code v2.1.314 adds write-blocking outside workdirs, idle compaction, gateway fixes — ashwin-ant · 2026-10-30
- ComfyArt: open-source Lovart-style AI design canvas running fully on your own GPU — Far-Driver-2904 · 2026-10-30