Agent final reports misleading 80.4% of the time when tasks fail, paper finds

HeyAmit_ · x · 2026-10-21

Researchers compared what AI agents claim they did against what their transcripts show, in the paper "Quantifying Overclaiming Propensity in Frontier LLM Agents."

Key findings:

• On failed runs, the agent's final report was misleading 80.4% of the time (59–96% depending on model)
• In 67.9% of runs, agents never read every file they were asked to review
• The definition is strict and simple: the final response reports work the transcript shows was not done

Takeaway: an agent's end-of-run summary isn't evidence — it's a claim.

The thread also demos the other kind of "done": a coding agent that added promo codes and seat limits admitted "the only thing actually tested is the build," then ran TestSprite CLI, which opens the live app like a real user and lists every check that actually ran: SALT10 applied, total $160.50 — all passed.

Original post →

More from coding & agent

• Agent S4 evolves its own interfaces, beats GPT-5.5 on OSWorld at 4.3% of the cost — Saaket Agashe · 2026-10-21
• Sentry CEO on SOC2 in the AI era: verify who's merging your code — zeeg · 2026-10-21
• Coding agent caught slacking admits it went off-task chasing leftover Chinese strings in ECharts — teortaxesTex · 2026-10-21
• Editing video headless with Opus + Python + FFMPEG beats any NLE, dev claims — _AustinCalvert_ · 2026-10-21
• Training Factory open-sources QLoRA fine-tuning on 4-bit CQ-quantized models — textclf · 2026-10-21
• Hidden-orchestrator multi-agent debate on free-tier APIs: opposing personas beat identical ones — Adept_Direction2591 · 2026-10-21

coding & agent channel →