Agent verified 'success' after double-voting: a lesson in harness vs model bugs
TejasKumar_ · x · 2026-10-26
Paul Iusztin shares Tejas Kumar's classic agent failure case: an agent built on GPT-3.5 Turbo was tasked with upvoting exactly ONE unvoted top Hacker News story. It succeeded — but upvoted TWO stories, and the verifier still reported success.
The failure chain:
- The agent clicked upvote while logged out, redirecting to a login page
- After the harness logged in, the original vote had already gone through but wasn't recognized
- The harness told the agent to go finish the task — it did, voting again
- The verifier checked whether an upvote click happened, not whether exactly ONE story was upvoted
The accompanying table summarizes three things a harness can trust and their blind spots: the model's last message (misses: success reported from a login page), the tool trace (misses: side effects no tool reported, like the vote the login completed), and world state before/after (misses: changes the system of record doesn't show). Kumar also found his login harness was buggy on revisit, and wrote up 4 key decisions for solidifying harnesses and telling harness bugs from model bugs.
More from coding & agent
- Dev argues there's still alpha in working with your coding agent interactively — alex_frantic · 2026-10-27
- Researchers launch AMA on patterns for safer AI agents used by citizen developers — Loose_Rip359 · 2026-10-27
- Gradio Adds In-Chat Human-in-the-Loop So ML Intern Pauses Before Training — Gradio · 2026-10-27
- Building Halo Infinite Forge maps with local LLMs, DCjson and Blender MCP — swagonflyyyy · 2026-10-27
- Ex-Unity dev Lucas Meijer says OpenClaw looks cleaned up and worth another run — lucasmeijer · 2026-10-27
- Agent skills started contradicting each other after months — how to keep them clean? — Successful-Tax1306 · 2026-10-27