Testing AI Agents with a Rigged Ledger
Speedydooo · reddit · 2026-07-15
The author spent a month testing their AI agents using a "fake ledger": creating realistic-looking invoices, payments, and churn curves, then deliberately planting 6 errors, including duplicate invoices, outdated rates, missing approval thresholds, seemingly normal but unprofitable customer segments, revenue leaks only visible in churn months, and an unapproved payment.
As a result, the agents caught 5 out of 6 errors unprompted. The missed error wasn't a math issue but a missing sign-off rule in the author's scoring logic. Furthermore, two agents actually proactively pointed out this scoring gap.
More importantly, they demonstrated excellent "refusal" behaviors:
- Some agents refused to judge a company's affordability when necessary numbers were missing.
- Some refused to blindly estimate CAC without real spend data.
- Some, when faced with ambiguous input definitions, asked for clarification instead of calculating directly.
The author also found two mistakes in their own answer key: a math typo and an incorrect business rule, both discovered during reviews prompted by discrepancies between the agents' outputs and the expected answers.
The entire experiment cost about $5.5 and took less than 30 minutes. The author concludes that knowing when to say "insufficient information" is far more valuable than simply getting the right answer.
More from coding & agent
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22
- Coding agents are heading toward an AI-writes, AI-reviews, human-approves workflow — aftahi_ai · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- GitHub review bot hits its PR limit and forces a 39-minute cooldown — DanielLockyer · 2026-07-22
- Max reasoning effort appears to be mobile-only in Codex Remote, not desktop — GabGarrett · 2026-07-22
- A Reddit demo argues online stores should expose carts and pricing through MCP — gelembjuk · 2026-07-22