Agent evals: valid permission and approval can't catch stale facts before execution
jylusdev · reddit · 2026-10-11
The author shares a failure case worth adding to agent evals: a manager approves an order because a supplier is active, but before the agent executes, a new record puts that supplier on hold. Approval isn't revoked, permissions are intact, the document is genuine—yet the facts behind the decision have changed.
- Correctly retrieving the approval isn't enough; the system must connect order to supplier, resolve current state, and expose changed evidence
- A separate execution problem remains: action APIs must enforce preconditions; a fresh Context Pack can't make external actions atomic
- A useful eval asks three questions: what evidence supported the original approval, what changed before execution, and is current evidence sufficient to proceed
The post promotes the author's product Jylus and invites discussion on what truly invalidates earlier decisions in live agent stacks.
More from coding & agent
- Open-source Hermes for Tern: a native frontend rendering agent chats inside Tern — Teknium · 2026-10-11
- 112 bugs across 84 projects: LLMs pass proof-of-concept but fail developers' own tests — lulzxdxdxd · 2026-10-11
- RegistrumMCP gives AI agents live UK Companies House data with no API key or signup — modelcontextprotocol · 2026-10-11
- Hive Evaluator MCP server ships NEED/YIELD/CLEAN-MONEY gates with EIP-3009 attestations — modelcontextprotocol · 2026-10-11
- One prompt, 5 minutes: marclou has Opus 5.5 build him a macOS mouse remapping app — marclou · 2026-10-11
- How do you audit what your business AI agents did yesterday? — Accomplished_Team_19 · 2026-10-11