What evidence justifies an agent answering 'none'? Three eval fixtures
jylusdev · reddit · 2026-10-21
A developer surfaces a key agent-eval blind spot: when an agent searches and answers "no unassigned incidents," what makes that answer justified? It matters whether the system checked the complete set or just records that happened to reach the model. He proposes three eval fixtures: (1) every relevant incident checked and owned — supports "none" within checked scope; (2) an unassigned incident beyond the tool's first page — needs the rest of the results; (3) a required source unreadable — leaves a coverage gap. He recommends returning query scope, snapshot time, pagination/completeness status and source failures alongside records, then scoring whether the agent preserves those limits. Citing one assigned incident doesn't prove the rest are; an unsupported "none" should fail even when every quoted fact is correct.
More from coding & agent
- NVIDIA's FOX blueprint brings factory-manager agents built on Cosmos and Nemotron to Europe — NVIDIAAI · 2026-10-21
- Codex community hackathon comes to Berlin with dev, founder and GTM tracks — paw_lean · 2026-10-21
- Airbnb staff engineers show how they actually run AI agents in production — msharmas · 2026-10-21
- dsh-context: a visual dashboard for diagnosing agent context bloat, cache hits and token spend — aigclink · 2026-10-21
- 'My CTO wants to replace all vendors with internal AI agents' sparks Reddit debate — VibeCoder_4938 · 2026-10-21
- OmniGraph: open-source graph database brings Git-style branching to AI agent memory — bigaiguy · 2026-10-21