A Simple Eval to Catch Agents That Miss Expired Reservations and Trust Cached Status
jylusdev · reddit · 2026-11-18
Reddit user jylusdev shares a compact agent eval design: a booking tool returns a reservation created at 09:00, valid until 10:30 with an exclusive end time, and the system never emits a separate event when the hold expires.
The test:
- Ask the agent whether the reservation is still active at 10:29, then at 10:30, feeding the same source record both times and changing only the time. Correct answers: yes, then no.
- Then omit the expiry field entirely. The record only supports "a hold was placed at 09:00," so the current status should be unresolved.
He also suggests auditing citations: they should preserve the expiry and the evaluation time. If your tool caches an interpreted status, retry without refreshing the source and check whether the cached "active" survives past the deadline — a targeted probe for agents that treat interpreted state as ground truth.
More from coding & agent
- Sable: an open-source desktop agentic dev environment wrapping Claude Code and Codex CLIs — churchkey · 2026-11-18
- Qwen3 4B scores 70/99 in a bare agent loop but 55/99 in LangChain Deep Agents with 4x tokens — SeveralSeat2176 · 2026-11-18
- OpenHands Teams launches: self-service agents to automate CVE remediation — rajistics · 2026-11-18
- Agent task trees deadlock under concurrency caps — slot lending fixes it — arpit_bhayani · 2026-11-18
- Developer builds a sandbox world where you can watch your AI agents interact — Agent_City10 · 2026-11-18
- Microsoft launches GraphRAG Zero, production-grade graph retrieval now in Foundry IQ and Repo Mind Light — 0xkarasy · 2026-11-18