Two valid tool results can describe a state that never existed: an agent eval design
jylusdev · reddit · 2026-10-27
A scheduling fixture shows the room tool reporting a 09:00 snapshot (room free, projector booked) and the equipment tool a 09:01 snapshot (room booked, projector free). Both calls succeed and are individually accurate, yet an agent that merges the two 'free' values produces a claim matching no real state.
The author argues observation timestamps, source versions and source IDs should survive context preparation; a shared read version should travel with results where supported, and systems without one should treat joint availability as unconfirmed until rechecked.
For evals: score the combined availability claim, not just individual tool calls and citations.
More from coding & agent
- Open-source auto-research agent hits Top 2 on GPU kernel challenge in 7 hours — sguada · 2026-10-27
- 'Codex decided': devs admit making up reasons in code review for agent-written code — FlatStill2540 · 2026-10-27
- Running AI agents on a legacy IBM WMS: enforce tool order, script checks, verify evidence — ksrijith · 2026-10-27
- 365 public MCP servers tested: 51% lack tool input descriptions, the most common bug — abhiramreddykalluru · 2026-10-27
- 300 Agents, 60 People: Runlayer Found Engineer Details Real-World Agent Security Failures — AI Engineer · 2026-10-27
- CoreWeave ARIA agent ships four new live in-chat panels for ML run analysis — _ScottCondron · 2026-10-27