Agent eval design: testing partial updates vs full snapshots with a truck dock case
jylusdev · reddit · 2026-11-05
A developer shares a concrete recipe for designing agent evals around state-update semantics. Example: event A at 08:00 assigns truck T17 to dock 4 and sets status to waiting; event B at 08:05 sets status to ready with no dock field. If B is a partial update, the correct answer is "ready, dock 4"; if B is a full snapshot with omitted fields treated as unknown, the dock becomes unknown.
His suggestion: turn both semantics into separate eval cases, changing only the declared update type, and score dock and status independently — in the partial-update case dock needs evidence from A and status from B; in the snapshot case the agent should report the dock as unknown.
He also recommends keeping the update rule and the source of each resolved value in the context sent to the model, so a wrong dock answer can be traced to either misread evidence or a wrong upstream state.
More from coding & agent
- Claude Code cheat sheet: CLAUDE.md, Skills, MCP and the plan-implement-iterate loop — nikola_mr64990 · 2026-11-05
- Bobby Iliev's open-source 'Introduction to Docker' book joins ChapterPal with an AI tutor — burkov · 2026-11-05
- CloudOpsBench: open-source benchmark tests AI agents on 45 real cloud ops scenarios — DarkMG1 · 2026-11-05
- Mid-steps of an AI level-building pipeline: playtest first, then batch concept art with GPT Image — FinanceYF5 · 2026-11-05
- Building a Complete Original Game in 24 Hours with Opus, Flux, Tripo and Three.js — FinanceYF5 · 2026-11-05
- Team automates 100% of product design with Claude, hasn't touched Figma in months — manosaie · 2026-11-05