Snorkel AI says agent benchmarks should be rebuilt from production traces

AI Engineer · youtube · 2026-07-25

Rustem Feyzkhanov of Snorkel AI argues that agent evaluation should move from public benchmarks to private, production-derived simulations.

Start from real production traces, then reconstruct the database state, files, and tools the agent touched so the task can be replayed under identical conditions.
Public benchmarks like WebArena give only a single success rate, while teams actually need cost per solved task, latency, and policy compliance on their own workflows.
The benchmark should reuse the same skills, tools, and evaluators the production agent sees, so models can be compared apples to apples.
For long-horizon tasks, a verifier checks the final state and an LLM judge checks policy adherence, with early stopping when the run clearly goes off track.
The hard problems are reward hacking, missing fixtures, and tasks that turn out to be unsolvable.
The talk’s core claim: every company will eventually need this as part of an agent CI/ops loop that keeps pace with changing agents.

More from coding & agent