Agent Evals as Replay: Save Past Work, Test New Models

Hammer shares a practical methodology for building agent evals: treat evaluation as replaying past tasks with new models or prompts, and invest 90% of effort in measurement itself.

2026-10-11 ~ 2026-10-12 · 2 related posts

• Evals are replays: store tasks in R2, sandbox in Docker, and 90% of the work is measuring — danshipper · 2026-10-11
• How to Build Good Evals: Save Past Work and Replay It on New Models — danshipper · 2026-10-12