How to Build Good Evals: Save Past Work and Replay It on New Models
danshipper · x · 2026-10-12
A practical take on building good evals without getting lost or in trouble:
Treat evals as saving past work and replaying it with new models or prompts/skills to see if it improved. That buys you three things:
- See where it fails so you can tell your boss/users.
- Optimize until it improves.
- Make it work on a cheaper or open-source model.
Measuring "if it improved" is 90% of the work; infra is only 10%. For infra, just tell your agent to store tasks in Cloudflare R2 (a cheap/free cloud file store) in Harbor task format — saving the task instructions and how the folder looked before the task started (whether the agent has all the files you had access to).
Related event: Agent Evals as Replay: Save Past Work, Test New Models(2 posts)→
More from coding & agent
- Tov: A TypeScript-like language for coding agents that beats Rust in several benchmarks — JosephJacks_ · 2026-10-12
- The model swallows the harness: agent frameworks' logic migrating into models — khademinori · 2026-10-12
- AI collaborator Anima draws termite gut cell, calls it her closest self-portrait — RileyRalmuto · 2026-10-12
- Editing video purely with chat and cloud agents: a full workflow breakdown — mattyp · 2026-10-12
- What if every query had its own harness? — amankhan · 2026-10-12
- MCP Server for Images: Compress, Convert, and Optimize a Whole Site's Images Safely — Able-Place · 2026-10-12