Testing AI agents with simulations: replay real traces, 50 curated sims beat 1000 random ones
HamelHusain · x · 2026-10-13
Raindrop cofounder Ben Hylak joined Hamel Husain's AI Evals course to explain agent testing via simulations: replay real production traces against a change to see what breaks. Key points — cached tool results tip agents off that they're in a sim, so rebuild the world around the agent; 5–50 well-chosen simulations beat thousands of random ones; evals are turning into simple functions over traces; includes how OpenAI tests Codex this way.
Related event: Hamel Husain interviews benhylak on simulation-based agent evals(4 posts)→
More from coding & agent
- Developer says the Sketch MCP tooling is 'actually useful' in practice — nathanborror · 2026-10-13
- Pulse: open-source mission control for Claude Code with live agent map and local voice alerts — ThirdEyeDry · 2026-10-13
- Every trades personal agents for one shared company agent in Slack — every · 2026-10-13
- Dev shares prepended prompt that stops coding agents from bloating code — gabriberton · 2026-10-13
- One prompt orchestrates Haiku 5.5 builders and Astra reviewers into an autonomous PR loop on t3code nightly — talkaboutdesign · 2026-10-13
- Running tens of millions of experiments on useless programs with LLMs — bushibuilds · 2026-10-13