Swapping only the eval harness moves results 30%: designing agent evals like a scientist
AI Engineer · youtube · 2026-10-23
On the AI Engineer podcast, Stephanie Jarmak (former astronomer, now building evals at Omni Analytics after a large coding benchmark at Sourcegraph) explains why most agent evaluations are easy to misread.
Key points: swapping only the harness shifts results by 30%; public benchmarks are for shortlisting, not decisions; design evals like experiments — realistic tasks, repeated runs, full traces, and metrics matching what you care about. She also covers quality gates that actually stop work, the human-review bottleneck, untrustworthy agent records, and why production is the final test.
More from coding & agent
- Devs on Reddit swap tactics for stopping prompt regressions in AI agents — SleepFlimsy6963 · 2026-10-23
- Claude Code + ComfyUI MCP workflow for client work: paid models or open weights? — newaccount47 · 2026-10-23
- Social scientist: CS treats agentic development as a magic black box — RexDouglass · 2026-10-23
- Someone vibe-coded a Tesla Roadster game — before GTA VI — Jasonio · 2026-10-23
- Codex 0.164.0 'account/read failed' TUI error — fix is restarting the app-server daemon — bytebot · 2026-10-23
- Same native UI, two stacks: Swift build is ~80MB Mac-only, Zig build is ~10MB cross-platform — jedisct1 · 2026-10-23