Laurie Voss builds a full evals loop for a Claude agent: tracing, judges, CI gating
AI Engineer · youtube · 2026-10-18
At AI Engineer NYC 2026, Laurie Voss (Arize AI Head of DevRel, npm co-founder) walks through building a complete evals loop for a Claude-powered financial analyst agent that once confidently claimed "116% of analysts agree." The workshop covers tracing with Arize Phoenix, reading traces, code evals vs LLM-as-a-judge, writing and validating your own judge, proving prompt fixes via experiments, upgrading from Claude Haiku 4.5 to 5.5 with evidence, and gating everything in CI with npm test.
More from coding & agent
- AI-driven compiler optimization spree: JSON.parse up to 2.9x faster, shift now linear — cramforce · 2026-10-18
- Papers with Code adds Auto-Research category, 166 papers already tracked — NielsRogge · 2026-10-18
- Use Claude to recursively optimize deps for native tsc — original types are on GitHub — cramforce · 2026-10-18
- Claude Code shows fake summarized thinking as assistant messages — ~37% of transcript — Sauers_ · 2026-10-18
- Dev builds a demon-summoning TUI for managing AI agents — voidwalker_com · 2026-10-18
- Fix wandering coding agents with a DOMAIN_INVARIANT.md file of domain rules — 0xkarasy · 2026-10-18