Arize AI workshop: how to write your first AI eval, from vibes to LLM judges
AI Engineer · youtube · 2026-10-26
A hands-on AI Engineer workshop by SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals:
- Evals vs unit tests: why vibe checks don't scale and why agents make evals harder
- Eval types: LLM-as-a-judge, decision models, code evals, annotations, agent-as-a-judge
- Practical path: reading traces, defining what "good" means, writing concrete pass/fail criteria, building datasets from traces
- Building an LLM judge: constructing and iterating a judge in Arize's prompt playground, plus next steps like alignment and online evals
A complete intro to eval engineering for developers building agent products.
More from coding & agent
- App Store Connect CLI can list screenshot sets and screenshots in one call — rudrank · 2026-10-26
- App Store Connect CLI 5.17.0 ships one-call screenshot reordering, 8k stars on GitHub — rudrank · 2026-10-26
- Dev shows GPT-6 Astra Ultra coding for 41+ hours straight on a Dreamcast game — AIandDesign · 2026-10-26
- Bitpanda crypto exchange gets an MCP server for trades, wallets and transactions — modelcontextprotocol · 2026-10-26
- GitLab MCP connector wraps REST API v4 with bring-your-own API key — modelcontextprotocol · 2026-10-26
- PyTorch nested graph breaks deliver 4.53× steady-state speedup on B200 — cfregly · 2026-10-26