Agent evals roadmap talk from AI Engineer NYC: video, slides, full guide
cwolferesearch · x · 2026-11-06
cwolferesearch has published his AI Engineer NYC talk on a practical roadmap for agent evals, focused on subjective/non-verifiable tasks but applicable to any eval. Topics: human evals, quality guidelines, LLM judges/rubrics, calibration of humans and judges, data sampling, statistical tools, A/B testing and production monitoring, and continual benchmark evolution. The talk is accompanied by slides and a full written guide with further reading.
More from coding & agent
- FastAPI creator tiangolo says he now uses voice-driven Codex way more after jxnlco's talk — jxnlco · 2026-11-06
- AWS WAF + Stripe lets sites charge AI agents via HTTP 402, with fiat payouts — jeff_weinstein · 2026-11-06
- The computable enterprise: capturing business logic that lives in employees' heads — vishalmisra · 2026-11-06
- Claude skills are useless — until you hook them to your actual business — aryanXmahajan · 2026-11-06
- Do you use MCP firewalls with Claude, Codex or Cursor in production? — vdobroskok · 2026-11-06
- Firecrawl launches Alexandria agent data layer with Benzinga, announces $75M Series B — devdigest · 2026-11-06