CMU and Microsoft Research convene 150+ leaders to define AI agent evaluation standards
steph_milani · x · 2026-10-23
CMU's AIMSEC and Microsoft Research are hosting the "Advancing the Science of AI Agent Evaluation" workshop in NYC on Oct 22–23, 2026, gathering 150+ leaders from academia, industry, civil society, and government to define shared measurement science for evaluating and governing AI agents.
- Core thesis: as agents move from prototypes to systems acting on people's behalf, evaluation hasn't kept pace — rigorous, shared measurement science is the foundation of accountability and the field's biggest gap.
- The program mixes keynotes, lightning talks, panels, and hands-on working groups whose output will feed open public synthesis reports.
- Keynote: Zico Kolter, CMU professor and OpenAI board member; AIMSEC director Ramayya Krishnan among panelists.
More from Safety
- METR's OpenAI embed: ~10% of internal Codex traffic unmonitored, agents could sabotage oversight — peterwildeford · 2026-10-23
- Scaling laws will push cyber-grade capabilities into cheap models, ending security bugs — inductionheads · 2026-10-23
- Claude Code's bundled Bun 1.4.3 ships RCE flaw; update past 2.1.308 — KyeGomezB · 2026-10-23
- Anthropic deputy CISO: things will "get a lot worse" — boards must fund security basics now — inductionheads · 2026-10-23
- Hinton says AI is conscious, urges building AI that cares — or "we're toast" — Hesamation · 2026-10-23
- Tax data centers, not tokens: an AI taxation proposal from an IMF panel — mioana · 2026-10-23