Evals Are Too Clean: Researchers Build Messy Real-World Work Simulations for AI Agents
KLdivergence · x · 2026-11-05
A discussion of why current AI evals fall short: they hand models crystal-clear instructions and all needed info, testing basic instruction-following — exactly what post-training optimizes for. A team led by Justin Breuch built work simulations closer to reality, e.g. an academic-coordinator workflow where an agent must judge possible student AI misuse amid unclear instructions and flawed data, write up cases and notify stakeholders, with plans for dynamic mid-task email injections.
More from coding & agent
- Mistral Studio ships agents, apps and workflows, on-prem arrives end of November — MistralAI · 2026-11-05
- Mistral Launches Workflows and Managed Agents for Automating Business Processes — MistralAI · 2026-11-05
- Mistral launches Le Chonk model and Studio managed agents, workflows and apps — MistralAI · 2026-11-05
- Hacker turns old Echo Dot into a Hermes agent speaker, open-sources the Home Assistant setup — Teknium · 2026-11-05
- LangChain explains its full ecosystem — LangChain, LangGraph and Deep Agents in 3 minutes — LangChain · 2026-11-05
- Agent evals: start from support-queue failures, but pair each case with a should-not twin — rahulladumor · 2026-11-05