Why Evals Mislead: Clear Instructions Only Test What Post-Training Already Optimizes
KLdivergence · x · 2026-11-05
Follow-up in the same discussion: current evals give crystal-clear instructions and complete information, so they measure basic instruction-following — exactly what existing post-training architectures optimize for — rather than performance in messy real-world work.
More from coding & agent
- Mistral Studio ships agents, apps and workflows, on-prem arrives end of November — MistralAI · 2026-11-05
- Mistral Launches Workflows and Managed Agents for Automating Business Processes — MistralAI · 2026-11-05
- Mistral launches Le Chonk model and Studio managed agents, workflows and apps — MistralAI · 2026-11-05
- Hacker turns old Echo Dot into a Hermes agent speaker, open-sources the Home Assistant setup — Teknium · 2026-11-05
- LangChain explains its full ecosystem — LangChain, LangGraph and Deep Agents in 3 minutes — LangChain · 2026-11-05
- Agent evals: start from support-queue failures, but pair each case with a should-not twin — rahulladumor · 2026-11-05