Microsoft dev blog: building AX evals that actually work, when scalar scores mislead
lee_stott · x · 2026-09-07
Microsoft Principal Developer Advocate Waldek Mastykarz wraps up his Agent Experience (AX) series with a guide to building agent evals that actually work. Most evals, he argues, produce confident, consistent, and meaningless results: contaminated data, scenarios that don't reflect real usage, criteria checking the wrong thing, and scores that rise while developer experience stays flat.
Key points:
- Evals are a means to an end: find where models have gaps using your tech, then fill those gaps.
- Generated instructions (have a strong model write instructions for a cheaper one) work surprisingly often but small models have limits — the only way to know is evals.
- Prior articles covered what to measure, why benchmarks don't transfer, and hidden variables.
More from coding & agent
- OpenAI computer-use agents near superhuman speed, say the team building them — a16z · 2026-10-26
- Qdrant ships one-CLI Chroma migration tool that copies embeddings without re-embedding — qdrant_engine · 2026-10-26
- Jailbroken 10-year-old Kindle becomes an AI token usage dashboard via Codex — OpenAIDevs · 2026-10-26
- Farmer with no engineering background builds Codex-powered greenhouse ventilation — OpenAIDevs · 2026-10-26
- $13 Bluetooth speaker becomes a sarcastic AI rock via Codex and GPT Live — OpenAIDevs · 2026-10-26
- Codex-built LED handheld console plays games by tilting, mostly built in one day — OpenAIDevs · 2026-10-26