Microsoft's AX Playbook: Coding Agents Don't Hallucinate Your SDK, They Follow Stale Training Data
大模型之路 · wechat · 2026-10-12
Microsoft DevRel released the Agent Experience (AX) Practitioner Playbook on October 9, addressing a common pain point: coding agents produce code that compiles and looks reasonable, yet uses the wrong SDK version, deprecated auth patterns, or unrecommended configs. Its core thesis: waiting for better models is not a strategy — what you can fix is the surface agents consume: docs, MCP tools, skills, plugins, instructions, CLIs, and APIs.
Key points:
- Evaluations can convincingly deceive you: Microsoft saw a code sample that never compiled score full marks, and a check labeled "uses PlatformX" pass regardless of agent behavior. The cheapest fix is a run gate (prove code actually runs) before semantic scoring.
- Criteria belong to humans, not models: judging criteria must be consistent, calibrated before launch, and versioned when the product changes; a judge that flips on re-runs tells you nothing.
- Three failure tiers, different fixes: extension never loaded vs. loaded but never called vs. called but used wrong — rewriting a skill the agent never opened is wasted work. Check discovery before fixing content.
- From fall 2025, Microsoft ran evaluations with real developer prompts across Azure, Cosmos DB, SPFx, and M365 Copilot extensions, distilling nine recurring failure patterns; agents aren't hallucinating but strictly following training data.
- The method shipped real results: 46 improvements to the Azure Cosmos DB AgentKit; an SPFx upgrade scenario walked end-to-end. It's also packaged as an AX Practitioners skill (95% average alignment with the handbook across 330 questions), and is evaluation-system agnostic.
Bottom line: making agents use your tech correctly is an ops discipline, like unit tests — costly to set up, but it immediately tells you when model upgrades or doc changes break agent behavior.
More from coding & agent
- Daniel Jeffries: only ~2% can actually use AI agents, mass adoption is 10-20 years away — Dan_Jeffries1 · 2026-10-13
- Fully local AI short film mimics Wes Anderson style with a detailed open pipeline — Mist3r-Robot · 2026-10-13
- Controlling his Mac's terminal from a phone via t3 code beta while waiting at speech therapy — NERDDISCO · 2026-10-13
- LLM cache verifier drift: reject-only calibration tripled error rate; 5% audits fix it — Reasonable_Royal_621 · 2026-10-13
- A style guide for making Claude-like models write normally, and its linguistic traps — gleech · 2026-10-13
- Open-source desktop agent dodaios runs local Qwen models via ollama — and shows where 16GB models fail — InternationalGap3698 · 2026-10-13