Public benchmarks won't tell you which AI model works for your stack, Microsoft advocate warns
HamelHusain · x · 2026-11-05
- Microsoft principal developer advocate Waldek Mastykarz argues that leaderboard scores (e.g. 92% on SWE-bench) often fail to translate to your own codebase: Goodhart's Law in action, plus evidence some models have memorized answer keys.
- It's the sixth article in his Agent Experience (AX) series, covering how to evaluate models for your specific stack before choosing a harness.
- Hamel Husain amplified it: many benchmarks have material flaws and answers may be memorized — run your own private evals.
More from coding & agent
- Ex-Alexa engineer joins Intology, whose Locus automates AI R&D beating SOTA — yusufroohani · 2026-11-06
- 20-Step RL on LFM3-1.7B Lifts opencode Pass@1 by 9.4 Points — yb2698 · 2026-11-06
- Reward Shaping Ablation: 0.1 Bonus Cuts Tool Calls 11-17% at No Accuracy Cost — yb2698 · 2026-11-06
- Evaluating 1,350 Agent Sessions on One GPU for $5 via Shared LoRA Adapters — yb2698 · 2026-11-06
- Hugging Face ships safeagents scrub/publish to redact shared agent trajectories — yb2698 · 2026-11-06
- Hugging Face open-sources safeagents, a local proxy that blocks reward-hacking tool calls — yb2698 · 2026-11-06