Cameron Wolfe on reliably evaluating AI agents for highly subjective tasks
cwolferesearch · x · 2026-10-28
Cameron R. Wolfe, Ph.D. published a long-form overview, Evaluating Agents on Highly Subjective Tasks, companion to his AI Engineer conference talk, tackling agent evaluation—one of the hardest problems in current AI research.
Core challenges
- Benchmarks suffer from benchmaxxing and saturation; new ones keep appearing without measuring real capability
- Agent tasks are broad, long-horizon and complex, making evaluation expensive
- Agent non-determinism makes it hard to draw reliable conclusions
- Subjective/open-ended tasks lack deterministic verification, forcing reliance on noisy proxies like human review or LLM judges
Approach
The piece lays out a repeatable framework for building reliable evaluations even for subjective tasks, stressing diverse task curation (avoiding inflated scores by only testing agent strengths), with an Epoch AI survey of recent benchmarks and a reference to his evaluation roadmap blog post.
Related event: Cameron Wolfe Outlines Principles for Reliable LLM and Agent Evaluation(2 posts)→
More from coding & agent
- Chat Brawlers: three AI models debate and vote — and one tricky stopping rule — lluisantoni · 2026-10-28
- Dumb chunk-and-rerank baseline beats top agentic memory systems in 6 of 7 benchmarks — JeremyCMorgan · 2026-10-28
- Microsoft's MANDATE: 23 failure categories from 163k Reddit posts on AI agents — dair_ai · 2026-10-28
- LangSmith now tracks cache hit rates for your agents — LangChain · 2026-10-28
- MOSS-Image ecosystem roundup: ComfyUI 0.43, QwenVL layout node, new style LoRAs — optimisticalish · 2026-10-28
- Ex-PC repairman builds JinnexGuard with Claude Code to scan MCP servers before install — Complex-Pin-4616 · 2026-10-28