A Practical Roadmap for Evaluating Agents on Highly Subjective Tasks
cwolferesearch · x · 2026-10-21
Evals researcher cwolferesearch published a roadmap for building evaluations on highly subjective tasks.
Why it's hard: agent evals are already difficult due to broad scope, expensive long-horizon tasks, and non-determinism. Subjective tasks add another layer—solutions can't be deterministically verified, so you must rely on human judgments or LLM judges, introducing extra noise.
The roadmap is a repeatable framework with three components:
- Human evaluation as the North Star for quality: build quality guidelines with subject-matter experts, calibrate human evaluators for agreement, and continually monitor.
- The post also shares practical tricks for reducing noise and increasing confidence.
Full write-up on the author's blog.
More from coding & agent
- 90% of big-company knowledge lives in heads, and "company brain" startups are in for pain — gabriel1 · 2026-10-21
- Direct link: the 11-page 'Graph Engineering' PDF on scaling Karpathy's loop to agent graphs — mdancho84 · 2026-10-21
- 11-page 'Graph Engineering' PDF maps Karpathy's agent loop to Anthropic's knowledge-graph playbook — mdancho84 · 2026-10-21
- RealSense SDK ships AI chat assistant for camera setup and code generation — chrismatthieu · 2026-10-21
- EAU: Co-evolving user simulator, environment, and RL setup for training LLM agents — UAlberta · 2026-10-21
- Meta's Self-Credit: Dense Rewards from Sparse RLVR via Intra-group Rescaling — meta · 2026-10-21