AI Eval Experts: Claude Excels at Top-Down Evals, But Bottom-Up Criteria Are All You
HamelHusain · x · 2026-08-24
petergyang talks with AI eval experts Shreya Rajpal and Hamel Husain about two types of evals. Top-down: given only a task description in a vacuum, design evals from scratch—Claude is very good at helping here. Bottom-up: look at lots of sample outputs and externalize your gut feedback into eval criteria—Shreya stresses Claude is "very, very bad" at these; that part is all you.
The two have taught AI evals to 4,500+ engineers and PMs. On the show they audit petergyang's evals for his creator skills live and demo a free skill usable in Claude Code or Codex that turns feedback into reusable evals. Key point: eval fundamentals haven't changed—start by looking at real data; what's new is using agents to help you look thoughtfully. The agent's job is not to invent new feedback but to externalize your judgment.
Related event: AI Eval Experts on Building Evaluations with Claude Code(3 posts)→
More from coding & agent
- Study: Agents read instructions/notes 60.5% of the time, rarely touch API docs — dair_ai · 2026-08-24
- Claude Verifies 43 Lean Modules autonomously, Tackling Theoretical Physics — Tkaraletsos · 2026-08-24
- Compound Engineering Plugin Rewrite Cuts Context Usage by 70% — iamrobotbear · 2026-08-24
- Lighter's prover challenge breaks 100k TPS, 10x above launch baseline in weeks — econoar · 2026-08-24
- AI fakes memory: why it gets confidently wrong without forgetting — PrajwalTomar_ · 2026-08-24
- Observation suggests Codex continues running tasks long after weekly credits run out — gandamu_ml · 2026-08-24