Experts: AI Struggles with Bottom-Up Evals, Focus on Taste
petergyang · x · 2026-08-23
This post summarizes key insights from a discussion with AI eval experts Shreya and Hamel. Key takeaways: fundamentals of evals remain, but agents aid analysis; AI is bad at bottom-up evals (deriving criteria from samples); agents help distill feedback into rubrics rather than inventing it; and success depends on taste in eval design, not just using Claude to find errors.
Related event: Experts discuss building AI evaluations with Claude Code(2 posts)→
More from coding & agent
- Codex user keeps hitting niche bugs — every one already has an open GitHub issue — xeophon · 2026-08-23
- vLLM keeps hitting 400 context errors with DeepSeek harness; llama.cpp runs 24h+ fine — cviperr33 · 2026-08-23
- Dad ditches OpenRouter token burn for local Qwen 3.8 + Pi agent, controlled via Telegram — dcnotpc · 2026-08-23
- FatherLode dev update: Claude-built game adds weather, museum with 30+ treasures, Suno 5.5 music — Extension-Parsnip789 · 2026-08-23
- Open Source MCP Connector: Links Claude with Office 2019 — FarRespond73 · 2026-08-23
- Can Local Qwen Models Handle Real-World System Programming like GTK4/Qt? — MongoWithBongoss · 2026-08-23