Yoav Goldberg proposes a new agent eval: locating real GitHub artifacts via API
yoavgo · x · 2026-09-09
AI researcher Yoav Goldberg (@yoavgo) describes a non-trivial yet very natural search task for agent evaluation: find the artifact people are talking about in an online environment — concretely, given online conversation about a coding project, use the GitHub API to locate the exact Issue, Commit, or PR being discussed. It tests an agent's ability to disambiguate and search with real-world tools rather than answer static questions.
More from coding & agent
- Eraser Open-Sources AI-Native, Coordinate-Aware Diagram Format — yawnxyz · 2026-09-09
- Code formatting was solved in the 80s: Ada's DIANA IR stored no text sources at all — maxleiter · 2026-09-09
- QA Engineer Uses Codex to Learn DevOps, Then Wonders What's Left to Learn — Recent-Tangerine2745 · 2026-09-09
- WebMCP Gains Steam: MCP Tools Embedded in Web Pages for Browser Agents, No Server Needed — every · 2026-09-09
- Reducto doubles Extract speed, hits 99% citation coverage, flat 2¢/page pricing — ycombinator · 2026-09-09
- Vercel Switches to Flat-Rate CDN Pricing to End Surprise Bandwidth Bills — cramforce · 2026-09-09