ICAE-Bench tests coding agents as interactive project builders from fuzzy briefs
Zhongyuan Peng · hf · 2026-07-24
ICAE-Bench evaluates coding agents as interactive project builders
The paper argues that current coding-agent benchmarks lag behind the reality of vibe coding, where agents are expected to turn fuzzy product intent into working software rather than simply complete fully specified tasks.
It introduces ICAE-Bench, which starts from ambiguous requirements grounded in real open-source repositories with executable behavior. To make the setup realistic and reproducible, it adds:
- an automated User Agent that can reveal hidden constraints without inventing new requirements or leaking implementation details
- black-box tests plus multi-dimensional diagnostics
- metrics for functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality
The goal is to evaluate agents in a dynamic, project-building setting instead of static task completion.
More from coding & agent
- Webcmd says browser agents waste tokens on navigation, not the actual task — heyshrutimishra · 2026-07-24
- AREX introduces a recursively self-improving deep research agent with inner and outer loops — _reachsumit · 2026-07-24
- A local open-source agent skill flags the rest of your GitHub diff — Saboo_Shubham_ · 2026-07-24
- Stop exposing thousands of tools to agents; use a code sandbox instead — edgestone22 · 2026-07-24
- “Letting Claude read my codebase is basically open-sourcing it,” says developer — _Stocko_ · 2026-07-24
- Roboto Agents adds root-cause analysis that links robot logs to source code — Scobleizer · 2026-07-24