ICAE-Bench tests coding agents as interactive project builders from fuzzy briefs
Zhongyuan Peng · hf · 2026-07-24
ICAE-Bench evaluates coding agents as interactive project builders
The paper argues that current coding-agent benchmarks lag behind the reality of vibe coding, where agents are expected to turn fuzzy product intent into working software rather than simply complete fully specified tasks.
It introduces ICAE-Bench, which starts from ambiguous requirements grounded in real open-source repositories with executable behavior. To make the setup realistic and reproducible, it adds:
- an automated User Agent that can reveal hidden constraints without inventing new requirements or leaking implementation details
- black-box tests plus multi-dimensional diagnostics
- metrics for functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality
The goal is to evaluate agents in a dynamic, project-building setting instead of static task completion.
More from coding & agent
- LangChain and Tavily to co-host NYC event on production agents using live web data, Nov 12 — LangChain · 2026-10-26
- Dev hacks reMarkable e-ink tablet to let Opus draw and animate — Stefania_druga · 2026-10-26
- Godot games are trivially decompilable — compile C# to native DLL for real protection — mflux · 2026-10-26
- Techhalla shares full 2D-to-3D workflow with Opus 5.5 and Kling 4.0 prompts — techhalla · 2026-10-26
- Jason Furman: Building Econvitals pre-AI would need ~50 people, 3 years, $20M — HarrySurden · 2026-10-26
- Dev dictates to Claude Code for 4+ hours while riding indoor bike in aero position — jacob_posel · 2026-10-26