Tencent Hunyuan's WebCraftBench tests web apps like software, matching human preference 85.3%
TencentHunyuan · x · 2026-09-22
Tencent Hunyuan released WebCraftBench, an interactive benchmark that evaluates LLM-generated web applications from a software testing perspective.
Why: static benchmarks credit functionality in source code that's unreachable at runtime; interactive ones miss implemented features due to incomplete exploration and conflate app defects with agent execution failures.
How:
- Instruments each generated app and uses code coverage to guide an agent exploring it via simulated user interactions
- Abstracts interaction traces into a state-transition graph
- Scores along three dimensions: visual aesthetics, usability, and requirement alignment — with exploration decoupled from scoring
Scale & results: 369 real-world user requirements and 5,088 acceptance criteria; on 197 human-validated pairs it matches human preference 85.3% of the time, and the team benchmarked 17 frontier LLMs.
More from coding & agent
- Developer uses Codex computer use to navigate Austria's messy FinanzOnline tax portal — kevinkern · 2026-09-22
- Multi-agent group chats: drop a task at night, run 10+ agents unattended, review by morning — huangyun_122 · 2026-09-22
- 7 open-source repos for scraping millions of web pages, from Scrapling to ScrapeGraphAI — JafarNajafov · 2026-09-22
- If agents can shop for you, agent-native rivals will replace Zomato, Amazon and Blinkit — vaibhavbetter · 2026-09-22
- Refunds, fraud signals and high-stakes calls: where should agent autonomy stop? — ConvertMyStore · 2026-09-22
- Obsidian Starter Kit v4 turns your vault into an AI workspace with 375 skills and MCP server — dSebastien · 2026-09-22