ConformWeb: LLM-generated web apps pass only 11.7% of behavioral contract checks
algo_diver · x · 2026-10-26
A first-author poster at EMNLP 2026 Budapest presents ConformWeb, the first contract-grounded benchmark for behavioral conformance of LLM-generated web apps.
Key findings
- LLM-built web apps look functional, but only 11.7% of runs (252/2160) pass every public plus hidden behavioral contract
- Scenario-level pass rate is just 36.7%; step-level 49.4%
- The largest failure class is missing validation/state management; unaudited failures under the surface far exceed visible ones
- Adding a backend or more repair rounds barely helps: two repair rounds only lift full pass to 23.4%, and Tier D scenarios stay at 10%
- Capability correlates with score: Claude Opus 4.6 scores 70.8 full pass vs Gemini 3 Flash at 4.3
Method: behavioral contracts defined over actions, observations and relations, including hidden scenarios (private state, role permissions, persistence across reloads); 24 intents, 286 public + 468 hidden behaviors, 10 models, 7 app families.
Takeaway: static UI inspection is insufficient — UI-only repairs succeed just 23.4% of the time, and frontend capability gains don't buy backend correctness.
More from coding & agent
- OpenAI's agent built the same site 8 times — every deployment landed on Cloudflare — FinanceYF5 · 2026-10-26
- Coding agents now good enough to build your own ultra-specialized one-off tools — giffmana · 2026-10-26
- Vibe-coded research rule: don't attempt what you don't care to understand — generativist · 2026-10-26
- Fewer, well-designed agent harnesses beat sprawling agent fleets, argues vixsheikh — annetgriffin · 2026-10-26
- Agent memory explained in 12 minutes: working, procedural, semantic and episodic — _AustinCalvert_ · 2026-10-26
- Grok Bot v0.75.5 leak touts stored cards, email forwarding, 3x automation limits — mark_k · 2026-10-26