EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants
Yue Peng, Lanke Xia, Zihan Wang, Jiahao Ye, Ke Ning, Hongyi Wen
cs.AI, cs.SE
2026-08-30
EvoGenUI-Bench runs 150 five-turn tasks across eight models: Claude-Opus-4.7 reaches 74.9% turn pass but only 37.3% of episodes; tool-grounded adjacent pass retention is 52.4%.
LLMs can already emit a dashboard, a form, or a mini-app in one shot. Most existing benchmarks score that one shot: does it look right, does it run. Product generative UI almost never ships once. Users add filters, revise constraints, and attach external state. The same executable interface has to move with them, and still-valid earlier behavior has to survive.
WebArena-style suites test agents operating a website that already exists. FrontendBench and MiniAppBench test generating a page from scratch. The missing piece is the interface the model itself built, treated as a persistent interaction layer: can behavior, derived state, external state, and the assistant's claims stay in sync across successive revisions. NYU Shanghai calls that setting EvoGenUI.
EvoGenUI-Bench has 150 tasks and 750 turns, split evenly across three 50-task suites: information presentation, executable interaction, and tool-grounded external state. Each task is five turns of cumulative revision on one interface, not five independent pages. Every turn the model must return a user-facing reply plus complete source.
Generator-visible inputs are separated from private validation. The model sees the user request, public context, and tool contracts. Hidden requirements, backend state, and scoring criteria stay private. Each private requirement binds to at least one observable surface: screenshot, DOM, actor trace, source, reply, tool log, or runtime state. All tasks are human-authored; a second reviewer checks clarity, leakage, cross-turn pressure, and observability. Validation density is intentional: 3.0 private requirements per turn on Presentation, 5.9 on Interaction, 11.9 on Tool-grounded.
The harness builds returned source in a fixed React/Vite environment, serves it in a browser, and has an interaction actor click through the running UI. A turn passes only if Presentation, Execution, and Alignment each score at least 4. Presentation covers layout and domain fit. Execution covers whether the new request and still-valid prior requirements actually work. Alignment covers consistency among the reply, source, rendered UI, actor observations, and runtime logs. Build failures count as failures.
Four metrics share that official pass bit. Turn Pass is the per-turn rate. TP@5 is the share of episodes that pass all five turns. CPT is the mean length of the opening run of passes. APR is the probability that turn t passes given that turn t-1 passed and the generation call for turn t returned. The main run uses MiMo-V2.5 as both actor and evaluator.
Claude-Opus-4.7 is the strongest of the eight models: 74.9% overall Turn Pass and 83.6% APR, yet only 37.3% TP@5 and 2.81 CPT. Averaged across models, Turn Pass is 42.7% and TP@5 falls to 11.8%. Single-turn success does not buy a full revision sequence.
| Model | Turn Pass | TP@5 | APR |
| Claude-Opus-4.7 | 74.9% | 37.3% | 83.6% |
| GPT-5.5 | 61.7% | 21.3% | 71.0% |
| Claude-4.5-Haiku | 51.1% | 13.3% | 65.7% |
| Qwen3.6-Plus | 44.0% | 7.3% | 61.7% |
| Eight-model aggregate | 42.7% | 11.8% | 66.5% |
Tool-grounded work is the hard suite: 25.0% mean Turn Pass, 5.0% TP@5, and APR still only 52.4% (Presentation 71.1%, Interaction 68.7%). Only Opus and GPT-5.5 reach 20.0% TP@5 there; everyone else is 0. Pass rates sag after turn 2: 39.4% at turn 3, 35.1% at turn 4. Tool-grounded drops from 39.5% at turn 2 to 18.5% at turn 3 and 14.0% at turn 4.
Failure mechanisms split by suite. Among 2,750 executed non-passing calls: 859 information architecture, 586 derived-state propagation, 460 affordance binding, 410 requirement decomposition, 289 external-state grounding, 146 domain representation. Presentation fails mainly on information architecture. Interaction fails on stale derived views and controls that look live but are unwired. Tool-grounded adds mismatches with backend state and missed requirement splits. A post hoc audit of 110 failed APR transitions finds 52.7% involve regression on prior behavior and 47.3% fail only the new requirement.
On 240 blinded turns, the automatic evaluator hits 86.7% accuracy and Cohen's kappa 0.73 against majority human labels. An evidence ablation on a separate 240-case set drops from 87.5% with full evidence to 55.0% without the actor trace, worse than dropping the screenshot (78.3%).
If generative UI evaluation stops at a screenshot or a single-turn score, "can ship a page" gets read as "can maintain an artifact." Real use is almost always successive revision: add a filter, change a constraint, attach a backend. This benchmark moves the unit of evaluation from one page to one revision trajectory, and it insists that behavior, derived state, external state, and spoken claims stay aligned.
For people building coding agents and generative interfaces, the reusable piece is the failure taxonomy, not the leaderboard. Layout collapse, dirty derived state, unwired controls, and mismatched tool tokens are engineering bugs. A stronger model does not automatically erase them. Opus still finishes only 20% of tool-grounded five-turn episodes.
Tasks are human-authored five-turn scripts in a locked React/Vite environment. There are no naturally occurring logs, accessibility checks, or device conditions. APR is an outcome-level retention rate, not a causal attribution of what broke. Tool-grounded tasks use deterministic mock runtimes, so they miss latency, auth failures, rate limits, and API drift.
Actor and evaluator are the same MiMo-V2.5, which risks self-consistent grading. Alternative pairings agree with humans at 86.1% to 91.4%; the default sits at 89.7%, not an outlier on the high side. Three independent Qwen3.6-Plus runs move overall Turn Pass between 39.7% and 46.4%. Absolute scores wobble; the suite difficulty order does not. Tool-grounded contracts are also much denser, so cross-suite gaps mix "external state is hard" with "more checks per turn."