Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation
Tianyi Xiong, Zhengyuan Yang, Xiaofei Wang, Chung-Ching Lin, Ruichun Ma, Kevin Lin, Zhendong Wang, Linjie Li, Chenxi Liu, Ruibo Chen, Ramani Duraiswami, Heng Huang, Lijuan Wang
cs.CV
2026-08-25
A local CSS edit can wreck already-correct UI regions. RubSE picks one typed rubric per round and gains +1.20 points over naive self-evolution.
Screenshot-to-code already produces runnable HTML/CSS. Iterating at test time from the rendered image looks like free quality. On Design2Code, GPT-5.2's naive loop finished 2.73 points below round 0, with all 15 rounds worse than the first draft. GPT-5.4 gained a little at round 1, then fell below the initial sample.
The paper calls this visual repair coupling. Edit a parent, a flex rule, or a missing widget, and layout plus style dependencies move other regions. Free-form critique says what is wrong. It does not say what must stay still, so a fix becomes a rewrite.
RubSE writes visual feedback as a rubric: title, type, description. Types are fixed: layout geometry, spacing density, typography, styling, completeness. Invalid types are dropped so candidates do not all pile onto the brightest color bug.
Each round has three moves. Evolve proposes K typed rubrics from the target shot, the last code and render, and prior rubrics as an avoid list. Select keeps one rubric expected to help under a local edit. History appends that rubric. The code model then revises conditioned on the target, the previous state, and that single rubric. Evolve and Select are separate calls, so listing defects is not mixed with ranking them.
Six VLMs, three benchmarks, ten rounds. Final-round RubSE beats naive self-evolution on both metrics in 15/18 settings, averaging +1.20 overall and +0.11 aspect. Best-round wins 14/18, +1.13 overall. GPT-5.4 on UI2Code-Real goes from 85.6 to 87.9 at round 10. On Design2Code, naive self-evolution drops from 86.9 to 85.8; RubSE ends at 87.8. GPT-5.2 on Design2Code is the sharp case: naive 83.9 versus 86.4 direct, RubSE 87.2.
Frontier models hit their best checkpoint later (round 5.7 vs 3.0) and gain more over the first draft (+2.22 vs +0.77). Open-source Qwen3.5-9B and Qwen3.6-35B-A3B improve in both final and best rounds. Qwen3-VL-32B is mixed: it loses overall on UI2Code-Real and Design2Code-HARD, which the paper traces to over-weighting spacing and under-weighting completeness.
A collapse is an 8-point overall drop plus 0.35 aspect in one step. Frontier collapse falls from 18.9% to 12.8%; GPT-5.4 recovery rises from 18.5% to 50.0%. GPT-5.4 rubrics transferred onto Qwen improvers beat Qwen's own rubrics, especially early, because they talk about layout relations rather than named pixels. GPT-5.4 API cost is about 1.60× naive self-evolution.
For generation tasks whose artifacts are coupled, test-time iteration needs scoped repair context, not an unconstrained critic. A single typed rubric is a cheap control knob, and a strong model can write those rubrics for a weaker coder. UI-to-code agents can drop this loop on an existing VLM without retraining the generator.
The gain is a stabler trajectory, not a jump to production UI. One to two judge points on a 0-100 scale can be subtle; a 60-sample human study still preferred RubSE.
The study is HTML/CSS only. RubSE does not check that the code still parses, so a round can white-screen. The judge is GPT-5.2 averaged over three runs, and it can disagree with humans on late, tiny diffs. The Qwen3-VL-32B miss shows rubric quality is tied to the executor's own perceptual bias. Two extra VLM calls per round raise the bill. There is no comparison with agents that compile and debug first, so the increment over "make it run, then look" is unknown.