RULER: Instance-aware Rubric Rewards for SVG Generation
Hangyu Ran, Yuhao Zheng, Yingying Zhang, Kevin Qinghong Lin, Han Peng
cs.CV
2026-09-22
RULER builds a six-item rubric per SVG prompt, scores renders with a VLM, and trains GRPO. Qwen3-8B rises from 0.432/0.395 to 0.693/0.683, matching DeepSeek-V3.
Text-to-SVG has no single correct image. CLIP and aesthetic classifiers were calibrated on natural photos, so they often rank a broken vector drawing above a faithful one. Used as RL rewards, they invite hacking: aesthetic score climbs to 6.697 while the Icon rubric falls to 0.262 and mean length balloons from 0.3k to 6.3k tokens. Supervised fine-tuning mostly clones dataset templates. The missing piece is a training signal that needs neither paired SVG ground truth nor an easy scalar to game.
On 900 human-rated samples, a vision-language judge scoring a multi-axis rubric tracks people better than those scalars: Spearman ρ is 0.7929 versus 0.6051 for Aesthetic and 0.5518 for CLIP; pairwise Goodman-Kruskal γ is 0.7574 versus 0.5465 and 0.5295.
RULER turns that evaluator into a reward. A frontier model (mainly Claude-Opus-4.6 in the experiments) reads only the instruction and writes a six-item rubric spanning semantic fidelity, visual quality, and rendering style. Items describe design intent, not pixel or path constraints, so the checklist does not collapse into reconstruction. The policy samples SVG, a renderer produces an image, and a judge VLM scores each item in [0, 1]. The weighted average is the GRPO reward. Rollouts for the same prompt usually succeed on different axes, which supplies the within-group spread GRPO needs.
The backbone is Qwen3-8B with thinking mode off, eight rollouts per prompt, 4096-token responses. Training uses 20K MMSVG-Icon and 12K Illustration prompts. The headline metric is a separate universal rubric scored by GPT-5-mini, so the training judge does not grade itself.
On MMSVG-Illustration / Icon, rubric score moves from 0.432/0.395 (Qwen3-8B) to 0.693/0.683, above OmniSVG, IconShop, and JanusCoder, and slightly above DeepSeek-V3 at 0.686/0.673. CLIP and HPS stay competitive; the rubric gain is not paid for with alignment.
In a 150-prompt blinded study, non-tie win rate is 89.7% against Qwen3-8B, 96.5% against JanusCoder, and only 53.3% against VectorFusion. Holding the optimizer fixed, CLIP+Aesthetic+HPS hacks the aesthetic axis; a universal rubric reaches 0.660/0.591; instance-aware items add another 0.033/0.092. Dropping the visual-quality axis hurts most (0.693→0.580). A stricter Rubric-S prompt invites drawing the query as readable text, and the universal score falls to 0.536.
A 4B backbone still rises from 0.372 to 0.561 on Illustration. Swapping the rubric writer to GPT-5.5 yields 0.646 on Icon versus 0.662 with Claude.
For open-ended visual code, the bottleneck is the reward, not the optimizer. Instance-aware items turn “does this look right” into checkable sub-goals, and an 8B model can match a much larger generalist. The recipe is copyable: text-only rubric, render, VLM score, GRPO. Scalar metrics remain useful as side reports. They are a poor sole objective.
Two external models write and grade the rubric, so their taste and failure modes enter the reward. Every step renders and queries a judge VLM, which is far costlier than CLIP and will not scale cheaply. The primary eval metric is still a VLM rubric, in the same family as the training signal; the human study covers 150 prompts. The three-axis split will not capture every artistic intent. The C+A+H collapse already shows that a new rubric spec can open a new shortcut.