Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
Hongyang Du, Lan Yan, Christian Flores, Asim Kadav
cs.AI, cs.CV
2026-09-19
Adobe grows a natural-language skill bank around a frozen design agent: skills 76→139, Claude-Sonnet-4 GenEval2 success 72.7%→99.3%.
Professional graphic design is a long-horizon agent task. A finished, editable artifact comes out of dozens of interdependent tool calls. Terminal scores barely identify which step caused success or failure, and briefs mix hard constraints (text, color, placement) with subjective ones (hierarchy, composition). There is no unit-test oracle.
Weight updates are a poor fit. The backbone is often a hosted frontier model, demonstrations are expensive, and outcome rewards are noisy. Photoshop-style macros are too rigid; whole-trajectory memory is too broad. What is missing is a procedure that sits between an atomic tool and a full rollout: specific enough to steer, general enough to transfer.
A frozen language model drives equivalents of Photoshop, Illustrator, and InDesign through more than 230 tools, retrieves real assets, and renders intermediate documents for multimodal inspection. The only learned object is an external skill bank. Each skill is a natural-language playbook. At runtime the agent retrieves the top-k skills, injects the playbook, and shrinks the visible tool list. Turning retrieval off recovers the original agent, which is the control.
The bank grows on two axes:
Proposal and admission are split. A matched replay gate freezes upstream context (retrieved assets included) and compares candidate versus incumbent pairwise. A change ships only if it wins at least one prompt and loses none. Absolute grader scores are unused because VLM scores drift. Five rounds consume 1,406 briefs and 1,869 auto-graded trajectories, with no human labels and no weight updates.
The bank grows from 76 documentation-derived skills to 139. The gate rejects 100 of 231 rewrites and 67 of 136 mints. Minted skills sit farther from the seed bank (median nearest-neighbor 0.215 vs 0.158 inside the seed), so widening covers missed intents rather than paraphrasing the manual.
On 200 fixed human briefs, round 5 lifts the share of cases at completeness ≥0.5 from 86% to 93%, ≥0.9 from 43% to 56%, and perfect from 24% to 32%. The path is not monotonic: round 4 regresses on the low end because freshly minted skills have not been rewritten; round 5 commits 21 rewrites and 4 mints and recovers both tails.
| Backbone | Metric | Base → Evolve |
| Claude-Sonnet-4 | GenEval2 success | 72.7% → 99.3% |
| Claude-Sonnet-4 | DPG-Bench success | 82.7% → 100% |
| Claude-Sonnet-4 | GenEval2 quality | +11.99 |
| Claude-Opus-4.6 | design-bench win rate | 67.6% |
| Claude-Sonnet-4 | design-bench win rate | 61.8% |
The ablation is the load-bearing result. Rewrite-only or mint-only win 48.6% and 49.4% on the 200 held-out briefs; together they reach 58.5% (p=0.025). Completeness moves from 68.62 (cold start) to 74.04; aesthetics barely move (65.92→66.53). Latency overhead is 3.4%–6.2%.
This is a route to continual adaptation from user traffic without touching weights, aimed at creative software with huge tool catalogs and unverifiable outcomes. Widening adds coverage, deepening makes it reliable, and the two have to couple: new skills need rewriting to stabilize, and unfixable failures feed the minting pool. Documentation skills alone even lose to the no-skill agent (46.4% win rate). Stuffing a manual into context is not the same as evolving procedures.
For teams that already have a tool-using agent, the replay gate is the piece worth copying before another fine-tune.
A natural-language skill can describe a preferred procedure; retrieval does not force the model to follow it against a strong default. Fine geometry is limited by perception and by automated verification. Long procedures lose fidelity across many steps. The replay gate is local to the replayed cases and does not guarantee monotonic improvement on the full traffic mix; round 4 already showed a low-end regression. Scores and win rates rest on VLM/LLM judges, which still sit short of designer preference. Personalization skills are excluded from the public pool. Qwen3.6-27B posts high quality with low success, because quality is computed only on successful outputs and tilts toward easier prompts.