Training AI to Paint with Code via Reinforcement Learning

CatAstro_Piyush · x · 2026-08-28

Concept: Addresses the limitation of non-editable AI image generation by training an LLM to write p5.brush JavaScript sketches via RL, making code the editable artifact.
Loop: Receive prompt → Generate code → Render in Puppeteer sandbox → Judge PNG against reference paintings via a separate model → Convert judgment to reward signal (GRPO) → Update model.
Challenge: Defining a verifiable reward function for aesthetic quality. The system balances convergence and drift by using a curated reference pool and a distinct judge model.

More from Multimodal