Training an LLM to paint by writing code: RL meets creative tasks with a judge model

measure_plan · x · 2026-09-19

A thesis project by Surya Narreddi and Cameron Franz trains a language model to make images by writing code using reinforcement learning. The motivation: AI image generation locks you into prompting, while code is an editable artifact you can tweak granularly without re-prompting.

Training runs a four-step loop thousands of times:

The model receives a prompt (e.g. "draw a peach hibiscus in watercolour") and writes a complete p5.brush JavaScript sketch;
The sketch is rendered to a PNG in a sandboxed Puppeteer environment;
The PNG is compared against two random reference paintings from a hand-rated pool, with a separate judge model picking the better watercolour;
The judgment becomes a reward signal, GRPO updates the model, and the loop repeats.

The deeper question: how to do RL on creative and design tasks, where rewards aren't verifiable like math or games. The design problem becomes the reward function and judge criteria — too rigid and the model converges, too loose and it drifts.

Original post →

Training an LLM to paint by writing code: RL meets creative tasks with a judge model

More from Multimodal