Training AI to Paint with Code via Reinforcement Learning

kastnerkyle · x · 2026-08-25

This project trains an LLM to generate images by writing p5.brush JavaScript code using reinforcement learning, making the output editable code rather than static pixels. The system runs a four-step loop: the model receives a prompt and generates a sketch, which is rendered via Puppeteer into a PNG. A judge model compares this against reference paintings to produce a reward signal for GRPO updates. The core challenge explored is defining reward functions for subjective aesthetic tasks.

Original post →

Training AI to Paint with Code via Reinforcement Learning

More from Multimodal