An Embodied Companion for Visual Storytelling
Patrick Tresset, Markus Wulfmeier
cs.HC, cs.AI, cs.GR, cs.RO
2026-01-18
A Gemini-driven 4-DoF drawing arm co-sketches stories via speech and tool calls. Seven art experts score aesthetic identity 6.0/7; machine intent is lowest at 4.71.
Patrick Tresset spent fifteen years building drawing robots whose job was to keep the artist out of the mark. Autonomy produced consistent pictures. It also produced pictures that felt empty of lived experience. A curator later named the gap: the artist is absent.
Inside the EACVA project, Tresset and Google DeepMind's Markus Wulfmeier reverse that stance. Companion is not another autonomous draftsman, and it is not a text-to-image model traced by a plotter. It is a Gemini agent with a physical arm, a camera, a microphone, and a speaker, taking turns with a human on a 170 × 130 mm sheet to tell visual stories, using the simple motifs of children's drawings and cave paintings.
The hardware is a low-cost 4-DoF planar arm (shoulder, elbow, wrist, pen-lift) on Dynamixel servos. Backlash, compliance, and a flexible frame are treated as style, not defects: change the speed and the line quality changes. A Logitech C920 sits on a pan-tilt mount. Talk is the main interface. Rotate the shoulder about 10 degrees and recording starts; silence stops it. After adding marks on the page, rotate the camera to capture a frame. Low-level control runs on YARP rather than ROS.
A Python agent talks to the Gemini API. System instructions cast the agent as an imaginative storytelling drawing robot, with cave paintings for composition and children's drawings for narrative. Stick figures for people, splines for organic forms, slight asymmetry preferred. The simulated canvas is 1200 × 900 pixels, with a 40-pixel buffer between new and existing marks.
SVG generation was tried and dropped: the look was too predictable. The model instead calls drawing tools: scribble fill, hatching, circles, segments, Hershey text, cubic splines, and a scribbly-spline that oscillates along a center path. Function-calling parameters become robot paths.
Style is injected with in-context learning: 24 hand-drawn sketches, each paired with a short method that Gemini itself wrote from the sketch. Ablation in Appendix C is blunt. Images without methods yield fragmented, odd drawings; the model can see the example but cannot translate pixels into tool sequences. Images plus methods transfer action grammar (hatched ground as an anchor, curves for organic shapes) onto subjects that were never in the vocabulary. The paper calls this procedural style transfer. Methods that are too precise make the robot stiff: it starts asking for coordinates.
Speech is synthesized with gTTS or eSpeak. English text is read under a French or Spanish language setting, so the voice carries a non-native accent on purpose.
Evaluation uses Amabile's Consensual Assessment Technique. Seven experts (five professional curators, one collector, one artist who works with robots) scored a portfolio and interaction videos on a 1–7 Likert scale. All of them already knew Tresset's earlier drawing robots, so the scores target Companion rather than the novelty of a robot that draws.
| Dimension | Mean | Notes |
| Aesthetic Identity | 6.0 (SD 0.81) | Highest; interrater rwg = 0.67 |
| Originality | 5.86 | |
| Global Quality | 5.71 | Panel treats the work as exhibition-ready |
| Conscious Presence/Intent | 4.71 (SD 1.11) | Lowest, widest disagreement |
Experts compared the look to prehistoric petroglyphs, Paul Klee, and Basquiat. One curator said authorship "feels shared, not simply delegated."
Role studies are qualitative. As artistic director, Companion added clothes and a hat on request, declined dashed arrows for time, and once drew a very long arm instead of repositioning a character to catch a feather. As illustrator, The Hare and the Turtle was readable even though neither animal is in the ICL set: the turtle is repeated six times, the hare three, a vertical line is the finish, the last hare is scribbled over. The Crow and the Fox failed until the user showed drawings to copy; cheese then appears three times (beak, falling, fox's mouth). In a two-hand session, a car drawn high on the page became a flying car on a scribbly cloud.
The same prompt from Gemini-1.5-flash to Gemini-2.5-pro goes from unrecognizable (while the model still announces what it is drawing) to clear. Gemini-2.0-flash-exp and ChatGPT-4o image generation kept a more conventional look and could not pack a whole fable into one frame. A 24-drawing book is scheduled, plus two participatory installations in a 2026 solo show.
For people building embodied agents or creative tools, this is a closed loop that already convinced an art-world panel: speech in, camera on the page, function calls out, physical line quality as style. The ICL ablation is the part an engineer can take home. Visual demos alone do not cross the seeing-to-motor gap; procedural methods do.
Do not read it as a general drawing model. The workspace is postcard-sized. The scores come from seven people, not FID or CLIP. What it shows is a way of collaborating, not a new generator.
The authors' own panel put machine intent at the bottom. Experts did not buy conscious presence. Out-of-vocabulary animals needed live copying. Visual-only ICL interfered with the model's prior knowledge. The main text has almost no quantitative rates: no success counts, no head-to-head numbers against SketchAgent or Bossema et al., and the model-generation comparison is six pictures of one task. Appendix A also records spatial misses: a circle meant to enclose prior marks failed on the first try, centering on a cross can be off by 1–2 cm, and a maze path to the exit could not be drawn.
Every rater already knew the first author's older work. That helps isolate Companion's contribution. It also makes Originality 5.86 hard to compare with a cold jury. The TTS accent and joint-rotation turn-taking are designed persona. Whether companionship survives a user who has never seen the apparatus is untested. Accessibility for arm-hand motor disabilities is named only in the last paragraph.