FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos
Maya Moriya, Sigal Raab, Yael Vinker, Tali Dekel
SIGGRAPH ASIA 2026
cs.CV, cs.AI, cs.GR
2026-09-01
Weizmann and MIT pair a VLM with an origami simulator to recover executable Pureland programs from demo videos. On 27 sequences the full agent compiles 96% of steps and finishes every sequence; asking the VLM for crease patterns compiles 8%.
Human origami knowledge lives in demos: diagrams and how-to videos. Computational origami wants a structured object, usually a crease pattern with mountain and valley labels, or an executable action list. Most prior systems assume that geometry is already written down, then analyse foldability, simulate, or invert a crease pattern into a sequence. Recovering each step from a real demo video is largely open.
The hard parts are concrete. Hands occlude the sheet. Layers stack until the interior is invisible. One fold can move distant layers and stamp new creases far from the visible line. Sequential prediction also compounds early mistakes. The paper restricts the domain to Pureland: every step is a flat fold along a single straight crease. fold, unfold, rotate, and flip cover the vocabulary, and every intermediate state stays a flat layout of polygons.
Input frames are not auto-sliced from raw video. Keyframes are extracted by hand. State is an extended FOLD graph: vertices, edges, faces, plus front/back orientation and an explicit layer order. Layers are grouped into planes, maximal sets of faces joined by flat edges, which move together and are recomputed after each fold.
The action space has five composable primitives: addvertex inserts a point at a fraction along an edge, fold reflects one side across an edge, unfold restores a prior fold to a crease mark, rotate turns the model in plane, flip reflects about an axis. A single keyframe transition often needs a short composition, such as insert-then-fold or turn-then-fold.
A zero-shot VLM agent (Gemini 3.1 Pro Preview in the main experiments) talks to a deterministic controller. Tools fall in four groups: simulator actions, viewing frames and motion strips plus a 2D render, checkpoint save/restore, and a separate visual critic. The critic sees a four-image grid of source/target photos against source/target renders, and returns Match, Mismatch, or Extreme Divergence. Proposer and verifier are split so the agent is less free to excuse itself.
On a stuck transition the agent rolls back. After three failed Match attempts it inspects the full checkpoint strip, finds the earliest drift, and restarts from an earlier trusted state. Exploration is capped at 5 attempts per keyframe and 300 tool calls per sequence. If the budget dies, a light selector picks the explored state closest to the target and freezes the committed prefix. Mean cost per sequence is about 140 queries, 4M input tokens, 68K output tokens, $8.9.
PurelandFold is a new benchmark: 27 self-captured teaching sequences from OrigamiWay's Easy Origami set, with keyframe action and geometry labels. Mean length is 12 frames, range 5 to 21. There is no drop-in prior method, so the ladder is three ablations.
| Method | Seq. completed | Compile CV | TSS | GS | CS | FFS | CPD↓ |
| VLM-CP, predict crease pattern | 96% | 8% | 0.76 | 0.20 | 0.20 | 0.30 | 363 |
| VLM-S, predict actions | 85% | 96% | 0.79 | 0.51 | 0.53 | 0.21 | 268 |
| VLM-S-C, action plus non-agent critic | 85% | 96% | 0.80 | 0.48 | 0.49 | 0.19 | 216 |
| FoldingAgent | 100% | 96% | 0.90 | 0.58 | 0.77 | 0.39 | 189 |
Switching representation turns most outputs from uncompilable to valid geometry. The full agent’s extra gain is in topology, constraints, and final layering, because it can catch an early bad fold and rewind. A 600-judgment study on 10 sequences completed by all variants prefers FoldingAgent renders over VLM-S 84% of the time and over VLM-S-C 80%.
On a 5-sequence backbone check, GPT-5.5, Claude Opus 4.8, and Gemini 3.1 all improve under FoldingAgent; plain VLM-CP finishes 0/5 on GPT and Claude. The critic itself is right about 88% of the time. Wrong approvals propagate; rollback exists for that failure.
The pipeline turns "watch someone fold" into a program you can edit, simulate, and eventually hand to a robot. For graphics this is inverse origami. For robotics it is a route to harvest procedures from the pile of teaching videos already online. The recipe is ReAct plus a domain simulator. The contribution is the action space and the rollback protocol, not a newly trained origami net. It is also expensive: nearly $9 and millions of tokens per sequence, with human keyframes.
Quality tracks the underlying VLM. Heavy occlusion, by hands or by stacked layers, still fools the model. Simultaneous compound actions, rotate-while-flip, break the sequential primitive set. The scope is Pureland; richer folds need a larger action space. Keyframes are manual. Videos are self-captured on high-contrast Bristol board, not noisy internet clips. There is no end-to-end number on uncut long video. At $8.9 a sequence, bulk recovery of online origami is not close.