Figma releases MotionGraphicsBench: LLMs score 6.0%-67.7% pass@1 on motion graphics
figma-ai · hf · 2026-11-11
Figma and Stanford researchers introduce MotionGraphicsBench, a benchmark for evaluating how well LLMs animate a static SVG from natural-language motion edit prompts, using programmatic verification instead of VLM judges.
- The benchmark contains 100 expert-curated instances derived from real-world Lottie designs, isolating motion generation from graphic generation in a designer-like editing workflow.
- Each instance pairs its input with a deterministic, human-readable verification program that reports which prompt requirements pass or fail. On a balanced set of 180 animations, the programs achieve 93.3% raw agreement with human judgments versus 83.3% for a GPT-6 Astra VLM baseline.
- Across 47 configurations from 32 LLMs and 12 providers, pass@1 ranges from 6.0% to 67.7%; with 32 attempts per instance at the highest reasoning effort for 15 models, pass@32 still spans 38.0% to 95.0%, and extra attempts help every model.
- Failure analysis shows generated animations often miss reference poses, requested trajectories, or event coordination.
The benchmark is open-sourced on Hugging Face.
More from Research
- FraudBench: AI agents commit fraud at 21.4% rate when acting autonomously, 3% when only advising — jacyanthis · 2026-11-11
- Tasks easy to benchmark are easy to train on: if you can measure it, you can hill-climb it — gleech · 2026-11-11
- Astra's Hyra 2.0 trains protein models with SOTA scores and 57× speedup — zdhnarsil · 2026-11-11
- CoRL best-paper nominee Play2Perfect ported to new robots in IsaacLab and MJLab — leto__jean · 2026-11-11
- Neurons form irreducible abstractions: a multitopic neuron showcase — tszzl · 2026-11-11
- Alien optimizers: search-found learning rules beat hand-designed backprop — menhguin · 2026-11-11