VisualArtifactSpec 1.0: 211-task benchmark for agent visual artifact creation
Accio-Lab · hf · 2026-11-18
Accio-Lab released VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations/videos, and web interfaces, testing how agents translate task specs and source materials into practical visual outputs.
- Two axes: Task Fulfillment (explicit requirements and functionality) plus Artifact Performance (Content, Organization, Visuals, Delivery), with task-specific rating anchors per audience, purpose, and medium
- Results: nine model-harness configurations scored 42.12–70.52 overall; Artifact Performance accounts for 73.4% of the gap between best and worst configurations
More from Research
- EACL 2027 results: 1,118 papers accepted from 4,136 submissions, main track rate just 13.5% — IAugenstein · 2026-11-18
- PhilBench quizzes 97 LLMs with the 2020 PhilPapers Survey of 1,785 philosophers — pwlot · 2026-11-18
- CADMAGEDDON: open-source agent rebuilds editable CAD locally on a single GPU — Oleg Pavlov · 2026-11-18
- MillBill: local 27B model beats frontier models at generating machining programs — Gennadii Savrasov · 2026-11-18
- The 200-year prehistory of speech synthesis: von Kempelen's 1791 speaking machine to 1970 — ArtificialOther · 2026-11-18
- Ex-DeepMind's Ryutaro Tanno joins AI biotech Pathos as Chief AI Officer — davidstutz92 · 2026-11-18