Meta previews WildArtifactBench: Elo-based evals for open-ended multimodal agents
qinzytech · x · 2026-10-16
Meta previewed WildArtifactBench, an internal evaluation framework for agents on complex real-world tasks where outputs are open-ended artifacts (websites, games, files) with no single ground-truth answer.
- Method: instead of strict ground-truth rubrics, it uses win rates and Elo scores from human and agentic preference judges, expanding coverage of practical multimodal workflows.
- Tasks: 10 tasks released, e.g. bird song identification—classify 46 audio files against textual species descriptions in YAML, output a CSV, and no birdnetlib-style libraries allowed.
- Leaderboard: cost-score curves show Claude Opus 5 on top in both automatic and human evals ($40/task), with Muse Spark 1.2 and GPT-5.5 near GPT-5.6 Sol in the low-cost range ($2/task), well above Muse Spark 1.0, GPT-5.4, Opus 4.6 and Gemini 3.1 Pro.
Related event: Meta Open-Sources Muse Spark 1.2 and Previews WildArtifactBench(2 posts)→
More from Models
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23
- Gemini refuses to generate image in Docs, blames document size limit — viksit · 2026-10-23
- Dev jokes his true calling is telling everyone to try the latest DeepSeek — intellectronica · 2026-10-23
- SubQ Snap 1.1 ties for first on Artificial Analysis Decision Index with 384K context — ArtificialAnlys · 2026-10-23
- PolyAI unveils Dialog-RSN-1: audio-native voice model answering in under 300ms — matthen2 · 2026-10-23