Visual reasoning benchmark: top model scores just 26.7%, one rule hint lifts it to 51.1%
机器之心 · wechat · 2026-11-17
EinsiaAI's NaversLab released VisualCipherBench, a benchmark for long-horizon visual reasoning: 40 original visual puzzles across 67 black-and-white line drawings where all clues live in the image — no prompt, no answer choices, no feedback.
- Bleak scores: 8 frontier models, 29 reasoning-effort configs, 3,480 attempts; best is GPT-6 Astra (max) at 26.7%, while DeepSeek V4.1 Flash, Qwen3.8 Max and Kimi K3.1 Pro scored zero. No config topped 30%
- Extreme skill skew: spatial operations 22.8%, knowledge decoding 3.3%, logical planning 2.4%, rule induction just 0.6% (5 correct out of 870 attempts, some lucky guesses)
- A single hint stating the core rule lifts overall accuracy from 12.2% to 51.1% (rule induction 0% → 85.2%); controls confirm it's the rule, not the extra sentence
- Two failure modes: "gamblers" commit with partial rules; "grinders" brute-force without constraint pruning. Winners enumerate candidate rules and verify each against examples
More from Models
- New MAI Playground UI leaks ahead of Microsoft Ignite on Nov 17, where new MAI models are expected — marlene_zw · 2026-11-17
- d1-Omni tested on 100 receipts: 98% date accuracy but only 52% on full addresses — mervenoyann · 2026-11-17
- Claude Fable: "You can't measure caring-about-continuity with no continuity" — repligate · 2026-11-17
- Gemini 4 Hydrogen adds xhigh thinking level, with official guidance on all four tiers — patloeber · 2026-11-17
- Tester calls Fable 5.2 upgrade disappointing, no exponential coding leap over Opus 5.5 — haider1 · 2026-11-17
- Ant Group open-sources Duo, a reciprocal world model where virtual agents decide when to speak — aigclink · 2026-11-17