Visual Cipher Bench: best frontier model solves only 26.7% of long-horizon visual puzzles
_akhaliq · x · 2026-11-17
Einsia AI, Navers Lab, and Tsinghua University introduce Visual Cipher Bench, testing whether agents can discover rules hidden in images with no written rules and no correctness feedback within 60 minutes.
- 40 original puzzles (spatial, logic, rule induction, decoding), 67 drawings, 8 frontier models, 29 configurations, 3,480 graded runs.
- Best score: GPT-6 Astra at 26.7%; GPT-6.1 Sol and Claude Opus 5.5 at 21.7%.
- Three models — DeepSeek V4.1 Flash, Qwen 3.8 Max, Kimi K3.1 Pro — solved zero puzzles; two others managed only 0.8%.
Takeaway: coding agents grind long tasks with specs and test suites, but long-horizon visual reasoning without feedback remains a near-total failure mode. Paper, code, and leaderboard are open.
Related event: New Visual Cipher Bench Leaves Best AI Models at 26.7% Accuracy(2 posts)→
More from Models
- User's open letter to OpenAI: Dot agent degraded after a week, switching to Claude Duet — Danny7Guns · 2026-11-17
- Rumors swirl of OpenAI releasing GPT-6.2 Astra and Sol today, ahead of Anthropic IPO — kimmonismus · 2026-11-17
- Gemini 4 models in Antigravity reportedly skipping thinking entirely, even at High/xHigh effort — lyraxana · 2026-11-17
- Qwen Max going API-only isn't a pivot — it's the same layered open-weight strategy for years — natolambert · 2026-11-17
- "I don't even know why I'm using fable-5.2": dev admits model FOMO keeps him hooked — adonis_singh · 2026-11-17
- Gemini 4 Carbon cache hit rate averages 85.2%, raising cost-efficiency doubts — mariofilhoml · 2026-11-17