ScaleAI's HSS benchmark: humans score 93.1%, best model only 53.6% on visual intuition
新智元 · wechat · 2026-10-17
ScaleAI and Elorian released Humanity's Sixth Sense (HSS), a benchmark for intuitive visual reasoning: 522 open-ended tasks (288 images, 234 videos totaling 17.6 hours) across 4 domains — temporal/causal dynamics, physical/spatial logic, social understanding, and abstract/in-context inference. Answers must not be readable from pixels; only 15.1% of 3,466 candidate questions survived three review rounds.
Results: humans 93.1%; top model GPT-6-astra at max reasoning 53.6%; median across 25 models 30.9%. Blind (text-only) testing drops the best model to 6.6%, confirming it tests vision, not language priors. Social understanding is weakest (24.4% avg); 23/25 models do worse on video.
Error analysis: of 8,573 failures, 94% stem from perception or latent-inference errors, not logic — top causes: missed key cues (21.3%), misidentifying objects/people (19.9%), size/distance misjudgment (12.2%). Multiple models often fail the same question for the same reason, suggesting shared visual blind spots.
Agents help only partially: with Claude Code/Codex tools (crop, zoom, re-frame), scores jump (Claude-Fable-5.1 → 57.7%; GPT-6-astra → 59.3%), but 2D-overlap-mistaken-for-3D-alignment errors barely move (102 → 101) — the gap is a missing stable 3D representation, not a magnifying glass. Next-gen multimodal systems need better object binding, 3D representation, and temporal modeling, not longer chains of thought.
More from Models
- Quick take: GPT-6 matches Opus 5.5 at coding, far ahead at computer use — _vztu · 2026-10-18
- Open models now above Sol-level, outpacing expectations as community awaits V4.1 Pro — yacineMTB · 2026-10-18
- Dev's model vibes: Opus 5.5 writes best, Sol 6.1 best value, Sonnet skippable — ricklamers · 2026-10-18
- Codex took 3 days, Opus 5.5 nailed it in an afternoon — banned user's video protest — dotey · 2026-10-18
- Yacine MTB: Users will remember how Anthropic treated them once intelligence proliferates — yacineMTB · 2026-10-18
- Mistral's Timestral forecasting models start at 5M params and run on plain CPUs — lmoroney · 2026-10-18