TuringSpace: 125 interactive spatial CAPTCHAs stump frontier MLLMs, humans hit 94.1%
jiank_uiuc · x · 2026-11-21
A paper from UPenn, Minnesota, USC, Meta, MBZUAI and others introduces TuringSpace, 125 interactive spatial CAPTCHAs that models must solve by clicking and dragging on screen, spanning five spatial abilities. Each task is a program generating fresh instances with checkers and oracles, doubling as a scalable training gym.
Key findings:
- Humans solve 94.1% of tasks; the best model, GPT-6 Astra, still fails 25.9%, and the best open-weight model, Kimi K3, fails 68.0%
- Most failures stem from misreading screenshots; model reasoning rarely corrects a wrong belief and instead hallucinates evidence to defend it
- Fine-tuning Qwen3.5-9B on the gym lifts success from 1.6% to 33.1% (beating Kimi K3) and improves all seven out-of-distribution benchmarks, including a 9.9-point gain on real-world indoor videos in VSI-Bench
The takeaway: grounding spatial intelligence in screen actions exposes MLLM weaknesses and offers a scalable way to teach it. The benchmark will be released openly.
Related event: TuringSpace Benchmark: Top MLLMs Still Fail 26% of Interactive CAPTCHAs(2 posts)→
More from Models
- Tesla FSD 15's real-time view isn't 3D rendering — it's the AI model's own synthetic perception — mark_k · 2026-11-22
- Gemini Black Friday half-price: JP-region trick gets Pro annual plan for $91.54 — huangyun_122 · 2026-11-22
- Microsoft's USB4 RDMA lets mini PCs pool memory to run Qwen3-235B — JFPuget · 2026-11-22
- Google AI Mode now enforces usage limits, users surprised by rate-limit popup — elvano124 · 2026-11-22
- Sol 6.2 now refuses reverse engineering that Sol 6.1 happily did for a month — cardak98 · 2026-11-22
- Anthropic's record ~$81B IPO, GPT-6.2 pricing, Qwen4-Max and Copilot Autopilot: week in review — ksraj1001 · 2026-11-22