TuringSpace: interactive CAPTCHAs expose spatial gaps — GPT-6 fails 25.9%, Kimi K3 fails 68%
meta · hf · 2026-11-20
Researchers from Penn, UMN, USC, Meta, and MBZUAI present TuringSpace on arXiv: a synthetic web environment of 125 interactive CAPTCHAs spanning five spatial abilities, solvable only by clicking and dragging — the same observe-act-correct loop humans use.
- Humans solve 94.1%; the best model, GPT-6 Astra, still fails 25.9%, and the best open-weight model, Kimi K3, fails 68.0%
- Most failures come from misreading screenshots; model reasoning rarely corrects a wrong belief and instead hallucinates evidence to defend it
- Tasks are programmatically generated with a checker and oracle, doubling as a training gym: fine-tuning Qwen3.5-9B lifts success from 1.6% to 33.1% (above Kimi K3) and improves all seven out-of-distribution benchmarks, including +9.9 points on real-world indoor videos of VSI-Bench
- Conclusion: grounding spatial intelligence in screen actions exposes MLLM weaknesses and offers a scalable way to teach them, with gains transferring to real scenes. TuringSpace will be released
More from Models
- Mamba+MoE hybrid reasoning model Olala-v1.0-mini-7A1B trends on Hugging Face — OVHaiLLM · 2026-11-20
- Local inference users want 512GB RAM and 1M context, says current boxes fall short — AIFlow_ML · 2026-11-20
- Astra xhigh is surprisingly good at ML engineering, says tester — _Stocko_ · 2026-11-20
- Leaked slide shows OpenAI's unannounced cyber model hitting 77% on ExploitGym benchmark — TheMoonMidas · 2026-11-20
- Beth Barnes clarifies METR didn't evaluate alignment of Fable/Mythos 5.2 — RyanGreenblatt · 2026-11-20
- Tencent Hy4 lands in vLLM day-0: 774B MoE with 49B active, 1M context, 8-GPU FP8 recipe — vllm_project · 2026-11-20