64 humans think aloud on ARC-AGI-3: dataset shows humans and LLMs learn worlds differently
xuanalogue · x · 2026-10-31
An NTU-led team released a think-aloud dataset on ARC-AGI-3: 64 participants produced 4–7 full-solve trajectories per game across 25 public games with time-synced audio, annotated against source code to pinpoint when each mechanism is discovered, plus 250 public LLM agent trajectories for comparison. Findings: LLMs anticipate mechanisms before triggering them more often (7–11%) than humans (4%); humans form qualitative mechanism accounts in 50% of trajectories vs 14% for LLMs, which describe mechanisms precisely enough to implement as code; uncertain humans explicitly say so 37% of the time. A rare process-level dataset on how world models form during interaction.
More from Research
- New paper proposes the Piggyback Hypothesis: chat-template tokens carry emergent misalignment out of domain — davidbau · 2026-10-31
- FPV-SLAM claims 1st place on ETH Zurich's city-scale LaMAria mono-inertial benchmark — ravithejads · 2026-10-31
- FDSML (WACV 2027): fractal-parameter metric learning lifts image retrieval up to +22pt — HirokatuKataoka · 2026-10-31
- Early OpenAI interpretability researcher: most neurons in trillion-param models are now legible — CatAstro_Piyush · 2026-10-31
- FDIF pretrains on formula-generated synthetic 3D data, hitting 81.89% without real medical images — HirokatuKataoka · 2026-10-31
- Image-Space Rule Discovery paper accepted to ACCV 2026, updated arXiv version released — HirokatuKataoka · 2026-10-31