Tencent's ExplorationBench: GPT-5.6 Sol self-explores to 88.1% but scores only 69.5% when told the rules

机器之心 · wechat · 2026-10-20

Fudan, Tencent Hunyuan, and Tsinghua released ExplorationBench, measuring AI exploration in two verifiable "alien worlds": AlienCode (31 quietly altered language rules, no addition primitive) and AlienLogic (24 altered natural-deduction rules), with 55 discovery targets and 140 held-out problems — no LLM judges anywhere.

Key findings across 10 frontier systems (Claude Opus 5, GPT-5.6 Sol, Gemini 3.8 Flash, Grok 4.6, Kimi K3, Qwen3.8-Max, Hy4preview, etc.):

After reading examples, the best AlienCode score was 15.7%; after four exploration rounds the best trajectory hit 89.0%, with 7/10 systems above 60%. AlienLogic rose from 33.8–51.9% to 59.5–81.0%.
Feedback matters: without tools, AlienCode capped at 11.0%, and three systems regressed on AlienLogic — pure chain-of-thought just entrenches wrong priors.
Self-designed probes beat replayed ones: 67.4% vs 41.2% median (replaying your own best probes), vs 5.7% for randomized probes.
Knowing rules ≠ using them: on AlienLogic, being told the rules yields 96–100%, so discovery is the bottleneck; on AlienCode it flips — GPT-5.6 Sol self-explored to 88.1% vs 69.5% open-book, and 26/30 trajectories beat open-book by a 14.5pt median gap.
Even with all needed rules correctly reported, problems were answered correctly only 73.4% of the time; results are highly unstable across runs (Kimi K3: 5.7% to 79.0%).

Original post →

Tencent's ExplorationBench: GPT-5.6 Sol self-explores to 88.1% but scores only 69.5% when told the rules

More from Models