Weak LLMs spot good research ideas they cannot generate, across 200 ICML papers

Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game

Muhan Gao, Hangxiao Zhu, Xinqi Su, Yu Zhang, Daniel Khashabi, Kuan-Hao Huang

cs.AI

2026-11-11

Six LLMs rediscover the core ideas of 200 ICML 2026 papers by yes-no questions, with almost all gains in the first 30, and weak models can judge ideas they cannot generate.

What problem this solves

Research ideas move by feedback, and current evals hide that search. One-shot ideation benchmarks ask an LLM judge to score a single proposal for novelty, so they never show an idea being rewritten. End-to-end science-agent benchmarks fold the idea together with coding, debugging, and experiments, then return one outcome. A miss can be a bad idea or a broken run.

Re-Search measures the search alone. The format is Twenty Questions: one side holds a hidden answer, the other may ask only yes-or-no questions. The hidden answer is the core method of a published paper. The inquisitor sees the research problem and may ask up to 100 questions. An answerer that has read the full paper replies Yes, No, or Unknown. Execution is gone. Each question can be lined up with the idea the model writes down later.

Method

Each game is a pair (p, s). The problem p has four fields: domain, goal, obstacle, and the gap in prior work. The solution s has four: contribution, approach, technical pieces, and what is new. The inquisitor sees only p. The answerer reads the paper, returns one of three labels, and keeps no dialogue state. Every 10 questions the inquisitor files an idea card, a short guess at s. Three LLM judges, Gemma-4-31B, Qwen3.6, and GPT-OSS-120B, score the card against s from 1 to 10. The mean is the recovery score. Scores are not shown back, so the model can climb only on the yes-or-no answers. Pairwise Spearman correlations among judges run from 0.85 to 0.88. On 498 card pairs, human raters reach Fleiss' kappa of 0.83 and match the judge majority 91.2% of the time.

The set is 200 ICML 2026 method papers, all first public after 31 August 2025, later than the models' reported knowledge cutoffs. GLM-5.2 extracts p and s. Three CS PhD students checked 100 papers; 93% are correct by majority vote, and the misses mostly drop a component that separates the method from others. The answerer is always Gemma-4-31B. On 795 human-labeled answers it agrees 94.0% of the time, against 86.2% for GPT-OSS-120B and 78.4% for Qwen3.6, and it swaps Yes and No on 1.1%. The inquisitors are that same Gemma plus GPT-OSS-120B, DeepSeek-V4-Flash, Qwen3.6-35B-A3B, Claude Sonnet 4.5, and GPT-5.4, three games per paper.

Two interventions split the bottleneck. GPT-5.4, Qwen3.6, DeepSeek-V4-Flash, and GPT-OSS each propose a question from the same history, and one of them picks. A second setup attaches a survey of 5 to 7 dimensions with 4 to 6 known technique families each, visible while asking and hidden while the idea card is written. Cfull names the paper's real method. Cdecoy replaces only that phrase with a plausible decoy. One sentence differs.

Results

Opening cards score very low. The first 10 questions add 1.3 to 3.2 points, and questions 11 to 30 add 0.2 to 0.8. After that, DeepSeek-V4-Flash, GPT-OSS-120B, and Qwen3.6 barely move across the remaining 70 questions. Gemma-4-31B and Claude Sonnet 4.5 climb slowly. Only GPT-5.4 keeps rising. It leads by a wide margin, Gemma and Sonnet follow, and the other three finish at about two thirds of its score. The paper does not list a final mean for each model.

Early questions lock the ending. The best and worst openings are cut at round k and each continued three times. Correlation with the score at question 100 is r = 0.76 to 0.87 at question 10, and r = 0.87 to 0.92 at question 30. Across cuts at questions 10, 20, and 30, the better prefix still leads at the end on 63% to 79% of papers in Table 1. On a 100-paper subset GPT-5.4 beats GPT-OSS-120B by 2.54 points. On the 24 papers where both can list the core idea in a closed-book probe, the gap is still 2.28. Prior knowledge closes about a tenth of the gap.

ComparisonResultBaseline
Gain in first 10 questions+1.3 to +3.2+0.2 to +0.8 on questions 11 to 30
Question 30 vs finalr = 0.87-0.92r = 0.76-0.87 at question 10
GPT-5.4 minus GPT-OSS2.54 points2.28 where both can name the idea
Shared pool, 30 questions5.19-5.34Solo near 4.2; GPT-5.4 drops 0.59
Self-sample 4, then pick4.16 to 4.57Cross-model pool 5.20; random pick 4.43
Naming the core ideaup to +1.25vs a one-phrase decoy, r = -0.74
Top-level branchespeak near 8.35.2-5.5 too narrow, 13.7 too wide

Whoever picks, the four models score 5.19 to 5.34 after 30 shared-pool questions. The weaker models gain +1.04 to +1.09 over solo play. GPT-5.4 loses 0.59. If GPT-OSS samples four of its own questions and then picks, it moves from 4.16 to 4.57. Random picks from the same pool score 4.43, not significantly above solo play. Naming the method adds up to 1.25 points, and the gain correlates with the unnamed score at r = -0.74, so the weaker the model, the larger the lift. Surveys list 20 to 42 candidates, yet models ask about the named idea within a median of 3 to 6 questions, in 67% to 89% of games, against 29% to 52% for the decoy sitting on the same line. GPT-5.4 finishes in nearly the same place under all four survey conditions.

The 1,200 games become trees. DeepSeek and Qwen open 5.2 to 5.5 top-level branches, and 23% of their questions enumerate variants inside a direction they already confirmed. GPT-OSS opens 13.7 and is the only model that switches branch more often after a Yes than after a No. The three stronger models open 7.2 to 8.2. With model and paper effects removed, recovery peaks near 8.3 branches. After a Yes, GPT-5.4 makes the next question a follow-up 39% of the time, against 24% after a No.

Why it matters

Weak models recognize a good question or a good method once it is on the table. Every selector in the shared pool finishes near 5.2, while four draws from GPT-OSS itself only move it from 4.16 to 4.57. Bring candidates in from outside, and let a weaker model choose. Putting GPT-5.4 on that pool costs it 0.59, so the strong generator should keep writing its own questions. A survey that names the idea helps the weak models most and barely moves GPT-5.4.

The time to intervene is question 30. DeepSeek, GPT-OSS, and Qwen are already flat over the next 70.

Whether plugging in a retriever would produce better papers is untested. The answerer holds the full text, and yes-or-no questions are almost free. Real experiments are not. What the controls show is narrower: given the same feedback, the scarce step is producing the candidate.

Limitations

The test bed is 200 ICML 2026 method papers. The game could transfer to drug design or materials science, and that was not tried. Recovery rewards resemblance to the published solution, so a different method that would also work scores low. When a solution has several equally fair wordings, the judges err more often. Three judges overlap the inquisitor pool. Dropping the same-family judge moves the gain from naming the idea by at most 0.08. The bias is small and not fully ruled out. GPT-6 Astra labels each question's parent in the trees. Humans checked the judges and the answers, not the branch structure at the same density, so the branch counts can carry annotation error.

Extraction is right on 93% of the audited papers, and a dropped distinguishing component can inflate recovery. Newer models were left out over contamination risk. The answerer is Gemma on every game, so one misread paper hits every inquisitor. Yes/No flips are 1.1%, small and not zero. The flat stretch after question 30 could also be questions that repeat bits the model already has. The trace shapes support a search-pattern account, and information gain per question is not measured.

Terms

Source

What people are saying

Related papers

All paper explainers