Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking
Yepeng Huang, Jiawen Zhang, Michelle Dai, Xiaorui Su, Shanghua Gao, Zi Wang, Marinka Zitnik
cs.AI, cs.CL, cs.LG
2026-08-15
MT-InfoSeek formalizes multi-turn information seeking as k-underspecified CSPs (5,251 problems, 9,006 instances): models know to ask but under-predict missing information ~4x as often, and no model exceeds 0.4 exact-set accuracy at k=2.
Ask an assistant to "book my Wednesday flight to Beijing" and a capable one should notice what is missing: departure time, cabin, ID document. It must know that it lacks information, know how much, ask the right questions, and only answer once the answer is determined. Benchmarks overwhelmingly test answering under complete context. Multi-turn information seeking, the skill behind clarifying questions, has no formal evaluation: does a model recognize insufficiency, gauge its degree, pick a minimal set of queries, and stop only when enough?
QuestBench-style datasets hide a single variable, which modern models fill in trivially. This paper formalizes the task as a k-underspecified constraint satisfaction problem: the target answer Y depends on a set of variables, the observed context fixes all but k of them, and acquiring those k values is what makes Y unique. For the first time, what to ask and how many times has an exactly computable ground truth.
MT-InfoSeek contains 5,251 problems and 9,006 task instances across five domains. In four structured domains the queryable variables are predefined and the model selects one per turn; problems are built by masking variables from fully specified states and verifying the masked set is a minimal sufficient set (k-MSS), meaning no smaller subset determines the answer:
Evaluation runs along three axes. When: predict the degree k, scoring over- and underestimation. What: identify the k-MSS, scored by exact-set accuracy and Jaccard similarity. Whether it helped: sequential task solving, one query per turn. The central design is splitting final accuracy (is the answer right) from final sufficiency (does the acquired information uniquely determine the answer). Sufficiency is judged before answer generation, separating the inability to seek information from the inability to compute.
Performance degrades with k across all models and domains.
| Observation | Number |
| Misclassifying underspecified problems as fully specified | only 0.6±0.8% |
| k-prediction accuracy | below 0.5 for k≥2 across models |
| Under- vs over-predicting degree, logic at k=2 | under-prediction 4x as frequent |
| Exact k-MSS identification without k given | no model above 0.4 at k=2, 0.2 at k=3 |
| Given the true k | only marginal improvement |
Models know they should ask, systematically underestimate how much, fail to pick a minimal sufficient set, and often stop and answer before the information determines the answer. In tasks with ordered dependencies, an incorrect query order reduces final accuracy even when all necessary information is eventually acquired. On 20Q, questions with higher information gain match blinded human preference in 76.5% of 200 pairs (86.5% at high annotator confidence), yet models of similar informativeness differ substantially in final accuracy: asking well and answering well are distinct skills.
For anyone building agents or conversational systems, this is a ready-to-use diagnostic frame. Clarification behavior in products is currently prompt-engineered; the measured underestimation bias explains why it stays brittle. Final sufficiency is the clever part of the design: it scores information acquisition separately from answer generation, a dimension that accuracy-centric leaderboards cannot see. Five domains spanning math, logic, biology, medicine, and open-ended QA make the failure pattern hard to dismiss as an artifact of one synthetic setting.
Stated by the authors: the user model is idealized, while real users give ambiguous, noisy, or inconsistent answers; the formulation covers missing variable values, not missing rules or constraints; every query is treated as equally expensive, though asking a patient for history and ordering an invasive test differ vastly in cost and risk. The oracle in ClinGuide-MT and 20Q is an LLM; consistency checks suggest it is not the failure driver, but a human-audited protocol remains future work. And the original GSME extension was saturated by the models themselves, a reminder that difficulty calibration in synthetic data is fragile.