LLMs stop asking too early: correct answers can hide poor information gathering
rohanpaul_ai · x · 2026-09-13
Rohan Paul summarizes a new paper (arXiv:2608.14808, "Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking"): LLMs frequently underestimate how much information they need and stop early, so evaluation should test when they stop rather than rely on answer accuracy alone. A correct answer can hide poor information gathering — the model may use prior knowledge or guess correctly without collecting enough evidence.
More from Research
- Stanford/MIT paper: same LLM shows up to 6x gap depending on the harness — rohanpaul_ai · 2026-09-13
- Stanford's CS 312 'Deep Learning Alchemy' by Hashimoto & Kotha to release all materials publicly — anshulkundaje · 2026-09-13
- SGLang Team Open-Sources Miles v0.1: 744B Model RL on 64 GPUs at 263s/Step — aigclink · 2026-09-13
- Dev reverse-engineers HL2 zombie mod into C++/WASM for parallel deep RL training — shakoistsLog · 2026-09-13
- Dev touts char-level models: no retokenization bugs, dead-simple string munging — cephaloform · 2026-09-13
- SEED-UMI: human-robot shared exoskeleton for one-to-one dexterous demonstration collection — jeasinema · 2026-09-13