Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models
Wei Liu, Peijie Yu, Michele Orini, Yali Du, Yulan He
ICML 2026
cs.AI, cs.CL, cs.DB, cs.LG
2026-02-02
DDR-Bench gives models a database plus SQL/Python and no questions. Claude 4.5 Sonnet leads at 47.7%; long-horizon exploration is still unsaturated.
Most agent benchmarks start with a question already written. The model is scored on whether it carries that question out: search, SQL, a report. The authors call this executional intelligence. A working analyst more often sits down with a database and no brief. Goals, hypotheses, and stopping time are the job. That capacity is investigatory intelligence.
Web deep-research suites mostly reduce tools to search queries and grade reports with rubrics or faithfulness to reference pages. LLM-for-data-science papers often bury the research targets in the prompt, and many stay under a few dozen interaction steps. DDR (Deep Data Research) removes both crutches: a structured database, SQL and Python, no question, no turn cap. The model has to explore, write insights, and stop on its own.
The start prompt is a handle, not a task, for example "Start analysing the user with userid=2048." The scaffold is a thin ReAct loop over MCP-exposed SQL and Python. No planner, no memory module, full history each turn. When the model stops it emits two insight types: a paragraph after every round (message-wise) and a global report over the whole trace (trajectory-wise).
DDR-Bench has three real databases, 291 entities, and 2,058 checklist facts:
Facts are mined from the unstructured side with GPT-5 mini, then the same model checks whether the agent's insights support each fact. GLOBEM uses closed-form questions. More than fifty domain experts screened that every fact is reachable from some queryable subset. Column names are rewritten; the agent never sees the checklist while exploring.
Averaged across sample/item splits and both insight types, only Claude 4.5 Sonnet exceeds 40%, at 47.73%. DeepSeek-V3.2 (38.80%) and GLM-4.6 (37.52%) sit next to GPT-5.2 (37.09%). Llama 3.3-70B is at 12.30%. The task is not close to saturated.
| Model | Overall avg. accuracy |
| Claude 4.5 Sonnet | 47.73% |
| DeepSeek-V3.2 | 38.80% |
| GLM-4.6 | 37.52% |
| GPT-5.2 | 37.09% |
| Llama 3.3-70B | 12.30% |
Claude's 10-K message-wise sample accuracy is 77.61%; MIMIC is 36.07%. Message-wise and trajectory-wise scores often diverge: a model can comment well turn by turn and still drop key points in the final write-up.
When each checklist item is rewritten as an explicit query, Qwen3-Next-80B-A3B rises from 32.59% to 43.21% on average, and 10-K jumps from 45.58% to 70.55%. The underlying work is mostly solvable once the goal is named. Finding the goal is the hard part. GLOBEM is the exception: reactive scoring falls to 31.95% from 35.40% when the model explores on its own.
Inside the Qwen family, a 10x parameter jump adds less than 3% final accuracy. Longer context windows do not reliably add turns or score. Qwen3-4B and Qwen3-Next-3B, with fewer active parameters, explore more and hit a higher ceiling. Extra reasoning budget cuts turns but accuracy swings; a long-short memory note makes 10-K and MIMIC trajectory scores worse.
Of 206 hand-labeled failed checklist items, 58% come from exploration that is too narrow or too shallow. Stronger models over-reason; weaker ones loop on debugging. Across 1,850 sampled insights, hallucination rates for most models stay under 5% and do not track accuracy. The checker is stable (coefficient of variation below 5%) with macro F1 around 90% versus humans.
If you ship data agents, this split between finishing a question and picking a question is the useful cut. Extra scaffolding, size, and context are not the main drivers in this minimal setup. Stronger models keep a stable implicit plan: coverage is neither tiny nor a uniform spray. Test-time scaling is sigmoid; a few late, high-information queries do the lifting, and most tokens are database returns.
The training implication is blunt. Agentic pre-training and post-training beat stacking ReAct modules. Checklist coverage is incomplete by construction, but pairwise usefulness of leftover novel insights ranks models in the same order as checklist accuracy. Strong models are not gaming the list by writing around it.
The checklist is extracted and checked by GPT-5 mini. That is cleaner than a free rubric, and still model-on-model. Residual errors include an overly strict checker (insights that omit numbers get dinged) and rare cases where the checker reasons correctly then votes wrong. Novelty can only be ranked pairwise.
The scaffold is intentionally thin, so "intrinsic strategy beats plugins" is a claim inside that thin setup. Stronger memory, retrieval, or multi-agent stacks are not compared at scale. All three databases mix structure with text that can be turned into facts; a live warehouse behind access control is a different object. On MIMIC, models invent extra drugs after SQL LIMIT truncates a list. That pattern does not change the score and would be unsafe in clinic.