IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
cs.CL
2026-09-10
Yale's IdeaAmbig (660 gaps): best model 9.6% real defect recovery vs 80.6% clarification given the defect; gold resolution lifts ready rate 14% to 98%.
Research agents now couple ideation to coding. The method write-up in the middle is rarely checked. An idea can be novel, coherent, and scientifically plausible while the core mechanism still omits a choice that two competent implementers would fill in differently. Existing evals score idea quality or paper–code consistency and assume the method was specified. The gap becomes a silent wrong implementation, and a failed reproduction looks like an execution bug.
Yale frames this as codification readiness: whether an implementation-facing method write-up gives a competent implementer or coding agent enough to build the intended prototype without unsupported assumptions about the core method. Seeds, hardware paths, and conventional batch sizes need not be pinned. A blocking defect is an omission, ambiguity, or inconsistency that leaves a method-defining decision open.
IdeaAmbig is 660 evidence-grounded, single-defect instances. 163 are real gaps from ML Reproducibility Challenge, ECIR, and TMLR reports plus GitHub issues on paper-linked repos, each with a source-supported resolution. 497 inject exactly one implementation-critical change into a codification-ready reference taken from a successful reproduction. Level-1 labels are ambiguity, incompleteness, and inconsistency, with ten Level-2 types. Real gaps skew toward ambiguous procedures; synthetic gaps skew toward omitted procedures.
Three tasks are scored separately:
Thirteen models are evaluated. Semantic judging uses Claude Opus 4.8, calibrated against human labels.
Best model, GPT-5.6-Sol, on the real subset: 67.5 Macro-F1 on readiness, 9.6% Macro DRR on localization, 80.6% Macro-CAS on clarification once the defect is given. The synthetic subset is easier: 86.4 / 12.2 / 96.2. It accepts 31% of underspecified write-ups and rejects 34% of ready ones. Reason-grounding peaks at 0.36 real and 0.49 synthetic.
Localization is the bottleneck. On real data, Level-1 accuracy is 60.1, Level-2 25.2, and Loc-Acc for recovering the actual blocker is 16.0. Dropping taxonomy labels and asking only for the same blocker lifts 50 real items from 10% to 40%: labels add difficulty, finding the decision remains hard. Coarse gaps (a whole component missing) are easier than local ambiguities inside an otherwise specified component. All 13 models share the pattern: clarification given the defect is much easier than finding the defect.
Same model, same real Task 3 items: end-to-end clarification without the defect is 13.6 Macro-CAS; with the defect it is 80.6. In an oracle study, feeding the gold resolution raises the downstream ready rate from 14% to 98%. On 20 synthetic items with reference implementations, passing all preset tests rises from 45% to 85%, and faithful implementation of the target detail from 30% to 90%. Tests can pass while the method choice is still wrong.
Automated research pipelines need a specification gate before code, not another idea scorer. Models already ask useful questions once a gap is pointed out. They rarely pull that method-changing choice out of a write-up that looks complete. Forcing localization, then allowing coding, is a better defense against silent method drift than checking paper against repo after the fact. For anyone using agents to reproduce or extend papers, this is a measurable failure mode, not another generator.
Instances are reconstructed after the fact, not original handoff transcripts, and models never see the downstream repo. Single-defect items are a diagnostic abstraction; a 50-pair multi-defect ablation did not make localization harder, but the sample is small and response counts are unmatched. Clarification is scored as one action, and the oracle utility study supplies the gold answer, so it measures an upper bound, not an end-to-end agent. Coverage leans AI, NLP, and ML. The executable check is 20 items. Task 1 uses a balanced 100/100 ready–not-ready sample, not the natural mix.