AutoResearch: Insight In, Hallucination Out
Yiming Ren, Xiang Liu, Qumeng Sun, Xiao Zhang, Jiahao Li, Haoyang Zhang, Junjie Wang
cs.AI, cs.MA
2026-08-18
AutoResearch grounds ideas before experiments and claims before acceptance. On RSICD, mR rises from 32.84 to 34.69 with 5 audit issues versus 11-27 for other systems.
Autonomous research systems can now chain idea, code, experiment, and write-up into a long workflow. Automating more of that chain does not make the process scientifically grounded. Many systems start from a user-supplied idea or a fixed experimental setup. On the execution side, a bug, a bad measurement, or an over-read can still be wrapped into a coherent-looking claim. EvoMap and Tsinghua call that system-level hallucination.
Two grounding checks are missing. Discovery has to say why a direction is worth testing. Execution has to say why a conclusion deserves to be accepted. AutoResearch wires those checks into one process: pin the insight first, then keep unsupported claims from becoming results.
The system has two stages: Idea Generation and Idea Execution.
Discovery keeps two contexts in play: new external signals (papers, repositories, technical media, plus fast sources such as X and Xiaohongshu) and a curated domain knowledge base. Signals pass source-quality priors, deduplication, and model screening. A useful seed is a mechanism that can be lifted out of its original setting, not a hot topic or a reported gain. Several frontier models propose cross-domain hypotheses independently, and a generator may reject a pairing when no real transfer exists. The current setup uses three generators and three reviewers; an idea needs at least two positive reviews, then freshness and domain-consistency checks, and finally becomes a hypothesis plus an executable plan.
Execution turns the plan into a task graph with dependencies and dispatches ready tasks in parallel. Implementation and critical review are split. The reviewer gets a fresh context: hypothesis, plan, artifacts, and criteria, without the producer's reasoning trace, and returns PASS, PARTIAL, or FAIL. Failure triggers diagnose, revise, or rerun, instead of being narrated away. A claim is accepted only if its evidence set passes independent verification. State is resumable. The dashboard is read-only and does not vote. Given the evidence, the system may continue, revise, scale, or stop, and a negative result can be the recorded outcome.
Three settings: open-ended method exploration, systems work with hard contracts, and benchmark-driven machine learning.
Cross-modal retrieval uses RSICD and mean Recall (mR) in both directions. Idea Forge proposes a staged method: stronger global image-text alignment, text-guided local aggregation, and explicit entity-position links. Added one stage at a time under a fixed protocol, mR moves from 32.84 to 33.89, 34.04, and 34.69, a total of +1.85. Audit-confirmed issue events: 5, against 11 for R&D-Agent, 15 for AutoResearchClaw, 18 for Agent Laboratory, and 27 for The AI Scientist.
The matrix-multiply contract is stricter: 1024×1024 FP32 in 200 ms, relative error under 10⁻⁵, coefficient of variation under 20% across ten wall-clock trials. Speed and numerics passed; stability did not. The system refused the fast pilot, then found a timing bug: CPU time under multithreaded BLAS had been treated as wall-clock time. After a corrected rerun, the baseline is 3.4 ms, about 626 GFLOPS, roughly 58 times inside the 200 ms bound. Issue events: 4, against 5 to 8 for the others.
Three Kaggle tasks test stop-or-go. Titanic five-fold accuracy goes from 0.822 to 0.843, above the 0.830 target, so the run scales up. House Prices RMSLE falls from 0.2008 to 0.1251, short of 0.120, so revision continues. Disaster Tweets F1 goes from 0.763 to 0.805, well under 0.835, with thinning gains, so the run stops and keeps the negative result.
On one server with dual Xeon Platinum 8563C CPUs, eight L20 GPUs, and 944 GiB of memory, a week of continuous operation produced about 2584 candidate ideas, about 355 queued experiments, about 22 executed runs, and about 14 empirically supported ideas.
The +1.85 mR is not the reason to care. The engineering split is: no mechanism, no project; no evidence, no conclusion; reviewers do not read the producer's chain of thought. For anyone building a research agent, that is more useful than another automatic paper writer. A negative result is a valid ending. Running forever is not the default.
The retrieval gain is small, and the Kaggle tasks are close to teaching problems. The studies show that the loop can close. They do not show autonomous discovery at scientific scale. Whether the baseline systems were tuned equally is not reported. Issue events use an in-house audit definition. The authors list coverage of external signals, knowledge-base quality, and the need for explicit experimental criteria as bottlenecks. 2584 ideas down to about 14 validated ones is a thin conversion, and there is no human-scientist baseline. Feeding verified evidence back into the knowledge state is listed as future work, not a result in this paper.