Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, Shumin Deng
cs.CL, cs.AI, cs.IR, cs.LG, cs.MA, cs.SE
2026-09-01
AutoSciRub induces an evidence-grounded rubric before a research agent runs, gaining 2-3 points on ResearchClawBench and 16.8 on a 20-task AstaBench subset.
Research agents can already chain literature, experiments, code, and a report. The instruction usually names only a high-level goal. Which analyses to run, which methods count as appropriate, and what evidence would justify a claim stay implicit. The report can look finished while missing a required comparison, using the wrong procedure, or asserting more than the artifacts support.
Rubrics in current scientific-agent benchmarks are mostly post-hoc scorers, and expert-written ones do not scale to new tasks. Automatic rubric generators drift in this setting: without literature conventions and a look at the files on disk, the criteria become infeasible or scientifically unfounded. The gap is concrete. A rubric should be an execution-time specification written before the agent starts, then used to steer experiments, check each criterion, and revise the missing pieces.
AutoSciRub has two stages. Automatic rubric induction first splits the instruction into atomic scientific goals. That skeleton says what the task must address; it does not name methods or expected numbers. Each goal is then grounded in papers and web sources for standard methods, metrics, controls, and robustness checks, with queries that avoid task IDs and a hidden target-paper blocklist. A light scan of task-visible data records which of those analyses the files, fields, and labels can actually support. Synthesis turns the three inputs into executable criteria: data sources, experiments, metrics, comparisons, artifacts, and a pass condition.
The backbone research agent then runs against that rubric. It produces an initial report and artifacts, a verifier scores each criterion, and failed items become targeted revision notes: add an experiment, fix a figure, or drop an unsupported claim. The loop stops when every criterion passes or the revision budget is spent. The layer is a plugin. Backbone models and tools stay the same.
On all 40 ResearchClawBench tasks across ten domains, with Codex fixed, GPT-5.4, GLM-5.2, and MiniMax-M3 move from 18.66, 20.86, and 19.05 to 21.04, 22.73, and 21.04, a 2.08-point mean gain. With DeepSeek-V4-Flash fixed, Claude Code, OpenClaw, and OpenScience gain 2.14, 3.11, and 3.60. Forty-nine of sixty domain pairs improve. Absolute scores stay low: the benchmark treats 50 as roughly target-paper rediscovery, and the best configuration here is 22.73.
On a 20-task Easy subset of AstaBench End-to-End Discovery, three harnesses gain 16.8 points on average. Claude Code and Codex rise from 18/20 to 20/20 completed tasks. A stage-wise ablation on OpenClaw plus DeepSeek-V4-Flash gives +0.36 from the skeleton, 18.31 (+1.06) after grounding, and 20.36 (+3.11) after revision. Rubric-free self-refinement reaches only 19.08 after three rounds and dips in the middle; rubric-guided revision climbs monotonically, about 2.7 times the cumulative gain. Thirty-five of forty tasks pass the verifier within three rounds.
Rubric quality itself moves: mean scores across four dimensions go from 2.20 to 3.84, with specificity 1.65 to 4.40. Scientific-core coverage falls slightly, 3.35 to 3.07. The method is better at making a chosen direction executable than at correcting a wrong framing.
For people building research agents, this is a control loop you can drop in: write what "done" means, then let the model run. Gains repeat across backbones and harnesses, which points at specification as the bottleneck more often than the next larger model. The larger AstaBench jump suggests that cost of missing specs grows as tasks look more like open discovery.
The gains are incremental, and ResearchClawBench remains far from rediscovering the hidden target papers. Treat this as an execution-time acceptance layer on existing agents, not a production scientist.
There is no dedicated limitations section, but the body already shows the weak spots. The dip in scientific-core coverage means grounding can elaborate a secondary analysis without fixing a bad skeleton. Domain scores are not uniformly up; information science, materials, and math drop in some configs. Absolute ResearchClawBench scores sit near 20 against a 50 rediscovery mark. AstaBench uses 20 Easy tasks sampled once. Scoring is LLM-based (GPT-5.1 on reports, MiniMax-M3 on AstaBench), so agreement between induced rubrics and hidden expert checklists still goes through a model judge. Revision adds extra turns; wall-clock and dollar cost are not reported.