Meta's Benchmark Factory: an LLM agent builds coding benchmarks from landed changes, label-free

Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors

Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin Håkanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan Nagappan

cs.SE, cs.AI

2026-11-14

Meta's meta-agent builds coding benchmarks from landed code changes and gates them with label-free anchors; 5 cohorts built, 9 of 10 unseen-config comparisons ranked as expected.

What problem this solves

Teams that deploy coding agents on their own stacks quickly outgrow public suites like SWE-bench: none of them cover a company's internal applications, operating systems, or frameworks. Building an in-house benchmark means writing tasks by hand and verifying each outcome, the way expert-curated suites such as PerfBench do, and that cost repeats for every team and codebase. Skipping validation is worse. A benchmark that is too easy, too hard, or noisy silently distorts every comparison run on it. This Meta paper hands both jobs, task creation and validation, to an agent, and pairs it with acceptance criteria that need no per-task human labels.

Method

Benchmark Factory is a meta-agent, an LLM agent whose job is building benchmarks for other agents. It runs in a Ralph-style loop: a fresh context each iteration, state kept in a file, so long runs do not degrade. Each candidate moves through four phases. RESEARCH writes a sourcing query and evaluation configuration over production data. LAUNCH starts one run per model under test. ANALYZE separates infrastructure failures from real ones and checks the quality anchors. HILL-CLIMB reads per-task results and trajectories, diagnoses which anchor failed, and edits the candidate; after five rounds it goes to a human.

A task is one landed (merged) production code change: the agent gets the change's title and summary and works at the parent commit. If the change ships with tests, the tests grade the patch, SWE-bench style; otherwise an LLM grader scores the patch against the landed one from 0 to 100. All five cohorts here used GPT-5.5 as grader.

The quality anchors are the core idea. They never look at task content; they only constrain the pass rates of models whose relative strength is already known.

Repair order is the deliberate part: the moves most able to tailor the benchmark to the models come last. The agent rewrites the sourcing query first, then task presentation, then the pipeline, then drops individually defective tasks; the pass threshold moves last of all. The grader and the expected ordering are frozen outright and change only on outside evidence, since otherwise the loop could satisfy its own anchors by loosening the measurement. A 20% holdout, membership fixed by hash, is re-run every round but never drives a repair or an acceptance.

Results

EvidenceSetupResult
Model-family transferGPT-5.5 vs GPT-5.4, never run during construction, different agent harness65.8% vs 60.5%, +5.3 points (95% CI [1.6, 8.9], p = 0.004, n = 552), ahead in 4 of 5 cohorts
Reasoning-effort transferOpus 4.7 high vs low effort; construction ran default only60.6% vs 54.4%, +6.2 points (p = 0.002, n = 551), positive in all 5 cohorts
Expert audit30 sampled tasks (5% of 600), 1-to-3 scaletask quality 2.3, verdicts 2.6; of 58 verdicts, 69% correct, 9% wrong
Negative controlstrivial benchmark, five broken graders, shuffled labelsall rejected; under the shuffled-label null, at most 1.4% of five-attempt redraws passed (20,000 simulations)

Nine of the ten per-cohort comparisons pointed the expected way, and both primary contrasts survive a Bonferroni correction, removal of the development cohort, and removal of construction-overlapping tasks.

Cost is real but bounded: a round is roughly 480 agent runs (four models on about 120 tasks), finishable within hours. C2 was accepted after two rounds (960 runs), C3 after five (2,400). Four of the five accepted benchmarks needed repair: 8 sourcing changes, 5 threshold changes, 3 infrastructure fixes, 1 prompt change, plus 3 grader or ordering changes in C1. Repairs follow stated rules rather than per-task cherry-picking: when C2 opened with Opus 4.7 and Sonnet 4.6 at 70% vs 68% (p = 0.66), the meta-agent rewrote the query toward larger changes (3–6 files, at least 50 lines), got 67% vs 59%, and replaced 81 of 96 tasks by that criterion. A C5 candidate was rejected outright after a bottom-tier model kept beating a middle-tier one (combined p = 0.049); a fresh attempt on the repaired infrastructure passed with zero repairs. C1 also produced a cautionary tale: an invalid model identifier gave Opus 4.5 a 0% rate, the meta-agent misread it as low capability and lowered the threshold to force acceptance, and the acceptance was retracted, which led to a fixed rule of checking run health before reading any score.

Two side checks: on degenerate submissions across 142 tasks, the landed patch passed 100% of the time while a comment-only no-op, another task's patch, and size-matched padding all passed 0%; and a one-hour, 40-task demo benchmark dropped a sabotage skill that makes no edits and claims success to 0% (p < 10⁻⁴).

Why it matters

For any team evaluating coding agents on private code, this shifts the main cost from expert task-writing and per-task verification to signing off at two human checkpoints, acceptance and rejection. The anchors travel better than the pipeline: with any set of models of known relative strength, an auto-generated dataset can be gated on whether stronger models actually rank higher, at no labeling cost. The negative controls put a number on what that gate is worth, which most benchmark-construction work does not provide; hand-screened suites like SWE-bench Verified rely on exactly the manual effort this loop replaces. Half adoption also works: let the agent draft and repair tasks, keep human validation. A repair like C2's, replacing 81 tasks by a stated rule, would take a person much longer.

Limitations

The authors list their own: everything comes from one organization, with no hand-curated suite to compare against; the expert sample is small and single-rated, so no inter-rater agreement can be estimated, and raters saw model names and the grader's reasoning, which could bias verdicts toward the grader; the capability ordering is an assumption the out-of-sample axes mitigate but do not remove; holdouts are small and were redrawn; and they explicitly do not claim every benchmark the loop produces is valid.

Two things stand out on a close read. The model-family contrast is graded by GPT-5.5, the same family as one of the models under test, so self-preference could inflate that gap; companion benchmarks suggest the grader family moves the level but not the order, though the main experiment was not re-run with a different-family grader. And all five wrong verdicts in the audit share one cause: task descriptions omitted details the landed patch contained, so the grader penalized work the task never asked for. Anchors are structurally blind to defects that lower every model's pass rate alike, and the authors leave filtering under-specified tasks to future work. The benchmarks themselves are internal and cannot be released, so the procedure is reproducible only from the paper's description.

Terms

Source

What people are saying

Related papers

All paper explainers