Meta's Benchmark Factory auto-builds coding evals from your own codebase
omarsar0 · x · 2026-11-20
A Meta paper introduces Benchmark Factory, a multi-agent meta-loop that turns merged production code changes into coding benchmarks with no per-task human labels. An LLM agent writes sourcing queries, runs a ladder of models with known relative strength as label-free "quality anchors," diagnoses failures, and repairs the benchmark until it passes.
- Anchors were stress-tested: overly easy benchmarks and five kinds of broken graders were rejected; with shuffled model identities, a bad benchmark slipped through at most 1.4% of simulations.
- Benchmarks were built for five developer cohorts at Meta; expert audits scored tasks 2.3 and verdicts 2.6 out of 3.
- Out-of-sample, benchmarks ranked Claude Opus 4.7 high effort above low effort (60.6% vs 54.4%, p=0.002) and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%, p=0.004), with 9 of 10 cohort comparisons pointing the right way.
More from coding & agent
- Genie 4 API goes live: precise camera-control for reframing and video — korymath · 2026-11-20
- GrepMem: Plain-Text Log Plus Regex Beats Summarization for Long-Horizon Agents — dair_ai · 2026-11-20
- Tripo Launches Agentic Canvas: AI Plans and Runs Your Full 3D Workflow — LinusEkenstam · 2026-11-20
- OpenAI to Cut Cursor Model Access on Nov. 12 After SpaceX's $60B Anysphere Acquisition — shashib · 2026-11-20
- Infor and AWS Break Down Their Enterprise Agent Stack, With Claude Sonnet 4.6 as Supervisor — shashib · 2026-11-20
- Treg Launches /REA-marketing Skill to Reverse-Engineer Any Brand's Playbook Week by Week — cneuralnetwork · 2026-11-20