Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
Yining She, Lei Lin
cs.AI, cs.SE
2026-09-18
On 574 production-agent runs, 2PL adaptive testing recovers the full pass rate within 1.03 pp using 200 of 519 items. Meta shipped simpler difficulty-stratified subsets.
Every time a production LLM agent changes its model, prompt, or tools, the benchmark has to be rerun. The system here is an analytics agent at Meta, serving tens of thousands of monthly active users. Its main suite has 519 human-written data questions and takes about three hours per full pass. Development and monitoring already generate tens of thousands of runs. Full reruns become a queue.
Subset selection, IRT reconstruction, and adaptive testing already exist for public LLM benchmarks. Those methods are usually calibrated on a panel of distinct models, each answering once. This paper asks a different question: when one production agent evolves along a timeline, can earlier pass/fail records cut later work while still recovering the full pass rate and the ranking among runs.
The data are 574 real runs over 52 days, split in time: 287 calibration runs from days 1-28, 287 held-out runs from days 29-52. Questions were written by internal analysts, contain no user data, and are graded automatically as pass or fail. Runs with valid outcomes on at least 80% of the suite are kept; the reference score is the pass rate over valid items.
Four partial-evaluation methods share that history.
Random sampling draws k items uniformly and reports the subset pass rate. Historical caching looks back seven days: items whose recent pass rate is near 0 or 1 reuse the majority verdict, the rest are executed; the stability cutoff ε is swept from 0 to 0.30. Fixed subsets pick k items once from calibration and reuse them every later run. Selection compares difficulty stratification (Rasch difficulties, one item near each stratum median), K-means on historical response vectors, and K-means on 2PL item parameters. Scores can be stratum-weighted or gp-IRT, which blends the weighted subset with IRT predictions on the unrun items. Adaptive testing picks the next most informative item inside the current run. Rasch uses Fisher information; multidimensional 2PL uses D-optimal selection so later items cover ability axes the early items have not pinned down. Because the selected items are informative rather than representative, the subset pass rate is not the estimate; p-IRT fills unobserved items with predicted pass probabilities.
On 287 held-out runs, multidimensional 2PL adaptive testing wins on score fidelity. At 200 items (38.5% of 519) pass-rate MAE is 1.03 pp, against 1.79 for random sampling, 1.40 for difficulty-stratified gp-IRT, and 1.37 for Rasch adaptive. At 300 items, 2PL Spearman correlation with the full ranking is 0.992. At small budgets the fixed subset is more reliable among non-adaptive methods: at 100 items, difficulty-stratified gp-IRT is 2.65 pp, random 2.94, Rasch adaptive 2.92; 2PL adaptive is already 1.97.
| Method | k=100 MAE | k=200 MAE | k=300 Spearman |
| Random sampling | 2.94 pp | 1.79 pp | 0.975 |
| Difficulty-stratified gp-IRT | 2.65 pp | 1.40 pp | 0.981 |
| Rasch adaptive | 2.92 pp | 1.37 pp | 0.986 |
| Multidimensional 2PL adaptive | 1.97 pp | 1.03 pp | 0.992 |
Caching saves executions but the score drifts. Mean execution fraction ranges from 20.3% to 81.6%, MAE from 7.36 pp down to 0.81. At 69.1% execution, MAE is still 1.54 pp; nearby at 67.4%, 2PL adaptive is 0.46 and fixed-subset gp-IRT is 0.73. Rankings hold up better than the MAE suggests: Spearman is 0.904 even at 20.3% execution.
Clustering selectors fall below random sampling once the budget is moderate. TinyBenchmarks found IRT clustering useful; here it does not. Recurring runs of one agent likely produce fewer distinct response patterns than a panel of models, so multidimensional item parameters are noisier. The sampling fraction is also much larger (100 of 519 versus 100 of about 14,000), so uniform sampling is already accurate.
Production still shipped difficulty-stratified fixed subsets at k in {100, 200, 300, 400}. The subset is chosen once, the same items run every time, execution can stay parallel, and question-level diffs remain available for debugging. That subset, with no recalibration, was applied to 299 runs from five other agent families. At k=200, gp-IRT MAE is 1.47 pp versus 1.40 on the original agent, and it beats random sampling at 11 of 12 budgets. Shrink the calibration window to one day (14 runs) and k=200 gp-IRT is 1.41 pp, against 1.40 for the full four weeks, ranging 1.35-1.59. Short windows work here, consistent with a relatively mature agent whose difficulty order does not move much day to day.
This is a production playbook for cheaper recurring agent eval, not another public mini-benchmark. 2PL adaptive testing is the most accurate, and it fights a parallel eval harness because it has to pick items sequentially and update an ability vector per run. Fixed subsets give up a fraction of a point and can go live the same week. Transfer without recalibration, plus a one-day calibration window, suggests that item difficulty is more portable than how this agent answered last week.
A workable recipe: stratify by Rasch difficulty, offer a few budget knobs, recalibrate after material model or tool changes, and periodically run the full suite to watch the subset error.
All data come from one organization and one production benchmark. Cross-family transfer is shown only for fixed subsets; caching and adaptive testing were not transferred. MAE plus Spearman and Kendall measure score closeness and ranking agreement, not missed regressions, false alarms, or agreement with a release gate. The questions, the outcome matrix, and the code are not released; reproduction has to follow the Section 3 spec on another item bank. The agent looks relatively mature, which is likely why short calibration windows hold. A system still churning its tools may not look the same.