READY or Not: Reliable Enterprise Agent Deployment
Veronica Chatrath, Bryan Zhu, Jingxuan Fan, George Pu, Soham Dinesh Tiwari, Soham Dan, Ryan Young, Yuan, Li, Yuang Yao, Apaar Shanker, Minglai Yang, Daniel Yue Zhang, Yunzhong He, Ying Liu, Chenguang Wang, Zhijun Yin, Yuan Xue
Christy
cs.AI
2026-09-02
Scale AI's READY treats agent deployment as qualification: on 750 clinical-audit cases, systems 0.3 points apart in accuracy need 29.6% vs 39.2% review to hit 76% reliability.
Enterprises do not ship an agent because it sits high on a capability leaderboard. An 80% autonomous score means the system is right on four cases out of five. Deployability turns on whether the remaining fifth can be caught, how much human time that catch costs, and whether the human-AI loop can hold a stated reliability target.
Professional-work benchmarks such as Agents' Last Exam, GDPval, and τ-bench score whether the work product is good and, in τ-bench's pass^k, whether repeats stay consistent. Cascades and routers split traffic among models. Selective prediction and learning-to-defer decide when to abstain or hand a case to a person. None of those stack the same three requirements: a pre-specified reliability floor, the cheapest oversight policy that clears it, and a statistical claim on held-out cases.
Scale AI's READY (Reliable Enterprise Agent Deployment) writes that as constrained optimization. Among candidate oversight policies, pick the lowest-cost one that meets target reliability Y, freeze it, then qualify it on a held-out split. The artifact is a deployment profile: reliability, review burden, operating cost, and the assumptions under which the claim holds.
Each workflow keeps its own success predicate. Correctness, process, evidence grounding, and policy checks can all go in. READY only standardizes the qualification procedure: run the agent, select a policy on development data, freeze it, and test it on held-out cases.
The running example is retrospective clinical audit from CliniCARE-Bench. The agent sees a longitudinal record and an audit question, such as whether an ICU patient developed acute kidney injury within 48 hours of a CT scan. It has to retrieve evidence, apply the KDIGO creatinine criteria, and return one of four verdicts (Yes, No, Lack of Data, Medically Ambiguous) plus a stated confidence. A correct indeterminate is a success. It is not an automatic escalation.
The policy class is terminal accept-or-escalate: accept if confidence is at least τ, else send to review. Routing happens after the agent finishes, so many thresholds can be replayed on the same saved traces. Cost is modeled as model spend plus review rate times a human unit cost. With constant unit costs, minimizing cost is minimizing review rate.
Human-review success ah is a deployment assumption here, set to 0.90 in the main analysis. Deployed reliability is a convex combination of autonomous and reviewed outcomes, so it cannot exceed ah. Policies are selected on the development half against Y+δ with δ fixed at 0.05, so a point estimate that barely clears the line does not get to claim qualification. Qualification uses a one-sided 95% lower bound; the accepted-case piece is an exact Clopper-Pearson interval. Clearing the target in the point estimate is not enough.
Sixteen systems, 750 cases split 375/375, 12,000 runs. Label mix is unbalanced: Yes 46.9%, No 33.1%, Lack of Data 16.4%, Medically Ambiguous 3.6%.
Base accuracy a0 and confidence ranking (AUROC) are essentially uncorrelated (Pearson ρ = -0.12). Qwen-3.7-Plus is weak on accuracy (66.4%) and strongest on AUROC (0.711). AURC tracks accuracy tightly (ρ = -0.80). Gemini models report 100% confidence on up to 73% of cases, so the reachable operating grid is coarse; Gemini-3.1-Pro has only five distinct confidence values.
Held-out qualification at Y = 0.76 and ah = 0.90:
| System | Autonomous acc. | Review% | Reliability | 95% LCB | Qualified |
| GPT-5.5 | 75.7% | 21.3% | 0.843 | 0.811 | yes |
| Opus 5 | 75.7% | 22.4% | 0.839 | 0.807 | yes |
| Sonnet 5 | 72.5% | 29.6% | 0.856 | 0.826 | yes |
| GPT-5.4 | 72.8% | 39.2% | 0.828 | 0.797 | yes |
| GLM-5.2 | 75.7% | 10.9% | 0.762 | 0.725 | no |
| Gemini-3.1-Pro | 70.9% | 100% | 0.900 | 0.900 | review-all only |
GPT-5.4 and Sonnet 5 differ by 0.3 points in autonomous accuracy and by 9.6 points in review rate at the same target. Sonnet 5 reviews about one third fewer cases. AUROC is 0.601 versus 0.681.
GLM-5.2 ties for the best autonomous accuracy and posts the lowest review rate (10.9%). Its point estimate is 0.762, two thousandths above the target, but about 89% of its reliability sits on the measured autonomous mass, so the interval is the widest in the table and the lower bound is 0.725. Raising ah to 1 does not rescue it. Qualifying it would need δ at least 0.075, which lifts review to 32.0%.
Gemini-3.1-Pro qualifies only by reviewing 100% of cases. Reliability equals ah. That is not a deployment.
Without the margin, 46% of policies across the frontier sweep pass the held-out lower bound. With δ = 0.05, 95% do. Qwen and MiniMax share 66.4% development accuracy. Qwen has the better signal, yet its nearest reachable threshold overshoots to reliability 0.874 and 61.9% review, against MiniMax at 33.9%. Single-target review rate is a step function of a coarse grid. The fair comparison is the full reliability-oversight frontier.
For a buyer, the ranking question becomes: at the same reliability, who spends less human time. A poorly ranked confidence signal forces extra review even when task scores look tied. Builders should treat routing resolution and ranking quality as first-class metrics, next to accuracy.
READY does not replace capability benchmarks. It consumes their traces and adds a qualification layer. The reference implementation sits on Inspect AI, with workflow definition, execution, scoring, and qualification kept separate. Terminal policies can be replayed; interventions that change the trajectory need policy-in-the-loop execution.
This is a shift in the evaluation question, not a new agent architecture. Incremental as a method, but it turns a leaderboard hunch into an auditable statistical claim.
The study only instantiates terminal accept-or-escalate. Mid-trajectory review, correction, and takeover are formulated, not measured at scale. Review success, latency, and cost are assumed, not observed; every main claim is conditional on ah = 0.90. The paper reports a break-even ahmin, but it never measures whether the human path actually succeeds.
A qualification holds for the sampled case population and the versioned configuration. Distribution shift, tool changes, or a new model checkpoint all require requalification. Y = 0.76 and δ = 0.05 are research choices, not clinical standards. Review rate jumps on a discrete threshold grid, so one-target comparisons are noisy. A meta-analysis of human-AI teams finds they often lose to the stronger of the two components; treating ah as an exogenous 0.90 is likely optimistic.