ORCA-bench: How Ready Are Language Model Agents for Oncall?
Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, Raaz Dwivedi
cs.CL, cs.AI, cs.SE
2026-07-31
Frontier coding agents do oncall root-cause analysis on a live microservice system. Best accuracy on realistic incidents is 25%; the weakest model fabricates a root cause 40% of the time.
LLMs are now genuinely good at writing, patching, and searching code, and SWE-bench-style numbers keep climbing. Oncall is a different job. A pager fires at 3am with a user report that "checkout is broken," and an engineer has to reason over noisy metrics, logs, and traces, read source code, and work out which microservice failed and why. That diagnosis is root cause analysis (RCA), and its evidence is a live, changing system rather than a frozen repo or a failing test. The starting point is an ambiguous user report, often hours after the fault began.
Earlier benchmarks for LLM agents on RCA either injected a single fault and asked the model to name the cause, or exposed telemetry without source code. ORCA-bench tries to get closer to real oncall: hand the agent the full stack of metrics, logs, and traces plus complete source code, on a system under live load, and grade it against reports of increasing vagueness.
The substrate is Astronomy Shop, an OpenTelemetry-instrumented microservice demo: 19 services in 13 languages (Go, Java, Python, Node.js, C#), run for six days under simulated traffic to accumulate 50 GB of metrics, logs, and traces. The agent does not open a single file. Through the Terminus-2 agent harness it gets an interactive tmux terminal plus the three artifacts an oncall engineer actually reaches for: Prometheus for metrics, Jaeger for traces, and OpenSearch via Grafana for logs, with full source-code access in the terminal. Context is compacted automatically when it fills up.
There are 1,079 tasks (884 real incidents plus 195 no-fault controls), and faults come from toggling real feature flags in the system rather than a synthetic fault library. Three axes stretch the difficulty: report specificity (Easy gives a concrete error string, Medium names a broken feature, Hard only says "the site has issues"); time-to-detection from 15 minutes to 24 hours, so evidence gets progressively buried; and co-occurring fault structure across five types (isolated, independent, conflicting, cascading, sequential), forcing the agent to tell the real culprit apart from noise.
Scoring is per-fault on a 0-3 rubric, tracking both RCA Depth (partial credit for progress) and RCA Accuracy (every root cause must be named). Correctness is judged by an LLM (GPT-5.4), calibrated against independent human re-scoring of 40 tasks for a weighted Cohen's kappa of 0.90.
The verdict is uniform: even the strongest model is nowhere near replacing an SRE under realistic oncall conditions.
| Setting | Metric | Best score |
| Medium (realistic reports) | RCA Accuracy | 25.3% |
| Hard ("site has issues") | RCA Accuracy | 10.0% (Opus 4.7) |
| All 884 incidents (partial credit) | RCA Depth | 48.8% (GPT-5.5) |
| Easiest tier (concrete error) | RCA Accuracy | 30.6% (Sonnet 4.6) |
The five agents tested are Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, GLM-5, and DeepSeek-V4-Pro. Strip source-code access and RCA Accuracy drops 9 to 16 percentage points across every model, while hallucination rises; without code to check against, the models invent more. The weakest, GLM-5, fabricates an implausible root cause in 40% of incident reports, and even the steadiest, DeepSeek-V4-Pro, does so in 7%. An underappreciated friction: 26% to 40% of the telemetry calls the agents make either error out or return empty, so they must keep reasoning through a stream of nulls.
On a 32-task human-verified subset, Claude Fable 5 lifts RCA Depth to 58.2% and Accuracy to 40.6%, beating GPT-5.5 (21.9%) and Opus 4.7 (25.0%) on that subset. The authors stress that even so, the gap on Medium and Hard does not close.
The signal for practitioners is direct: do not extrapolate from SWE-bench scores to production reliability. Fixing a bug in a static repo with a test as ground truth and inferring a root cause from a vague ticket on a live system are different skills. In the near term, treating these agents as an oncall copilot that pulls logs, fetches metrics, and drafts hypotheses is far more realistic than treating them as autonomous SREs, and you have to watch whether they confabulate a cause with a straight face.
The more durable point is the authors' lower-bound argument. A curated 50 GB system whose code and instrumentation are all public is already a best case tailored to the models. Real production systems are orders of magnitude larger, change daily, and are full of private idiom the model has never seen. So 25% is a floor, not a ceiling; the real gap is wider.
The authors lay out the limits plainly. First, system priors are baked into the models: Astronomy Shop and OpenTelemetry are public and likely seen in pretraining, whereas real systems are private and idiosyncratic. Second, each task is investigated cold, with no accumulated memory and none of the months of system intuition a human SRE builds. Third, the setup is read-only; the agent can investigate but not patch, so it misses the strongest feedback signal of all, a symptom that resolves after a fix. Fourth, only one prompt template and one agent harness were used.
Two further concerns. All 1,079 tasks come from toggling flags in a single system, so fault types are bounded by that demo's design and generalization to other stacks is untested. And an LLM judging LLMs, even at kappa 0.90, may be systematically lenient on confabulation, which is itself a model strength.