Raven: The Harness of Harnesses for Composable Agentic Intelligence
EverMind AI
cs.AI, cs.CL, cs.GT, cs.MA, cs.NE
2026-09-27
Open-source Raven runs Claude Code and Codex as composable model-harness units in one typed DAG, beating the best baseline by 10.5 pp Exact Match on its new MAOB benchmark.
Agent systems are moving from isolated, single-domain tasks to long-horizon, cross-domain workflows. The paper's example: develop and operate an FPS game, from requirements research through gameplay programming, art, integration testing, and live operations, each stage with its own tools and completion criteria. An agent's practical capability depends on its model and on its harness, the layer of tool interfaces, context management, skills, and execution and recovery policies wrapped around the model. Two problems follow. As tools and execution conditions multiply, the interacting design choices inside a harness grow too fast to configure by hand across models and domains. And specialized harnesses hard-code assumptions about tools and outputs, so combining agents from different domains means matching subtasks to the right executor and making handoffs compatible. Failure analyses cited in the paper name inter-agent misalignment and inadequate task verification as recurring causes of multi-agent breakdown. Frameworks like AutoGen and MetaGPT put every role inside one shared architecture, with role configuration still done by hand. Raven reframes the problem: stop hand-building a stronger harness per domain; construct specialized harnesses automatically, improve them from experience, and orchestrate them across domains.
Raven is an open-source multi-agent ecosystem whose core abstraction is the executable model-harness pair as a composable unit of intelligence. Four layers:
The theory formalizes capability as reliable task coverage under a shared budget, with typed contracts on nodes and edges. Three results: a compatible plan that finishes within budget delivers a correct record; system success is bounded below by 1 minus the sum of per-operation error terms, with no independence assumption; and under complementary local capabilities, compatible handoffs, and bounded errors, the composed system solves tasks no individual agent can. The illustrative construction is clean: the task is the parity of two hidden bits, agent a1 can read only U and a2 only W, so every individual agent is capped at 1/2, while a three-node plan achieves it with zero error. The design choices target named failure modes: completion adjudication attacks the agent that declares success without producing the artifact, and staged guide loading saves the scarcest resource, context.
Evaluation is split by capability layer: orchestration, four native specialists, harness adaptation, skill reuse. Orchestration uses the paper's own MAOB, which compares a proposed DAG against a reference graph before any worker executes.
| Setting | Metric | Raven | Best baseline |
| MAOB / Qwen3.8-27B | Exact Match | 0.711 | 0.607 |
| MAOB / Qwen3.8-27B | Partial Order Accuracy | 0.950 | 0.780 |
| MAOB / DeepSeek-V4-Flash | Exact Match | 0.867 | 0.762 |
| SWE-bench Verified / DeepSeek-V4-Flash | Resolved | 76.5 | 68.9 |
| GDPval / Qwen3.6-35B-A3B | Score | 56.3 | 49.7 |
| Raven-Oncall, internal eval | Success / cost per case | 82.35% / $5.10 | 64.71% / $13.60 |
Raven ranks first on all four MAOB graph metrics under both backbones: Node F1 0.923 vs 0.776 and Exact Match +10.4 pp on Qwen3.8-27B, +10.5 pp on DeepSeek-V4-Flash. On GDPval with the Qwen3.5-397B-A17B backbone it scores 59.3 vs 56.0 (+3.3 pp). The internal on-call evaluation shows +17.6 pp success at 62.5% lower cost per case. Note the baseline convention: 'Best Baseline' is the strongest reported comparator for each backbone and metric, so the rival differs across settings.
For platform teams the value sits less in scores than in three engineering decisions. The harness becomes a versioned, evolvable artifact, and the composition layer only manages typed handoffs. Third-party agents join as workers through adapters: a team already invested in Claude Code or Codex adds an orchestration layer instead of migrating. And many details are portable without adopting the system: pre-dispatch validation, untrusted-data marking of upstream artifacts, independent completion judging, bounded exception recovery, reuse by reference, and the rule that approval requests are never auto-answered. On theory, the claim that composition covers tasks individuals cannot is usually left to intuition in systems papers; here the premises are explicit, so when composition fails you can ask which one broke. The code is open source.
The authors are candid: the results are sufficient conditions whose applicability depends on the implementation, and measured average success rates alone do not establish the conditional reliability premises. The parity construction caps individuals at 1/2 through artificially restricted interfaces; a single agent given both read interfaces would solve it too, and the paper concedes this is no claim of superiority over a differently-harnessed agent. On the empirical side, MAOB measures planning agreement before execution, not end-to-end outcomes, and the paper itself says graph agreement needs complementing with end-to-end results, cost accounting, and failure attribution. Baselines differ per setting, and several gains in Figure 1 sit under 3 pp (SWE-bench Pro 54.4 vs 52.4), so the 'significantly outperforms' framing rests mainly on MAOB, SWE-bench Verified, and the on-call result, which is an internal AI4S evaluation others cannot reproduce. Evidence for harness evolution and skill reuse is drawn from previously published HarnessBank and SkillCorpus experiments, not a fresh end-to-end evaluation of Raven. One engineering risk: a failed or timed-out judge counts the node as accomplished, and the paper states the verdict does not certify objective correctness, so completion statistics can drift optimistic. The theory charges planning, judging, retries, and handoffs to the budget, but only the on-call setting reports cost per case.