Can Agents Design Libraries for Agents?
Gabriel Orlanski, Alex L. Zhang, Avi Trost, Vincent Sunn Chen, Frederic Sala, Aws Albarghouthi, Ludwig Schmidt
cs.AI, cs.CL, cs.SE
2026-09-29
LibraryDesignBench scores libraries by downstream correctness and size. Opus 5.5 gets 48.9 vs production 46.6; 64% of extra code is rigid or verbose APIs, only 14% missing features.
Coding agents now inherit repositories other agents wrote. They rewrite more than they reuse, so later agents work in a larger pile. Software engineering exists because someone compressed a domain into a library. The question is whether an agent can design a library other agents will actually use.
Unit tests only check that a library is correct. Scoring an API against a spec freezes the design under study. Human review scores what people like. This paper grades a library only by watching three weaker coding agents, systems that edit a repo and run commands, write programs against it: are those programs correct, and how small are they.
LibraryDesignBench runs in two phases.
Fifteen library-design tasks and 242 expert-reviewed problems cover Python, TypeScript, Rust, and Haskell. Reference programs use mature production libraries, clap for clirs among them. Every problem must remain solvable with no library.
The library score is the squared test-pass fraction times simplicity relative to the reference, averaged over problems and the three implementers, then scaled by 100. Squaring weights correctness: passing 80% at reference size scores below a fully correct program 1.5 times as large. Simplicity averages four ratios capped at 1: cyclomatic complexity (branch count), cognitive complexity, Halstead volume (operator and operand bulk), and formatted source lines. A library that does not install scores 0.
Short and wrong, or correct and bloated, both land near the bottom.
Pass rates do not separate designers. They sit between 84.1% and 86.6%, and 86.4% with no library. Score gaps come from how small the downstream code is.
| Setup | Score | Pass | Simplicity |
| Opus 5.5 | 48.9 | 86.6% | 64.5 |
| Fable 5.1 | 47.5 | 86.1% | 62.7 |
| Production library | 46.6 | 85.4% | 61.5 |
| GPT-6 Astra | 45.1 | 85.7% | 58.7 |
| No library | 34.4 | 86.4% | 46.3 |
| DeepSeek V4 Pro | 31.2 | 84.9% | 42.0 |
Opus 5.5 is 2.3 points above the production library (4.9% relative). Production itself is 12.3 ± 0.4 points above no library, at about $0.14 extra per problem. DeepSeek V4 Pro sits 9.2% below no library: the library is a liability. In Haskell, about 70% of (designer, task) pairs lose to giving no library at all. On the three tasks where no library already beats production, Opus 5.5 still trails no library by 3.2 points.
The harness moves the number. Fable 5.1 scores 47.5 in mini-SWE-agent and 39.9 in Claude Code. All three implementers agree on the top three designers and on last place, and none prefers its own family.
On 11 of 15 tasks, designers reproduce the production library's abstractions. For clirs, Astra and Fable copy clap's builder chains and skip the shorter derive macros.
Downstream agents do call the library. They underuse it. In 810 partially passing solutions longer than the reference, 82% of extra-code labels land on library limits: rigidity plus verbosity are 64%, missing coverage is 14%. Across 13,562 rebuilt lines in 180 solutions, 41% redo a feature the library already shipped but hid or broke. For clirs, 24 of 33 libraries make long-option prefix inference painful; those that lack it score 16.7 points lower on problems that need it.
Even with the production library, the three implementers only reach 61.5 simplicity. LibraryUseBench freezes the library as production and swaps eight models in as users: Opus 5.5 leads at 66.9. For GPT-5.6 Luna, raising effort from low to high lifts pass rate from 51.4% to 84.5% and library files read from about 2 to 15. A more prescriptive prompt lifts simplicity from 48.7 to 60.4. Under the minimal prompt, 24% of trials never open the library, and 36% of the reference's library symbols stay unseen, versus 5% under the default prompt. Luna greps for API names it already expects, so capabilities it cannot recall go unused.
Appending an agent-first brief to GPT-6 Astra (Codex) (sketch consumer programs first, freeze them as runnable examples, test with subagents) drops exported-name overlap with production from 19.4% to 13.4% and raises the score from 44.1 to 46.4, still short of production at 46.6, with simplicity up 6.8% relative. Design cost nearly doubles ($8.24 vs $4.51); per-problem cost does not move. The three ingredients are not ablated.
This is the first benchmark that scores library design only through other agents using the result. If agents are going to build on each other's code, the measurement belongs on downstream correctness and size, not on whether the API looks human.
The finding is chilly. The strongest designer can edge past production libraries by reimplementing human abstractions. Downstream agents underuse even clap and pydantic. Telling the designer that only agents will call the library, then testing with subagents, shortens downstream programs and still only catches production.
What transfers is the evaluation recipe and the design loop: write the consumer programs first, lead with examples, accept the library with a subagent that never saw the design. That is useful for internal toolchains and SDKs aimed at coding agents. It is a poor overall leaderboard for which model designs libraries, because the score is locked to these 15 tasks, these three user models, and this budget.
The authors draw the boundary: scores describe utility under this task set, these consumers, and these budgets, not a consumer-independent ranking. The metric covers test pass rate and static size, not full correctness, security, runtime, or maintainability. References are not uniquely optimal. The workload favors problems where a library can save a lot of code. Confidence intervals cover reruns of this fixed set.
The failure taxonomy is labeled by a GPT-5.6 Luna auditor from the same family as one implementer, with no human labels and no agreement check, so 64% carries auditor bias. The two or three visible spec examples stay in the scored set, so designers have seen part of the exam. The agent-first intervention is bundled; there is no split among sketches, examples, and subagents. Pass rates sit between 84% and 87%, sometimes higher with no library, so the board is mostly a simplicity ranking. Agent libraries often go negative in Haskell, and equal language weights mix "can design" with "this language went badly."