Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance
Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee, Simon, Lee
Sang Won
cs.MA, cs.AI, cs.CL, cs.SE
2026-08-27
On LCB-100, a zero-shot manager-worker scaffold lifts Qwen3.8-27B from 63.0 to 86.4; GPT-5.6-Terra with it scores 85.0, near Fable 5 at one-fifth the cost.
Multi-agent LLM setups are often said to beat a single model. The evidence is mixed, and the comparisons usually move several knobs at once: token budget, tools, prompts. An aggregate gain then cannot say which knob did the work.
This paper, from Persis Capital, holds the model and the problem set fixed and changes only the scaffold: the same weights answering in one pass, versus the same weights inside a manager-worker loop over a shared filesystem. Nothing is trained. Nothing is tuned to the benchmark. They call it zero-shot self-orchestration. The question is how much accuracy that extra test-time structure buys, and at what bill.
State lives on disk, not in any one context window: the problem, a plan, a task list, accumulating notes, and the current best program. Every role is a fresh call to the same model. Coordination happens only through those files.
The loop is short and explicit. The manager writes a 3-6 sentence strategy and a handful of seed tasks. The first worker brainstorms only: difficulties, candidate approaches, pitfalls, written into notes, no code. The manager then curates the task list and names exactly one next step. A new worker executes that step, rewrites the program, and appends notes. In the v2 scaffold, the candidate is run against the problem's public sample tests whenever the worker produced new code. Those stdin tests cover 73 of the 100 problems; 27 LeetCode-style items are function-call tests the engine does not execute. A failing sample overrides a "done" verdict. Guards: 10 rounds max, stop if the manager reissues the same task, and a short summarizer if a worker hits the token cap.
The baseline is one call of the same model at temperature 0.2, no tools, no loop. Headline numbers use pinned backends (OpenAI, local vLLM, Anthropic) and v2. An earlier sweep of five more models went through OpenRouter on a thinner original scaffold and is reported separately.
Headline setting: 128k output cap, thinking on, five independent passes on the 100 latest hard LiveCodeBench problems.
| Model | Single | Manager | Δ |
| Qwen3.8-27B | 63.0±4.1 | 86.4±2.7 | +23.4 |
| GPT-5.6-Luna | 67.2±4.3 | 77.8±2.0 | +10.6 |
| GPT-5.6-Terra | 77.0±1.0 | 85.0±1.0 | +8.0 |
| Claude Fable 5 | 87.4±1.1 | not run | - |
All three within-model deltas clear p < 10^{-4}. The manager wins four times as many problem-passes as it loses, or better: 125 vs 8 for Qwen, 71 vs 18 for Luna, 51 vs 11 for Terra, over 500 cells each.
The scaffold roughly triples the bill: Qwen $20.44 to $51.75 per 100-problem pass, Luna $0.41 to $1.50, Terra $3.41 to $11.71. Fable 5's single call is $61.11.
Three points on the cost-accuracy frontier. Terra with a manager scores 85.0 against Fable 5's 87.4 (p=0.59) at one-fifth the price. Qwen with a manager scores 86.4 against the same 87.4 (p=0.73) at $51.75, $9.36 cheaper than Fable on OpenRouter list rates. Luna with a manager matches Terra's unaided 77.0 at 44% of Terra's price.
Part of Qwen's +23.4 is truncation rescue. At 128k, the single arm hits the cap on 150/500 cells and emits no code on 35; the manager converts 25 of those 35 into passes, +5.0 points, about a fifth of the gain. The two OpenAI arms never truncate and never return empty answers, so their +8.0 and +10.6 contain no rescue.
Transcripts keep showing two mechanisms. Context management: each worker sees only the plan, notes, current code, and one task, so a short call is less likely to run away. One Qwen single pass spent 675,000 characters of reasoning, repeated the same sentence 7,743 times, and emitted no program; the manager cuts the work into bounded calls so an answer actually reaches disk. Decomposition is the other: split a two-objective DP across workers, or write the lemma into notes before anyone codes.
On the OpenRouter sweep, Opus-5 goes 85 to 91 in one pass, the highest score in the paper, and it was not repeated. With thinking off, Kimi-K3 gains +30.4 over five 16k passes and +42 in one 128k pass; Minimax-M3 gains +11.0 and +12. Qwen3.6-35B does not: -1.2 at 16k off, -9 at 128k off.
For anyone shipping a coding agent, this is a training-free orchestration layer with a bill attached. Weaker models and thinking-off runs take the most from it. Strong models with thinking already on pick up a few points, and those points still often come cheaper than stepping up a model class.
The paper does not claim multi-agent is more information-efficient per token. The loop spends more. The claim is that the extra spend often beats buying a larger model.
The scaffold can make things worse. On one Qwen3.6-35B problem, the single call wrote a correct convex-hull-optimized DP; brainstorm discarded CHT as "complex and error-prone," committed to a slower O(n³) table, then implemented it with a bug. That model is the standout loser with thinking off.
Fable 5 has no manager arm. Qwen's 128k single scores are a replay of 250k generations truncated after the fact; a model told the smaller cap up front might budget differently. Sample tests cover 73/100 problems. All reported numbers are competitive-programming code generation. Math and knowledge probes sat near ceiling for frontier models and were not published.
The OpenRouter set counts gateway failures as wrong answers, so absolute scores there are a lower bound; that is why the headline moved to pinned backends. This is also not an equal-token comparison, so it does not sit on the same axis as Tran and Kiela's finding that single agents match multi-agent systems once thinking tokens are held constant.
They also found a real LiveCodeBench harness bug: the mock for sys.stdin.buffer.readline() always returned the first line. 311 of 3,456 coded outputs in §2.1 used that idiom, and 97% of those were scored wrong. Fable 5 never uses it; its scores did not move. A harness bug of this shape silently penalizes models by coding style.