Learning Self-Orchestration in Decentralized Agent Swarms
Guowei Xu, Zhenting Qi, Amani Kiruga, Himabindu Lakkaraju, Yilun Du
cs.MA, cs.AI, cs.CL
2026-10-16
MPT jointly trains actions and bids, lifting held-out iron-pickaxe return from 20.3 to 41.5, navigation from 0.52 to 1.23, and research accuracy from 9% to 21%.
Multi-agent systems usually freeze the turn order. A manager delegates, a verifier watches a generator, or everyone speaks on a schedule. Member weights can move. The choice of who acts next usually does not.
That choice does two jobs at once. It steers the trajectory, and it decides who receives experience and gradient. Once skills change, the useful order changes too. A logger that should open a Minecraft episode later just burns turns that mining and crafting need. Fixed orchestration trains only one side of this coupling.
The target is a swarm that learns how to act and when to step in, without a dispatcher calling names.
Market-based post-training (MPT) gives every agent its own parameters. On each turn the agents bid, and the highest bid acts. Bids are five levels: 0, 0.2, 0.4, 0.6, and 0.8. Ties are random. If every bid is 0, the turn is skipped.
The winning bid also sets credit. Accumulated value equals this turn's environment reward plus the next bid times the value that follows. The actor keeps (1 minus its own bid) times that accumulation, and the rest goes back to the previous actor. A high bid buys the turn and keeps a smaller share. The prompt describes the bid as paying for the state the previous agent produced. Whatever the first actor would pass further back is discarded.
Action updates are REINFORCE. Credit is centered with that agent's mean on the batch, and only action tokens from turns it actually took are updated.
Bids learn from counterfactuals. If the runner-up bid is nonzero, the state is restored and the winner resamples its bid at most three times, until the runner-up takes over and finishes the episode. The two branches are scored by environment reward from that turn on. A Bradley-Terry preference loss makes the original winner more likely to emit the bid from the better branch. Equal returns, and turns that never change hands, are dropped. Updates alternate four action steps and one bid step on the same weights.
Supervised cold start splits demonstrations by role. The monolithic baseline trains on the pooled set, with cold-start compute matched. Training FLOPs used for comparison count every bid, every action, the counterfactual rollouts, and the parameter updates.
Verbs are fixed in the prompt. Crafting and navigation are one body steered by three policies in turn. The research seats share one transcript and split the tools. Crafting uses three Qwen3-VL-4B-Instruct agents, logger, miner, and smith. Frozen STEVE-1 runs each instruction for 40 frames across 120 decisions, from an empty inventory to an iron pickaxe, with staged rewards capped at 140. Navigation uses three NaVILA-8B agents for turning, moving, and stopping, at most 64 decisions on Habitat VLN-CE, with reward for geodesic progress plus 2.5 for stopping within 3 meters. Research uses three gemma-4-E2B-it agents for search, open, and find, at most 64 steps, with correctness judged by qwen3.5-flash. The single-agent baseline is REINFORCE on the same base model at discount 0.95.
Held-out evaluation is disjoint from training: 32 unseen crafting seeds, 88 unseen navigation routes, and 50 research questions, three runs each.
| Task | Single RL | MPT |
| Crafting return | 20.3 ± 4.1 | 41.5 ± 5.9 |
| Navigation return | 0.52 ± 0.14 | 1.23 ± 0.18 |
| Research accuracy | 9.0 ± 2.1% | 21.0 ± 4.1% |
At matched training FLOPs the curves end near 41 versus 20, 0.9 versus 0.3, and 27% versus 6%.
On iron pickaxe, participation at decision 120 shifts from about 80% logger and 10% each for miner and smith before RL, to about 45%, 40%, and 15% after.
With the trained action policies held fixed, the same 32 seeds score 23.8 ± 4.2 for round-robin, 20.9 ± 4.2 for AgentNet, 25.4 ± 4.6 for Graph-of-Agents, and 41.5 ± 5.9 for learned bids, a 63.4% relative gain over the strongest alternative.
Initial bids with initial actions score 15.4 ± 4.1. Trained bids alone reach 21.5 ± 4.7, trained actions alone 27.2 ± 4.8, and both together 41.5 ± 5.9, which is 2.69 times the initial pair.
Near 5×10^18 FLOPs, full MPT is about 42, bids replaced by uniform random credit about 31, and round-robin with discount 0.95 about 15.
Continued training from the iron-pickaxe checkpoints onto Torch and Stone Bricks reaches about 35 and 38 for the swarm after roughly 10^18 adaptation FLOPs, against about 17 and 19 for the single model.
Copying each trained seat one to four times, with no further training, grows the population from 3 to 12. Pass@3 moves from about 78% to 97% on a wooden pickaxe, 69% to 94% on a stone pickaxe, and 66% to 91% on a furnace. At dropout probability 0.9 the 12-agent group still scores about 33, against about 19 for three agents.
Who acts and who receives credit are the same learned number. After compute is matched, crafting, navigation, and research all beat monolithic post-training. Code and trained models are public.
The reusable piece is the recipe. Humans still write the verb lists. The models learn when to bid and what to say inside those verbs. Extra copies at deployment raise the early crafting milestones without retraining a scheduler.
A return of 41.5 against a cap of 140 is about 30%. Held-out research accuracy is 21%. The gap versus single-agent REINFORCE holds. These runs are not finishing the iron tech tree, and they are not a research agent you would hand a question to.
Legal verbs live in the prompt, so the swarm learns when a seat should step in, not how to discover a division of labor. Growing to 12 agents copies the same three seats. Tied bids are broken at random, and the paper does not separate that redundancy from the learned policy.
The coordination bake-off and the ablations are iron-pickaxe only. Navigation and research are compared only with single-agent REINFORCE, not with standard multi-agent credit assignment. Navigation success rate and path efficiency are not reported.
With 32 seeds, a 63.4% relative gap will move. Research reward comes from an external judge, so judge errors enter training. The low-level controller is frozen and spends about two seconds on each instruction, so motor misses update bids and actions too.
At dropout 0.9, twelve agents expect about 1.2 members still available and three agents expect about 0.3. A score of 33 versus 19 mostly tracks how many agents remain available. Inference latency is also missing. Every agent bids on every decision, and that cost grows with population outside the training-FLOP ledger.