CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Jiaxuan Jiang, Liyuan He, Zhixuan Fang
cs.AI, cs.LG
2026-09-16
CERA-MoA co-trains a familiarity router with LoRA agents. On Qwen3-4B: 63.2 ID / 72.8 OOD vs AT-GRPO 61.0 / 71.5, with ~45% fewer tokens than fixed Top-2.
Mixture-of-Agents work has split into two camps. One camp treats agents as frozen experts and only tunes debate protocols or query routers. The other fine-tunes agents, but inside a fixed workflow, with data cut by hand. Both keep "who should answer this" separate from "how this sample should reshape someone".
That split is a real bottleneck. Agent skill keeps moving during post-training, so a static router goes stale. Uniform or hand-split data also fails to push complementary expertise, and everyone stays a generalist. CERA-MoA, from Tsinghua IIIS, Shanghai Jiao Tong, and Shanghai Qi Zhi Institute, puts the router and the agent policies in one RL loop.
The system is a parameterized router, N continually learning agents, and a voting aggregator. In the shared-backbone setup N=4, each agent a LoRA (rank 16, α=32) on a 4B model, about 4.4B in total. The heterogeneous pool uses Qwen3-4B, Llama-3.2-3B-Instruct, and Phi-4-mini.
Familiarity is scored without a full generation. The router freezes its backbone and concatenates hidden states at layers L/2 and 3L/4. Each agent has a trainable predictor head and a frozen, randomly initialized target head, orthogonal across agents. Familiarity is an exponential decay of the normalized Euclidean distance between the two projections: closer means more familiar. Reward becomes a relative push-and-pull around a fixed anchor, not a direct regression of absolute reward.
During training the ranking score adds a UCB bonus and historical generation entropy on top of familiarity. The cutoff still uses familiarity only: accumulate from the top until the sum hits τ (default 0.7); if the whole pool cannot reach τ, everyone is activated. Exploration terms are off at inference.
Selected agents update with DAPO plus GSPO sequence-level importance sampling, 8 rollouts per query. The router looks at relative advantage versus the current subset: winners are pulled toward the frozen anchor, losers are pushed away by a margin m=1.0. Prompts carry no role personas, only the raw question, so specialization has to come from data allocation.
For tasks with a unique answer, the aggregator does familiarity-weighted majority vote. For open-ended generation it just emits the most familiar agent's completion.
Training uses about 32k examples across math, code, instruction following, and BIG-bench Hard. OOD tests (no extra fine-tuning) are IFEval, HumanEval, AGIEval, ARC-c, LogicBench, and OlympiadBench.
| Setup | Method | ID avg | OOD avg |
| Qwen3-4B | Base | 49.6 | 53.0 |
| Qwen3-4B | AT-GRPO | 61.0 | 71.5 |
| Qwen3-4B | CERA-MoA | 63.2 | 72.8 |
| Llama-3.2-3B | AT-GRPO | 40.7 | 51.0 |
| Llama-3.2-3B | CERA-MoA | 47.9 | 53.9 |
| Heterogeneous pool | AT-GRPO | 61.8 | 69.1 |
| Heterogeneous pool | CERA-MoA | 63.0 | 71.8 |
CERA-MoA does not win every cell. On Qwen3-4B, MBPP is 60.3 versus MAPoRL's 64.2; HumanEval is 82.5 versus 85.1. The average, and the hard reasoning sets (DAPO-MATH, LogicBench, OlympiadBench), are where it pulls ahead.
Ablations on Qwen3-4B: direct reward regression 60.8 ID, multi-class classification 61.3, the geometric familiarity score 63.2. Fixed Top-2 gets 63.3 ID at 666 tokens; adaptive threshold 63.2 at 368 tokens, about 45% fewer. Drop UCB and entropy, ID falls to 57.1. Final-layer features lag mid-layer, 61.1 vs 63.2. Router latency is 11.87 ms versus 1350 ms for native generation, under 1%.
The four Qwen agents split work without role prompts: Agent 2 takes most long-chain reasoning, Agent 1 concise math and logic, Agent 3 code, Agent 4 general reasoning. t-SNE of mid-layer states shows queries for the same agent clustering together.
This is an incremental systems paper, not a new paradigm. The useful move is treating the router as a competence estimator that updates with the LoRAs, so data assignment becomes the cause of specialization. Four LoRAs on a 4B backbone stay near 4.4B parameters; 1,000 optimizer steps, 16k queries, cost about one H200 GPU-day. That is a workable recipe for people already doing MoA post-training. The heterogeneous pool also shows you need not force distinct bases into one debate template.
It fits single-turn tasks with verifiable rewards. Open-ended generation still picks one answer. Multi-turn and debate are not wired in.
The authors flag two limits. Familiarity is read from the initial prompt's mid-layer states, so required expertise can drift in multi-turn work and the current router will not track it. Open-ended tasks skip any synthesis of multiple completions, giving up the usual benefit of generating several candidates.
A few gaps they do not dwell on. Orchestration baselines simulate four experts with role prompts, which is not the same object as four trained LoRAs. On OOD, AT-GRPO is already close: Qwen3-4B only +1.3. The gain is clearer on ID and hard math. On HumanEval and MBPP, debate-style MAPoRL still scores higher, so familiarity-weighted voting is a weak aggregator for code. Training with τ=0.8 drops ID to 61.2; the threshold is not wide.