One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents
Jie Zhao, Ziyu Jiang, Suhang Zheng, Minghui Shan, Xiaoxiao Xu, Lin Qu
cs.SE, cs.CL, cs.LG
2026-09-20
Alibaba trains three RRE category experts and distills them with label-routed MOPD: 58.04% on Pro-618 (+5.39) and 59.00% on SWE-bench Multilingual (+2.78).
Repository-level SWE tasks share one interface: read the repo, edit code, run tests. The work itself is not one distribution. Fixing a service-layer bug, changing a UI, and patching infrastructure pull different evidence, tools, and checks. Most post-training recipes dump them into one pool, run joint RL, and report a single resolution number.
That number hides the trade. Alibaba calls it the category see-saw: one group of tasks rises while another falls, and the aggregate still looks stable. On Pro-618, an audited 618-instance slice of SWE-bench Pro split into A/B/C (221/201/196), pooled RL shows exactly those opposing moves. Balancing the mixture (516 tasks per category, 1548 total) keeps overall scores close and slightly lifts the weakest-category average gain. The see-saw does not go away.
The paper asks a different question: train specialists per category, then fold them back into one deployable policy.
Four pieces: executable tasks, SWE Labeler, Refresh-Repair-Expand (RRE) expert training, and label-routed multi-teacher on-policy distillation (MOPD).
SWE Labeler is a sourced multi-axis tagger. Two semantic axes: task type (26 L1 / 119 L2, grounded in ISO maintenance vocab, CWE, and Fowler refactoring) and repository domain (21 L1 / 108 L2, by software ecosystem). Three four-level scale axes cover change scope, cognitive load, and estimated time. One expert per fine label would shred the RL budget, so the main run uses only Domain L1, hard-mapped into three routes: A service/data/security, B user-facing apps, C systems/tooling/runtimes.
All experts start from the same Qwen3.6-27B with an R2E-Gym scaffold. No external model supplies solution trajectories. Reward is executable pass/fail.
RRE runs two rounds per category. Agentic-miniRL is MiniRL on long multi-turn SWE traces. RLOO leave-one-out baselines with G=8 keep a lone success at advantage 1.0 instead of 7/8 under inclusive group means; executable wins are already scarce. K1 regularization sits in the reward path to limit drift. Clipped importance weights correct rollout-versus-trainer probability mismatch. Loss is reduced per assistant turn because those turns vary wildly in length.
Refresh then re-probes the current training set with four fresh rollouts. Average gains do not mean every instance improved: across 2769 training records after initial RL, 1108 rose, 839 fell (30.3%), 822 were unchanged. Repair SFT replays only verifier-approved successes from that expert's previous RL phase, with more quota for low-mastery items (four trajectories if 0-1 of 4 succeeded, one if all four passed). Expand samples previously omitted base-0 and base-1 tasks, re-scores them, and rebuilds the next RL frontier. Saturated items and never-solved hard zeros stay off the main budget.
MOPD then trains one student from the original base on student-generated traces. Environment reward is not in the student loss. The advantage is imitation plus ReLU-gated reward extrapolation: only the direction where the routed teacher is more probable than the reference (the initial base), with λ=1.25. Ungated extrapolation mixed positive and negative teacher-reference gaps of similar size, so the net term nearly cancelled; the ReLU gives it a stable sign. A and B records are repeated to match C's 1652, for 4956 records and no new tasks.
Pro-618, three-round means:
| Policy | Full | Pro-A | Pro-B | Pro-C |
| Base | 52.64 | 51.73 | 54.39 | 51.87 |
| Pooled RL | 55.50 (+2.86) | 54.75 | 56.72 | 55.10 |
| Balanced RL | 55.34 (+2.70) | 54.45 | 57.05 | 54.59 |
| Routed experts (3 models) | — | 59.58 (+7.84) | 58.87 (+4.48) | 57.48 (+5.61) |
| MOPD (single model) | 58.04 (+5.39) | 58.07 (+6.33) | 59.37 (+4.98) | 56.63 (+4.76) |
Versus Pooled RL, MOPD is +2.54 overall and +3.32 / +2.65 / +1.53 on A/B/C, so the minimum category lift is 1.53 points. Versus Balanced RL the minimum lift is 2.04. Expert-gain recovery is 80.8% on A, 111.1% on B (student 0.50 above its expert), 84.8% on C.
On the training records, Repair SFT is sharper: success rates move from 46.05 / 45.54 / 43.28 after initial RL to 56.28 / 59.59 / 52.75. Of 839 records that dropped in initial RL, 751 entered repair; 69.8% / 73.0% / 61.3% of those recovered to or above base for A/B/C.
On SWE-bench Multilingual (300 tasks, SWE-agent): MOPD 59.00%, base 56.22% (+2.78), Pooled 55.56%, Balanced 57.00%. All three routed categories beat both joint-RL baselines. Category C is flat versus base (both 61.74%); the extra points sit in A (+6.48), B (+8.00), and 27 unassigned tasks (+6.17). The bootstrap interval versus Balanced RL crosses zero ([-0.78, 4.78]), so that comparison is weaker.
Three practical takes for SWE-agent post-training.
Aggregate resolution is a bad optimization target. Opposing category moves survive a balanced mixture, so the issue is not only sampling ratios. Treating repository-level SWE as one homogeneous domain hides internal redistribution.
Splitting data is not enough to make experts. Initial per-category RL is only +0.66 to +1.81 on Pro-618, still below the Pooled RL endpoint. RRE is what opens the gap: replay own successes, reselect tasks by mastery, and after two rounds the experts beat Pooled. Category split changes what is learned; RRE changes whether it sticks.
Distillation need not buy trajectories from a larger teacher. Same-origin experts, same tool interface, student rollouts, teachers scored on prefixes the student actually visits. For a team that already has sandboxes and a strong base, this is cleaner than sample-then-SFT-then-RL from a bigger model. Compute is the bill: three experts plus teacher forwards are not free.
This is incremental. MiniRL, MOPD, and ExOPD already existed; the work is the category diagnosis and the training loop that uses it.
Hard routing puts multi-category tasks into one expert. There is no ablation on K, compute split, replay policy, or routing granularity. Same-origin teachers may be too close to add signal, or too far after heavy specialization for stable distillation. Pro-618 drops known-bad environments and graders from 731 to 618 tasks; scores do not transfer to full Pro. Pass rate is not patch quality. The anti-hacking stack blocks local git archaeology and sandbox tampering, not pretraining memorization or every weak verifier. Single-run category curves carry evaluation noise; the paper says so.
A few results look thin on their own. Multilingual C does not move, and the 27 unassigned tasks lose ground versus both joint-RL baselines, which is a tax on tasks outside the router. MOPD's SSG (0.63) is wider than Balanced RL's (0.04): higher overall and minimum gains, less uniform ones. Expert checkpoints are picked on the matching Pro subset, so model selection peeks at the evaluation slice.