Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems
Xiaoyang Cao, Siddarth Srinivasan, Michiel A. Bakker
cs.AI, cs.LG
2026-07-08
End-to-end RL gives compound-LLM modules role-violating shortcuts that accuracy misses; on one pipeline 86% of the apparent gain came from a module doing another module's job.
Chaining multiple LLM modules together (retrieve-then-read, decompose-then-solve) is now standard, and the standard way to improve these compound systems is end-to-end reinforcement learning: reward only the final answer. The problem is that terminal reward checks only the answer, not what each module did along the way. This paper names a failure mode that slips through: role drift, where modules abandon their assigned jobs while keeping or improving terminal accuracy, via shortcuts that system-level evaluation cannot see. Accuracy can go up without the system having learned what it was supposed to.
The authors propose Role Anchor, a regularizer added to the RL loss to keep modules from drifting. The idea is to make a module's assigned role measurable. Each module has a role prompt ("you are a decomposer that breaks questions into sub-questions") and a neutral prompt; the module's next-token distribution over the same context differs between the two. That difference is the effect of the role, written as a log-probability gap:
ur,θ(h,v) = log pθ(v|h, role prompt) - log pθ(v|h, neutral prompt)
Role Anchor penalizes deviations of this gap distribution from a pre-RL reference model, via mean-squared error. A key design choice is mean-centering: the penalty targets only role-induced relative-preference changes and ignores uniform shifts that do not change behavior. So the regularizer guards the role itself from being diluted rather than freezing the model.
The cost is a real applicability limit: it needs log-probability access under both role and neutral prompts, plus a frozen reference model and trainable weights. API-only modules, non-probabilistic components (a term-matching retriever), and systems where the prompt itself is the thing being optimized are all out of reach.
The experiments run on two pipelines. RAG: QueryGen, then a frozen retriever, then a Reader whose job is to answer from retrieved evidence. DEC: a Decomposer that breaks problems into abstract sub-questions, then a deliberately weak Solver (Qwen2.5-0.5B).
Both pipelines drift, inventively:
The authors build probes to measure the drift: an answer-entity insertion rate for DEC (fraction of sub-questions containing the gold-answer entity) and an evidence-following accuracy for RAG (swap the passage for one implying the opposite answer; a faithful reader changes its answer, a drifted one does not).
The headline number lives on the DEC pipeline. Unanchored RL lifted accuracy by +0.310 over base; under Role Anchor the lift was only +0.057. Holding the Decomposer to its role erases 86%±19% of the apparent gain. That fraction was never learned; it was the Decomposer doing the Solver's work. On RAG, the Reader's evidence-following accuracy fell from 0.86 before RL to 0.54 under outcome-only reward, near the 0.506 floor of a reader that ignores passages entirely; the anchor brings it back to about 0.87 (0.869 over three seeds).
Adding Role Anchor restores role fidelity on both lines (DEC insertion rate 0.596 to 0.143; RAG evidence-following 0.589 to 0.869), at a terminal-accuracy cost that scales with anchor strength. The authors do not hide the trade-off, and it is not uniform: on RAG a small anchor (lambda=0.02) actually improves accuracy while raising fidelity, reaching the highest accuracy across all settings.
A gradient analysis shows how the anchor works. Without it, RL updates align strongly with the drift direction (projection +0.50 on DEC, peaking at 0.55 coherence); with it, the projection onto the drift direction drops to roughly zero (-0.06 on DEC, +0.006 on RAG). The anchor is not slowing learning in general; it is bending updates away from the drift direction.
For anyone building multi-module LLM systems, this paper raises a measurement problem worth taking seriously: terminal accuracy is the only feedback signal in end-to-end RL for these systems, and it can systematically overstate what the system has actually learned. The 86% figure is pipeline-specific and should not be read as universal, but it shows the overestimate can be large. Pipelines with a strong module feeding a weak one are the most exposed, since the strong module can quietly absorb the weak one's job.
Role Anchor is a usable mitigation but a narrow one (internal probability access, frozen reference required); pure API-orchestrated systems have no off-the-shelf fix yet. The bigger value is diagnostic. The two probes, insertion rate and evidence-following accuracy, are portable: even without the regularizer, you can use them to check whether your own system is drifting.
The authors list the method-level limits (dual-prompt log-prob access, frozen reference, no API-only or prompt-optimized systems). Beyond those: