SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
Yan Zhan, Shaobo Liu, Qiunan Liu, Yuanjun Shi, Siqi Xu, WeiYi Hou, Xiang Xu, Zekang Li, Weizhou Pan, Jiahong Yan
cs.AI, cs.LG
2026-09-24
SLCA-GRPO routes tool and summary advantages separately. On 7B, matched gaps are +2.53 pp Toucan Success and +9.15 pp τ²-Bench Pass1, with no extra rollouts.
A tool-calling rollout is usually two segments: reasoning interleaved with structured calls, then a natural-language summary for the user. GRPO-style on-policy RL broadcasts one trajectory-level advantage to every token. A fluent summary can reinforce a bad call. A wrong summary can punish a correct call. The paper names this cross-segment credit misattribution.
Local tool bonuses and planner-summarizer splits do not cut the summary-to-tool gradient inside a single policy. Diagnostics show tool and summary gradients staying near-orthogonal while training still spikes, which points to advantage contamination rather than a directional gradient fight.
SLCA-GRPO has three moving parts.
Segmentation reuses the ordinary token mask. The last contiguous run of learnable tokens is the summary; every earlier learnable run is the tool segment; environment-injected observations get no gradient. A rollout with no tool call is treated as summary throughout.
Rewards split the same way. Hierarchical Rewards (HierR) score the tool segment with a dense process term (names, arguments, match to gold) and the summary with a terminal preference. Each reward is group-normalized on its own, then routed: tool tokens see only Âtool, summary tokens see only Âsum. The tool-side gradient has no functional dependence on the summary reward. That is a structural axis, orthogonal to temporal credit methods such as VinePPO, GiGPO, and SPO. The paper does not test the combination.
To avoid live APIs at RL scale, a Schema-Guided LLM Simulator (SGLS) validates calls against the schema, then mocks returns with a frozen cross-family LLM. The objective stays PPO-clip on one backbone; only the advantage tensor changes. Group size G=16, one RL epoch.
Toucan supplies 42,423 SFT traces, 31,818 RL traces, and a 4,000-example test split. The matched baseline is SFT+GRPO with the same initialization, HierR, and SGLS.
Toucan strict [email protected] (process score ≥ 0.9):
| Backbone | SFT+GRPO | SLCA-GRPO | Gap |
| Qwen2.5-3B-Instruct | 74.12% | 76.47% | +2.35 pp |
| Qwen2.5-7B-Instruct | 76.60% | 79.13% | +2.53 pp |
| Qwen3-8B-Base | 76.97% | 79.02% | +2.05 pp |
Out of domain, the 7B matched gaps are +1.36 pp on BFCL and +9.15 pp Pass1 on τ²-Bench. On 8B, BFCL Overall is 70.31% against GRPO's 66.96%, and τ²-Bench is +10.03 pp. The 3B gaps are only +0.40 and +1.09 pp; the long-horizon lift shows up at 7B and 8B.
7B ablations, three seeds: drop SLCA and Toucan Success falls from 79.13% to 76.60%, τ²-Bench Pass1 from 41.02% to 31.87%. Drop SGLS schema constraints and τ²-Bench is 36.19%. Drop HierR and keep a sparse summary reward, Toucan Success is 73.44%. Training traces show SLCA ending with fewer tool turns and higher success; GRPO settles on longer trajectories without a matching success gain, consistent with padding calls to farm the summary term.
Anyone training a tool agent with GRPO should check whether one advantage is being painted across both segments. A correct call followed by a bad write-up should not update API selection. SLCA does not add rollouts from intermediate states. The 9 to 10 point τ²-Bench gaps at 7B and 8B are the number to watch; dual-control, long-horizon settings punish using the summary outcome to train the tools.
The in-domain +2.53 pp is incremental. ToolPO and RLTR look worse in the main table, but their reward protocols are not fully aligned, so those rows are not a clean knockout.
The authors list the caveats. Segmentation needs a clean boundary between calls and free text; inline code generation would need a learned cut. The process reward matches gold trajectories and can under-credit another valid API plan. Toucan [email protected] is a threshold on that process score, not an independent task label. Inside the tool segment a single scalar still cannot tell a good Action 1 from a bad Action 2; stacking VinePPO-style temporal credit is untested. Runs stop at 8B in a simulator, with no large real-API comparison.