ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction
Yongqi Tong, Tan Li Hui Faith, Choy Zhen Wen Marcus, Zhou Jin, Kewei Fu, Jiang-Ming Yang, Jianshe Li, Xin Zhang
cs.AI, cs.CL
2026-08-13
ARC compares GRPO rollouts only inside a strategy class. On Qwen3-8B, tau-airline rises from 31.33 to 44.00; INTER3 cuts think-style TTFT from 4.91s to 1.27s.
Group-relative RL such as GRPO treats rollouts in one group as comparable on quality. Open-ended interaction breaks that. At a given user state, answering now, asking a clarifying question, streaming progress while tools run, or confirming before an irreversible action can all be right. Reward models already lean on length and style. Center those rewards across strategies and the advantage mixes “which speaking style the judge likes” into “which reply is better.” Optimization then drifts toward the judge’s favorite interaction style.
Ant International calls this a reward-fairness problem and answers with ARC. At train time each example carries a strategy instruction; rollouts are sampled and compared only inside that class. At inference the instruction is removed and the policy picks a strategy on its own. The supporting shell is INTER3, which splits user-visible text from latent reasoning and tool calls so progress updates, clarification, and mid-flight steering can be labeled and grouped.
INTER3 streams <answer> spans to the user, treats untagged text as hidden reasoning, and keeps tool calls as internal actions. The implementation change is small: add tags to the tokenizer, then extract visible spans. Users do not wait for the full hidden trajectory. Strategies sit in four families: progress update, clarify first, alignment check, and direct answer, with nine concrete tactics underneath.
INTER3-86K has 86,796 examples. The SFT split is 57.9K: 34.2K tool-use, 17.2K multi-hop QA, about 6.4K logic, rewritten into the interleaved format by Qwen3.5-397B. The RL split is 28.9K tool-use only, each tagged with one plausible strategy by two models and written into the system prompt. Progress update is 75.3% of the RL set; alignment check is 2.5%. Some traces come from live customer-service traffic on a large payments platform.
ARC’s four steps: attach a strategy, sample inside it, compute advantages inside it, update with an entropy bonus. Without entropy the policy collapses into empty <answer> blocks that farm format reward. Main runs use Qwen3-8B, GRPO, no-think. The reward is format plus tool exact-match plus an answer judge that fires only when the tool call is correct.
In-domain tool use is where ARC moves the needle, and the move depends on the optimizer.
| Method | Avg. | τ-airline | τ-retail | τ²-airline | AIME 2026 |
| Qwen3-8B-Think | 32.82 | 28.00 | 36.81 | 29.75 | 47.92 |
| GRPO | 28.09 | 31.33 | 40.29 | 36.67 | 31.67 |
| GRPO+ARC | 33.46 | 44.00 | 50.00 | 48.00 | 40.83 |
| PPO+ARC | 28.57 | 39.33 | 46.09 | 41.61 | 19.17 |
GRPO+ARC is +5.37 on the reported average. PPO and DAPO pick up only 1.08 and 1.31. PPO+ARC drops AIME from 30.83 to 19.17, so out-of-domain reasoning is not a free side effect. At 4B, GRPO+ARC averages 34.23 on the five tool tasks against 22.33 for GRPO. Versus a think-then-answer baseline, INTER3 cuts TTFT from 4.91s to the 1.27s reported for GRPO+ARC. Plain GRPO in the same table is 0.61s; the ARC variant is slightly slower.
Keeping strategy instructions on for the whole of training beats linear annealing and a constant 20% drop, especially on τ / τ². At inference, matched or random strategy hints score 39.85 and 39.58, both below the no-hint 41.73. The instruction is a training-time comparison class, not a deploy-time switch. The full four-strategy suite averages 29.59 against 21.74 for progress-update only; τ average goes from 23.62 to 47.00.
If a team already runs group-relative RL on open agents, this recasts a familiar failure. Stronger judges, more entropy, or a switch to PPO never asked whether unlike strategies were sitting in the same group. ARC barely touches the optimizer. It changes who is compared with whom. The cost is strategy labels and a runtime that can split visible talk from hidden execution.
The recipe fits multi-turn tool agents whose interaction style actually varies. Short-form tasks with one verifiable answer are already comparable inside a group; expected gains are smaller, and the PPO numbers are the caution.
The strategy taxonomy is coarse. Real dialogue strategies are finer and more context-tied. The strongest numbers are tool-use; whether the same unfair comparison shows up in other domains is untested. Labels come mostly from two LLMs, with people only on low-confidence disagreements, so the tags inherit those models’ idea of an appropriate strategy. Progress update is three quarters of the RL set, which can inflate the value of minority strategies. The theory is a stylized variance split for the estimator, not a convergence result for GRPO. Customer-service traces from a payments platform may not travel.