From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL
Wenyue Hua, Zachary Huang, Tyler Payne, Safoora Yousefi, Saleema Amershi, Asli Celikyilmaz
cs.AI, cs.CL, cs.LG, cs.MA
2026-08-14
Microsoft post-trains Qwen3-4B with RL across six negotiation environments; the unified 4B reaches 0.627 average utility, above GPT-5.1 (0.619).
Agents increasingly act for their users: scheduling meetings, comparing offers, haggling over prices. Across the table sits a counterpart with conflicting goals. Microsoft's measurements show that the dispositions making an assistant pleasant make it a poor delegate: frontier models disclose the principal's private information unprompted and concede at the first sign of resistance. RLHF optimizes for cooperative dialogue, not for holding a position against an adversary.
SocialRL from Microsoft Research treats social reasoning as a direct post-training target: six negotiation and coordination environments, one 4B base model (Qwen3-4B-Instruct-2507), testing whether strategic delegation can be trained into a small model instead of borrowed from frontier scale.
The six environments span four structures: Deal-or-No-Deal and CaSiNo (multi-issue allocation), Craigslist and Marketplace (single-issue price bargaining), Job Interview (five-issue contracts, roughly 10^4 possible agreements per scenario), and Calendar (slot coordination). Rewards are terminal, outcome-only scalars normalized to [0,1]: allocation games reference envy-free Pareto-optimal outcomes, price games split the bargaining corridor, and Calendar normalizes by the preference range over feasible slots.
Training runs in two stages. Stage 1 trains one specialist per environment: direct PPO (a Qwen3-1.7B critic, up to 200 steps) for four environments, and SFT warm start plus PPO for the two price games, because the base model almost never explores anchoring on its own. Stage 2 consolidates the specialists either by cascade RL, ordering environments by the measured transfer matrix, or by MOPD, multi-teacher on-policy distillation where the student rolls out on-policy and specialist teachers supervise token-level logits under a reverse-KL objective, with sampling budget allocated to domains whose teacher advantage remains unclosed.
The transfer matrix is measured first: every specialist is evaluated on all six environments.
| Policy | DnD | CaSiNo | Craigslist | Job Int. | Calendar | Mkt | Avg |
| Base 4B | 0.583 | 0.476 | 0.318 | 0.479 | 0.301 | 0.174 | 0.389 |
| Domain specialist | 0.656 | 0.503 | 0.583 | 0.594 | 0.540 | 0.838 | 0.619 |
| GPT-5.1 | 0.671 | 0.499 | 0.607 | 0.596 | 0.573 | 0.767 | 0.619 |
| Cascade RL unified | 0.617 | 0.482 | 0.580 | 0.538 | 0.742 | 0.803 | 0.627 |
In-domain training closes 73-122% of the baseline-to-frontier gap on the negotiation games. Transfer is sharply asymmetric: Craigslist training lifts Marketplace from 0.174 to 0.491, the reverse direction adds +0.184, while Calendar is a destructive donor at -0.060 average out-transfer, dragging Craigslist down to 0.167. Marketplace itself reaches 0.838 in-domain yet contributes almost nothing to the other five.
The transfer-aware cascade order (Calendar, CaSiNo, DnD, Craigslist, Marketplace, Job Interview) reaches 0.627 average utility, above GPT-4.1 (0.625), GPT-5.1 (0.619) and GPT-5.2 (0.613); a random order gets 0.578 and an anti-transfer order 0.562. Calendar, trained first, ends at 0.742, above every GPT baseline, lifted by later stages rather than by its own checkpoint. MOPD is cheaper: 60 optimization steps recover 92.6% of the specialists' average advantage.
The behavioral changes are more vivid than the scores. Craigslist buyers opening below the target price rise from 3% to 78%, and messages revealing that private target fall from 52.7% to 1.0%. Marketplace openings drop from 0.956 to 0.361 of the reservation price, and the 62% of opening messages that leaked it essentially disappear. Calendar schedules fewer meetings (92% down to 74%) while max-preference slots rise from 16% to 38%: fewer agreements, better ones.
The theory-of-mind experiment carries a counterintuitive result. Prompting the base model with an explicit Infer-Act-Anticipate scaffold lowers average utility from 0.454 to 0.353. Distilling the same reasoning traces raises it to 0.546 versus 0.500 for ordinary SFT. Of the two ToM skills, only next-action prediction correlates with negotiation outcomes; preference inference alone does not.
Agent builders get one conclusion: negotiating on behalf of a user is trainable into a 4B model with outcome RL plus explicit reasoning distillation, no frontier scale required. Training practitioners get two cheap signals: the cross-evaluated transfer matrix works as a curriculum signal, worth 6.5 points between the best and worst environment ordering at identical budget, and gap-weighted teacher sampling compresses nine-tenths of the consolidation gain into 60 steps.
The authors flag one themselves: the environments provide no external price information, so learned market references are partly fabricated, learned bluffing rather than factual grounding. Further caveats: opponents come from three fixed types, with no human counterparts; GPT-5.5 still leads at 0.708, so matching GPT-5 holds only for 5.1 and 5.2; per-environment utility normalization makes the cross-environment average depend on that design choice; and the tension between training agents to withhold information and assistant safety alignment goes undiscussed.