USTC and Alibaba Propose PARPO: Personalized Agentic RL That Optimizes Preference, Not Just Correctness
量子位 · wechat · 2026-09-13
Researchers from USTC and Alibaba propose PARPO (Personalized Anchor Reward-Decoupled Policy Optimization), formally bringing personalization into the training objective of Agentic RL.
Problem
- In e-commerce assistance, trip planning and scheduling, the optimal trajectory differs per user; optimizing generic task success makes models learn the "average user's preference", and stuffing behavior history into prompts confuses conformity and popularity with true preference
- Three challenges: unclear personalized reward semantics, mixed preference signals, flat memory
Method
- PSGM: a heterogeneous graph memory (User/Skill/Tool/Scenario/Trajectory nodes) retrieved via skill-user expansion with relevance, complementary boost and conflict penalty
- Two-stage preference reward model: multi-view profiles for cold start, then LightGCN on the user-item graph separating orthogonal "interest" and "conformity" branches
- Optimizer: dual advantage tracks for generic vs personalized quality, with EMA user-level anchors so rewards compare against the user's own historical center
Results
- On ETAPP, ETAPP-Hard and SJAgent with Qwen3-4B/8B: Judge 0.8333 (ETAPP-8B), Reward 0.8270 (SJAgent), beating ReAct, GRPO, DAPO, GSPO, GiGPO, MemRL, SkillRL
- Ranked first in blind tests by 15 human experts and 4 LLM judges, with the largest gains on User Relevance
- Ablations: removing skill memory drops Judge from 0.7708 to 0.7006; user anchors and interest/conformity branches all matter
Limitations: limited human blind-test scale; the interest-conformity decoupling is operational, not strict causal separation.
More from Research
- AIR: Effective-Learning-Rate Scheduler Beats Cosine on 128B-Token MoE Pretraining — stochasticchasm · 2026-10-30
- NVIDIA researcher: train robot policies with privileged sim info, improving Pi 0.5 — wellecks · 2026-10-30
- DeepMind's f_charton shares his take on the current state of AI and mathematics in Sorbonne talk — f_charton · 2026-10-30
- Gym-Anything launches: 200 robot RL environments across 25 simulations — wellecks · 2026-10-30
- What coding agents actually see when searching MCP tools — and why discoverability is bad — _babush_ · 2026-10-30
- Interpretability goes to production: diagnosing real-world LLM failures via internal states — burkov · 2026-10-30