USTC and Alibaba Propose PARPO: Personalized Agentic RL That Optimizes Preference, Not Just Correctness
量子位 · wechat · 2026-09-13
Researchers from USTC and Alibaba propose PARPO (Personalized Anchor Reward-Decoupled Policy Optimization), formally bringing personalization into the training objective of Agentic RL.
Problem
- In e-commerce assistance, trip planning and scheduling, the optimal trajectory differs per user; optimizing generic task success makes models learn the "average user's preference", and stuffing behavior history into prompts confuses conformity and popularity with true preference
- Three challenges: unclear personalized reward semantics, mixed preference signals, flat memory
Method
- PSGM: a heterogeneous graph memory (User/Skill/Tool/Scenario/Trajectory nodes) retrieved via skill-user expansion with relevance, complementary boost and conflict penalty
- Two-stage preference reward model: multi-view profiles for cold start, then LightGCN on the user-item graph separating orthogonal "interest" and "conformity" branches
- Optimizer: dual advantage tracks for generic vs personalized quality, with EMA user-level anchors so rewards compare against the user's own historical center
Results
- On ETAPP, ETAPP-Hard and SJAgent with Qwen3-4B/8B: Judge 0.8333 (ETAPP-8B), Reward 0.8270 (SJAgent), beating ReAct, GRPO, DAPO, GSPO, GiGPO, MemRL, SkillRL
- Ranked first in blind tests by 15 human experts and 4 LLM judges, with the largest gains on User Relevance
- Ablations: removing skill memory drops Judge from 0.7708 to 0.7006; user anchors and interest/conformity branches all matter
Limitations: limited human blind-test scale; the interest-conformity decoupling is operational, not strict causal separation.
More from Research
- Proxy Policy Steering adapts frozen VLA models to new tasks at inference time — weichiuma · 2026-09-14
- New research: standard SGD matches AdamW for LLM RL training, with far less memory overhead — zhaoran_wang · 2026-09-14
- RSI work separates practical harness self-improvement from unproven intelligence explosion — arthurcolle · 2026-09-14
- Amazon Proposes Query-Aware Index Pruning to Optimize Retrieval Under Budget Constraints — _reachsumit · 2026-09-14
- New Paper Finds Retrieval Signals Give No Reliable Routing Gain in Adaptive Multimodal RAG — _reachsumit · 2026-09-14
- Google: Graph RAG Cuts API Hallucination Rate from 56.4% to 16.2% in Java-to-Python Migration — _reachsumit · 2026-09-14