Salesforce's Prospective Hindsight uses prediction-reality gaps to self-calibrate RL agents
Salesforce · hf · 2026-10-14
Salesforce AI Research introduces Prospective Hindsight (PH), a self-calibrating RL training principle for long-horizon agents.
- Problem: Retrospective RL assigns credit only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient.
- Method: PH adds a signal from the gap between the agent's prospective prediction (before feedback) and retrospective evaluation (after feedback). This per-rollout surprise identifies where the self-model is most inaccurate and amplifies those samples via a stop-gradient surprise-weighted advantage. Predictor and policy share parameters and co-evolve.
- Theory: Minimizing the surprise residual provides a descent pathway on miscalibration rate, so calibration emerges as a byproduct of optimization.
- Results: On single-turn verifiable tasks and a multi-turn personal-agent task (GRPO, on-policy distillation, and combinations), PH improves both performance and calibration across model scales. Notably, miscalibration flips from overconfident failures (single-turn) to underconfident successes (multi-turn), yet the same principle fixes both.
More from Research
- Embedding arithmetic demo: cartoon − animation − comic = diagram — mayfer · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Word Arithmetic on Embeddings: "Bathroom Minus Toilet Equals Kitchen" Beats SAE Probing — mayfer · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23