Amazon's ActObs: supervising observation tokens in SFT boosts agent RL exploration and pass@k
amazon · hf · 2026-09-18
Amazon researchers propose ActObs, which supervises environment observation tokens already present in agent trajectories during SFT, not just action tokens — teaching the policy to predict action consequences without extra data, parameters, or compute.
Key results:
- Methods look similar after SFT but diverge sharply after GRPO
- On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0
- On Qwen3-8B, it trades some pass@1 reliability for +3.4pp pass@16 and solves more distinct tasks
- Transfers cross-domain to unseen aider-polyglot code editing: +4.2pp pass@1 at 4B
Mechanism: action and observation gradients rapidly become orthogonal under action-only training, which degrades environment prediction below the base model; joint supervision preserves consequence prediction, keeps more entropy during RL, and better prepares the policy for exploration.
More from Research
- Paper: Suppressing LLM self-attributions shifts reported values away from human norms — coherence · 2026-09-18
- New weekly newsletter filters AI-for-Science papers down to 3-5 transferable ideas — bravo_abad · 2026-09-18
- New Multiscale Emergence Methods Applied to EEG Across Conscious States, Paper Published — anilkseth · 2026-09-18
- AndroidLife real-phone benchmark: Qwen3.8-27B fails 43% of 60 daily tasks — East-Muffin-6472 · 2026-09-18
- WeirdML v3 launches: agentic benchmark with 11 hand-made ML tasks — scaling01 · 2026-09-18
- Stanford researchers launch CoPaper, an AI co-authoring platform already behind published journal papers — JeremyNguyenPhD · 2026-09-18