ActObs: Supervising Observations During SFT Boosts Agent RL by up to 4.2pp
kastnerkyle · x · 2026-09-19
The paper "Don't Mask the Environment" introduces ActObs: instead of applying SFT loss only to agent action tokens, it also supervises the environment observation tokens already present in each trajectory, teaching the policy to predict action consequences with no extra data, parameters, sequence tokens, or forward passes.
Key results:
- Both methods perform similarly after SFT but diverge after GRPO
- On Qwen3-4B, ActObs-initialized GRPO achieves higher pass@k at every sampling budget on Terminal-Bench 2.0
- On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4pp at pass@16) and solves more distinct tasks
- Cross-domain code editing on aider-polyglot: +4.2pp pass@1 at 4B on unseen tasks
Mechanistic analysis: ActObs retains more entropy during RL with less policy movement, keeping the final policy closer to its SFT initialization; the authors trace the divergence to action/observation gradients becoming orthogonal early in SFT.
Related event: ActObs: Supervising Environment Tokens in SFT Boosts Agent RL(2 posts)→
More from coding & agent
- User hooks Grok into Cursor and gets addicted: frontend building on autopilot — Kyrannio · 2026-09-19
- New Paper Empirically Studies Harness Design for Coding Agents — _akhaliq · 2026-09-19
- Yacine: Even Top Models Make Bad Decisions—I Still Read Every Line of AI Code — yacineMTB · 2026-09-19
- TypeSafe AI's decision model Jev lands on Merge Gateway with free output tokens — shensi · 2026-09-19
- Jev Directory indexes 50 runnable evals and 722 community builds, agent-ready — thisiskp_ · 2026-09-19
- Frontier models for planning, cheap models like GLM and DeepSeek for execution — TheZachMueller · 2026-09-19