Pi harness adapter via RL: SWE-bench Pro 29.4%→62.7% with 55% fewer tool calls

yb2698 · x · 2026-11-17

The author shares the recipe behind a Pi harness adapter: every RL rollout is a real Pi session in an HF sandbox, with no agent-loop reimplementation inside the trainer — using TRL's AsyncGRPOTrainer + OpenEnv, with every tool call analyzed by safeagents. Results (chart): SWE-bench Pro (hard 51) pass@1 jumps from 29.4% to 62.7% and Terminal-Bench 2.1 from 39.5% to 55.8%, with 55%/62% fewer tool calls per task. A task where the base model hit the 100-step limit is solved in 8 tool calls; 16 newly solved TB-2.1 tasks and 18 on SWE-Bench Pro.

Related event: Harness Adapters Launch: LoRA Boosts Qwen3.8-27B Coding Agent Performance(3 posts)→

Original post →

More from coding & agent

coding & agent channel →