Pi harness adapter via RL: SWE-bench Pro 29.4%→62.7% with 55% fewer tool calls
yb2698 · x · 2026-11-17
The author shares the recipe behind a Pi harness adapter: every RL rollout is a real Pi session in an HF sandbox, with no agent-loop reimplementation inside the trainer — using TRL's AsyncGRPOTrainer + OpenEnv, with every tool call analyzed by safeagents. Results (chart): SWE-bench Pro (hard 51) pass@1 jumps from 29.4% to 62.7% and Terminal-Bench 2.1 from 39.5% to 55.8%, with 55%/62% fewer tool calls per task. A task where the base model hit the 100-step limit is solved in 8 tool calls; 16 newly solved TB-2.1 tasks and 18 on SWE-Bench Pro.
Related event: Harness Adapters Launch: LoRA Boosts Qwen3.8-27B Coding Agent Performance(3 posts)→
More from coding & agent
- Mnemos x NousResearch launches agent memory engine claiming 2x gains over Claude Code — RileyRalmuto · 2026-11-17
- Designer: beating AI at design isn't talent, it's a process — refs, direction, components — damienghader · 2026-11-17
- How Revid doubled its ChatGPT-sent signups from 355 to 781 per week in 10 weeks — tibo_maker · 2026-11-17
- FormMCP: one MCP connection to a family of desktop apps, tools loaded on demand — nodemode_ai · 2026-11-17
- Pi harness adapter ships as a LoRA runnable on vLLM, llama.cpp, and MLX — yb2698 · 2026-11-17
- Real SWE-Bench Pro task: base model hits 100-step limit, adapter solves it in 8 calls — yb2698 · 2026-11-17