Tuning the Pi agent harness for Qwen models: every tweak fails to move Terminal-Bench Lite pass rates
yb2698 · x · 2026-10-12
An experiment optimizing the Pi agent harness for the Qwen model family found that almost nothing improves Terminal-Bench Lite pass rates.
- Tested on Qwen3.5-9B (24 easy tasks, Pi baseline 54%) and Qwen3.8-27B (48 easy+medium tasks, baseline 77%) using OpenThoughts TB Lite, a cheaper proxy for Terminal-Bench 2.
- Tweaks included a verify-nudge, leaner system prompts, renaming tools to Qwen Code's names, model-card sampling settings, and even porting Qwen Code's full interface into Pi.
- All 95% CIs crossed zero (e.g. verify-nudge +2.1/+4.6 pts; qwen-code-full -5.3/-0.1). The harness components were made pluggable Pi extensions and released to the community.
More from coding & agent
- PageIndex SDK v0.4 claims tree indexing now cheaper than vector indexing by ~15% — solyarisoftware · 2026-10-13
- Economist: agentic coding lets you write your own Fortran optimizers, not just ask for Stata packages — Apoorva__Lal · 2026-10-13
- Tencent open-sources Agent Memory: local long-term memory, 61% fewer tokens, 48%→76% accuracy — solyarisoftware · 2026-10-13
- Testing AI agents with simulations: replay real traces, 50 curated sims beat 1000 random ones — HamelHusain · 2026-10-13
- How humans and Copilot agents split the work to build a GitHub Copilot beginner course — DanWahlin · 2026-10-13
- Engineers distill their AGI debate into an essay on self-verifying, continuously improving agents — dotey · 2026-10-13