Hands-on ablations show Pi harness tweaks fail to lift Qwen scores on Terminal-Bench Lite

User yb2698 ran a set of systematic ablation experiments based on the Pi 0.84.4 coding harness to answer the question: can a custom agent harness tailored for Qwen models improve benchmark scores? The conclusion was surprising: almost nothing improved the pass rate, and some changes even hurt performance.

Confirmed

• Starting from Pi 0.84.4, the experiments only adjusted pluggable Pi extensions and surface-level settings (system prompts, tool definitions, environment output handling, etc.), deliberately keeping components easy to modify and pluggable in the future for community distribution.
• Models tested: Qwen3.5-9B (24 easy-subset problems from Terminal-Bench Lite) and Qwen3.8-27B (48 easy+medium problems), with the Terminal-Bench Lite benchmark from open-thoughts.
• Five variants were ablated: verify-nudge (having the model self-check before submitting), lean-prompt (a shorter system prompt), qwen-code-tools (renaming tools to Qwen Code naming conventions), and more — none of the Qwen-specific changes delivered the expected gains.
• After switching to the full Qwen Code interface, the smaller model's score actually dropped by 5.3 points.

Why it matters

• The experiments pour cold water on the common assumption that tailoring a harness for a specific model can squeeze out higher scores, suggesting that surface-level harness customization may yield very limited gains on benchmarks like Terminal-Bench Lite.
• The author kept the changes pluggable and easy for the community to distribute, making replication and further experimentation easier; whether the results hold on other models or benchmarks remains to be seen.

2026-10-12 ~ 2026-10-12 · 5 related posts

• [source] Tuning the Pi agent harness for Qwen models: every tweak fails to move Terminal-Bench Lite pass rates — yb2698 · 2026-10-12
• Experiment details: Qwen3.5-9B and Qwen3.8-27B on Terminal-Bench Lite from open-thoughts — yb2698 · 2026-10-12
• [source] Pi 0.84.4 harness experiments stay within pluggable extensions and surface-level settings — yb2698 · 2026-10-12
• Pi harness ablations: five variants, Qwen-specific tweaks gave no expected gains — yb2698 · 2026-10-12
• [source] Pi harness ablations: full Qwen Code interface hurt small models (-5.3 pts), none improved — yb2698 · 2026-10-12