Pi harness ablations: full Qwen Code interface hurt small models (-5.3 pts), none improved
yb2698 · x · 2026-10-12
Ablation experiments on the Pi 0.84.4 coding harness tested five variants: verify-nudge (self-check before submitting), lean-prompt, qwen-code-tools (tool names matching Qwen Code), qwen-sampling (model card sampling settings), and qwen-code-full (the whole Qwen Code interface inside Pi). The chart shows that on Qwen3.5-9B and Qwen3.8-27B every variant's 95% CI crosses zero—nothing meaningfully improved. Surprisingly, the full qwen-code setup lowered the small model's score (-5.3 pts on 9B), contrary to the expectation that Qwen models would be tuned to their own harness. Changes are built as pluggable Pi extensions for the community.
More from coding & agent
- PageIndex SDK v0.4 claims tree indexing now cheaper than vector indexing by ~15% — solyarisoftware · 2026-10-13
- Economist: agentic coding lets you write your own Fortran optimizers, not just ask for Stata packages — Apoorva__Lal · 2026-10-13
- Tencent open-sources Agent Memory: local long-term memory, 61% fewer tokens, 48%→76% accuracy — solyarisoftware · 2026-10-13
- Testing AI agents with simulations: replay real traces, 50 curated sims beat 1000 random ones — HamelHusain · 2026-10-13
- How humans and Copilot agents split the work to build a GitHub Copilot beginner course — DanWahlin · 2026-10-13
- Engineers distill their AGI debate into an essay on self-verifying, continuously improving agents — dotey · 2026-10-13