Pi harness ablations: five variants, Qwen-specific tweaks gave no expected gains
yb2698 · x · 2026-10-12
Original write-up of the ablation study: starting from Pi 0.84.4, the author only tweaked pluggable Pi extensions or settings like system prompt, tool definitions and env output processing, testing five variants—verify-nudge, lean-prompt, qwen-code-tools, qwen-sampling, and qwen-code-full. The author expected Qwen-specific changes to help most, assuming Qwen models were heavily trained on their own harness; results showed no significant gains, with the full Qwen Code interface hurting the small model. Changes are kept simple and pluggable for release as Pi extensions.
More from coding & agent
- Meta launches Muse Wonder, its largest frontier model built for agents — ZeroStateReflex · 2026-10-13
- HeyGen launches Lessons: turn class notes into videos via MCP — HeyGen · 2026-10-13
- PageIndex SDK v0.4 claims tree indexing now cheaper than vector indexing by ~15% — solyarisoftware · 2026-10-13
- Economist: agentic coding lets you write your own Fortran optimizers, not just ask for Stata packages — Apoorva__Lal · 2026-10-13
- Tencent open-sources Agent Memory: local long-term memory, 61% fewer tokens, 48%→76% accuracy — solyarisoftware · 2026-10-13
- Testing AI agents with simulations: replay real traces, 50 curated sims beat 1000 random ones — HamelHusain · 2026-10-13