Qwen 3.8 27B pi adapter tested: 4x fewer tool calls, 54% faster, but no harder-task gains

Fabulous_Pollution10 · reddit · 2026-11-23

Testing the pi LoRA adapter on Qwen 3.8 27B across 10 hard SWE/ML tasks (3 runs each): the adapter teaches the model to use pi's native tools instead of falling back to bash — tool calls drop 244→62 (4x), agent time 39→18 min (-54%), pass@1 13.3%→23.3% with thinking off. With thinking on (xhigh, 81,920 output cap), turns drop 97→74 but reasoning tokens rise 57k→68k with no time savings; pass@1 33.3%→36.7%. Takeaway: the adapter improves harness efficiency, not problem-solving ceiling. Also: default 16k/32k output caps are too small — models can spend the whole response thinking and return nothing.

Original post →

More from coding & agent

coding & agent channel →