Salesforce: Imitating stronger models drops weak agent performance 4-30 points, on-policy correction wins
rohanpaul_ai · x · 2026-09-22
A new Salesforce AI paper, Co-Evolving Harnesses and Models, studies how agent harnesses (system prompts, tool sets, execution hooks, context scaffolding) should be combined with lightweight fine-tuning. The authors first evolve a harness with a weaker model (Qwen3-Coder-30B-A3B etc.) across 7 enterprise agent tasks, and find that a stronger expert often uses the evolved harness even more effectively—suggesting expert supervision could close the remaining gap.
Counterintuitively, training the weaker model on the expert's full trajectories under the evolved harness backfires: performance regresses by 4–30 points on all 7 tasks (across Qwen3-Coder and Gemma 4), even though the same procedure helps under an unevolved harness. Analysis shows imitation transfers knowledge but breaks model-harness fit: the weak model adopts the expert's planning strategy without the competence to execute it.
The fix is on-policy expert correction: the weaker model doesn't need to think like Gemini—it needs Gemini to correct the specific places where its own approach fails, which substantially outperforms full imitation.
Related event: Salesforce: Imitating Stronger Models' Trajectories Hurts Agent Performance(2 posts)→
More from coding & agent
- Extracting confidence probabilities from small local LLMs via logits — OneFanFare · 2026-09-22
- PE fund rolled coding agents across portfolio: only 1 of 5 workflows worked — alex_verem · 2026-09-22
- Building a racing game with Claude: 'AI dev is a settings simulator' — Grobot93 · 2026-09-22
- capy beats a major coding harness at half the cost and half the time, eval finds — garrytan · 2026-09-22
- XGrammar-2 ships strict tool calling for agents, adopted by xAI, DeepSeek and vLLM — vllm_project · 2026-09-22
- MechFaber: Claude Code designs a 99-part quadruped with firmware co-simulated in Renode and MuJoCo — SpeedyBrowser45 · 2026-09-22