Salesforce: Imitating stronger models drops weak agent performance 4-30 points, on-policy correction wins
rohanpaul_ai · x · 2026-09-22
A new Salesforce AI paper, Co-Evolving Harnesses and Models, studies how agent harnesses (system prompts, tool sets, execution hooks, context scaffolding) should be combined with lightweight fine-tuning. The authors first evolve a harness with a weaker model (Qwen3-Coder-30B-A3B etc.) across 7 enterprise agent tasks, and find that a stronger expert often uses the evolved harness even more effectively—suggesting expert supervision could close the remaining gap.
Counterintuitively, training the weaker model on the expert's full trajectories under the evolved harness backfires: performance regresses by 4–30 points on all 7 tasks (across Qwen3-Coder and Gemma 4), even though the same procedure helps under an unevolved harness. Analysis shows imitation transfers knowledge but breaks model-harness fit: the weak model adopts the expert's planning strategy without the competence to execute it.
The fix is on-policy expert correction: the weaker model doesn't need to think like Gemini—it needs Gemini to correct the specific places where its own approach fails, which substantially outperforms full imitation.
Related event: Salesforce: Imitating Stronger Models' Trajectories Hurts Agent Performance(2 posts)→
More from coding & agent
- Photoroom reclaims ~1,000 idle GPU hours a week with SkyPilot across Slurm and Kubernetes — skypilot_org · 2026-11-12
- Google launches Gemini Decisions API: scores 1,000 questions per pass, output tokens free — dean_rie · 2026-11-12
- Zed v1.28 adds background auto-fetch for current and default branch remotes — dean_rie · 2026-11-12
- Indie Britpop AI music video built with Nano Banana Pro, Kling, and free lip-sync via WAN2GP + LTX 2.5 — cocktailpeanut · 2026-11-12
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Engineer Curates 15 AI Agent Design Patterns Across Single-Agent, Multi-Agent and Feedback Tiers — MaryamMiradi · 2026-11-12