Salesforce: Fine-tuning a weak model to copy Gemini drops success 15%; correcting its own failures works
rohanpaul_ai · x · 2026-09-22
Salesforce AI research finds that once an agent's prompts, tools, and workflow are tuned around a weaker model (Qwen3-Coder-30B-A3B), fine-tuning it to copy a stronger Gemini model makes things worse.
- Across 7 enterprise tasks, tuning the agent setup around Qwen raised average success from 29.2% to 78.0%; Gemini reached 93.6% in the same setup.
- Fine-tuning Qwen on complete Gemini runs dropped success to 63.1%, falling on all 7 tasks — Qwen copied Gemini's planning style, which no longer fit the setup built around its original behavior.
- Keeping Qwen's failed runs and letting Gemini correct only the wrong step pushed success to 79.7% without losing earlier gains.
Lesson: tune the model without breaking the agent setup already working around it.
Related event: Salesforce: Imitating Stronger Models' Trajectories Hurts Agent Performance(2 posts)→
More from coding & agent
- YC-backed Roma turns your to-do list into an agent command center across Gmail and Notion — testingcatalog · 2026-11-12
- Boubou: open-source CLI turns your laptop into a fully local voice AI hub — alexcovo_eth · 2026-11-12
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12