Shopify: a continual learning loop lets a small model beat a frontier model
lmoroney · x · 2026-10-21
At PyTorch Conference in San Jose, Shopify's Cody Mazza-Anthony presented a continual learning loop built with PyTorch and vLLM, using Shopify's Buyer Profile as the example: a calibrated judge flags low-scoring outputs in production, critics diagnose each failure, tasks are replayed with the critic's hint and rescored, and passing repairs join the training data. The talk showed a small specialist model outperforming a frontier model on this task. Advice: start saving every low-scoring output with a short failure note — it's the ideal starting point for your next fine-tune.
Related event: Shopify's Continuous Learning Loop Beats Frontier Models with Small Models(3 posts)→
More from coding & agent
- HeyGen Founder Joshua Xu: Agents Should Explain with Video, Not Text Reports — toolstelegraph · 2026-10-21
- Claude Code Desktop task cards now let you pick the model per task — dotey · 2026-10-21
- Slack's first employee: agentic coding is my biggest career shift since 1994 — josh_wills · 2026-10-21
- steipete: a real skills gap is opening between people with personal agents and those without — steipete · 2026-10-21
- Agents relentlessly hillclimb non-hackable metrics — including SOC 2 certification — RylanSchaeffer · 2026-10-21
- For $20, a Coding Agent Devises a Surprising Physical Strategy to Move a Box — tomssilver · 2026-10-21