Shopify's Continual Learning Loop with PyTorch and vLLM Beats Frontier Models on Niche Tasks
PyTorch · x · 2026-10-21
At PyTorchCon 2026, Shopify Senior Staff ML Engineer Cody Mazza-Anthony presented a continual learning loop built on PyTorch and vLLM, using the Buyer Profile task as an example.
- A calibrated judge flags low-scoring outputs in production; critics diagnose each failure
- A replay with a critic hint is re-scored by the judge; accepted repairs expand the training data
- A small specialist model outperforms a frontier model on this task
- Inference cost is further optimized via prompt compression
The keynote was livestreamed, offering a reproducible playbook for production LLM quality iteration.
More from coding & agent
- How do you evaluate GenAI output quality before shipping? Devs weigh in — ssnuopi · 2026-10-21
- RunInfra launches $10/mo coding plan: fastest open models across 19 agents incl. Claude Code — ycombinator · 2026-10-21
- Matt Shumer launches an 'AI employee' you assign work to over email — mattshumer_ · 2026-10-21
- uv-ty integration: how to enable PEP 723 script type checking — charliermarsh · 2026-10-21
- Astral ships experimental uv-ty integration: uv check type-checks your project — charliermarsh · 2026-10-21
- Dev's self-built tracker claims 7.1x speedup with Claude, 2.5 work-years banked — arthurcolle · 2026-10-21