Test: 7 Models Building a T-Shirt Shop Prefer Stale Training-Data APIs Over Latest Docs
hazelcough · x · 2026-09-15
Stripe's Michelle Bu asked 7 models to build and deploy a novel t-shirt shop with payments and fulfillment across 21 runs.
Key finding: older API and framework versions are frozen in amber in training data, and models preferred to manifest code from their weights rather than check the latest docs. Search frequency roughly tracked familiarity: across 21 runs, React or Next.js was searched only once, Stripe 15 times, and the t-shirt printing API Prodigi 62 times.
Related event: Stripe engineer tests 7 models building t-shirt stores: 7 of 21 runs usable(6 posts)→
More from coding & agent
- MiniCPM5-2B runs fully in-browser via WebGPU with agent tooling — victor · 2026-09-16
- Open-Source Agentic AI Security Lab: 9 Scenarios Breaking Agent Authorization — Rewanth_Tammana · 2026-09-16
- Models don't have agency, systems do: the one-token trick that seeds tool use — sethjuarez · 2026-09-16
- Agent Name Service Gives AI Agents Verifiable Identities via SCITT Receipts and OAuth — someone_somewhere_9 · 2026-09-16
- Dev ships multi-tenant MCP server for maginary, an OpenRouter-like multimedia generator — lutian · 2026-09-16
- Lessons from building agentic dev workflows: let software own the control loop — mostly_deterministic · 2026-09-16