Test: 7 Models Building a T-Shirt Shop Prefer Stale Training-Data APIs Over Latest Docs
hazelcough · x · 2026-09-15
Stripe's Michelle Bu asked 7 models to build and deploy a novel t-shirt shop with payments and fulfillment across 21 runs.
Key finding: older API and framework versions are frozen in amber in training data, and models preferred to manifest code from their weights rather than check the latest docs. Search frequency roughly tracked familiarity: across 21 runs, React or Next.js was searched only once, Stripe 15 times, and the t-shirt printing API Prodigi 62 times.
Related event: Stripe Engineer's Test: Only 7 of 21 AI Runs Built a Working T-Shirt Shop(7 posts)→
More from coding & agent
- One stale date silently broke a WhatsApp reply agent: 19 messages unanswered — petrusenko_max · 2026-11-02
- The 10X engineer is now just a SKILLS.md: 6 workflow rules for AI coding agents — mdancho84 · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Sketchy v0.1.6: AI diagrams use 29% fewer output tokens than plain text — Nkgsporty · 2026-11-02