Stripe Engineer Tests 7 Models Building a T-Shirt Store: Barely Usable

Stripe engineer Michelle Bu launched a Substack newsletter, "Breaking Change!," documenting how technology work is undergoing "breaking changes" in the AI era. The first post reports a hands-on experiment: having 7 models build and deploy a brand-new T-shirt store with payments and fulfillment, 21 runs total, of which only 7 produced fully working stores. She argues that users building sites via "vibe coding" may not discover problems until customer-support emails and chargeback notices arrive.

Confirmed

• Experimental setup: 7 models, 21 runs, task was building a T-shirt store with payments and the full order lifecycle (charging, cost of goods, fulfillment). The author has built many such "non-serious" T-shirt stores over the years, seeing them as a great way to dogfood Stripe and the minimal testable unit of e-commerce
• Of the 21 runs, 7 produced fully working stores — a ratio the author considers quite impressive
• Many failures were non-obvious bugs that fail silently in production
• Not a single run produced a T-shirt design the author herself would wear out the door
• Key finding: old API versions and frameworks are "frozen" in training data; models prefer generating code straight from weights rather than consulting up-to-date docs, and search far too infrequently

Why it matters

• The experiment concretely exposes two systematic weaknesses of coding agents: stale knowledge (not checking docs, defaulting to old APIs from training data) and silent failures (bugs go unnoticed and only surface in production)
• The author reflects on how developer tools should evolve: five years ago, "putting yourself in the user's shoes" meant following the quickstart; today it means clearing obstacles for coding agents — completing 2FA, clicking through settings pages with no API, guiding the model to use real APIs. She argues developer tools should be redesigned around "agents that don't read the docs"

2026-09-15 ~ 2026-09-15 · 6 related posts

• Stripe Engineer's New Substack: Can Models Build Fully Functioning T-Shirt Shops? Sort Of — hazelcough · 2026-09-15
• Stripe Engineer: A T-Shirt Shop Is a Useful Minimal Testable Unit of E-Commerce — hazelcough · 2026-09-15
• [source] 7 Models, 21 Runs: Only 7 Produced a Working T-Shirt Shop, Failures Were Silent Bugs — hazelcough · 2026-09-15
• [source] Test: 7 Models Building a T-Shirt Shop Prefer Stale Training-Data APIs Over Latest Docs — hazelcough · 2026-09-15
• Same Test: 0 of 21 Runs Produced a T-Shirt Design I Would Wear — hazelcough · 2026-09-15
• [source] Stripe Engineer Builds a T-Shirt Factory Factory to Benchmark Dev Tools for Agents — hazelcough · 2026-09-15