Stripe Engineer's Test: Only 7 of 21 AI Runs Built a Working T-Shirt Shop

Stripe engineer Michelle Bu launched a Substack, "Breaking Change!", documenting how ways of working are being disruptively changed by AI. Its first post tested 7 models building and deploying a t-shirt shop with payments and fulfillment: of 21 runs, only 7 produced a fully working store. She warns that vibe-coding users may only discover problems via customer-support emails and chargebacks.

Confirmed

• Setup: 7 models, 21 runs; the task was a t-shirt shop with payments and a full order lifecycle (charging, cost of goods, fulfillment). The author has built many such "unserious" t-shirt shops over the years, seeing them as a good way to dogfood Stripe and a minimal testable unit of e-commerce
• 7 of 21 runs produced fully working stores, which the author considers a remarkable ratio
• Many failures were non-obvious bugs that fail silently in production
• No run produced a t-shirt design the author would personally wear out
• Key finding: old APIs and frameworks are "frozen" in training data; models prefer generating code directly from weights rather than consulting up-to-date docs, searching far too infrequently
• Per @bcherny, Stripe turned the experiment into the UnseriousT-ShirtShopBench benchmark: all models autonomously chose to integrate Stripe payments, and 5 of the 7 working stores were built by Claude-family models

Why it matters

• The test concretely exposes two systematic weaknesses of coding agents: stale knowledge (not checking docs, relying on old APIs baked into training data) and silent failures (bugs surfacing only in production)
• The author reflects on where dev tools should go: five years ago, "being the user" meant walking through a quickstart; today it means clearing obstacles for coding agents—completing 2FA, clicking through setup pages without APIs, coaxing models onto real APIs. She argues dev tools should be redesigned around agents that don't read documentation
• Productized as UnseriousT-ShirtShopBench, the experiment may become a public reference for end-to-end e-commerce building capability; Claude-family models' lead in working stores offers preliminary cross-model comparison data (relayed secondhand; details subject to Stripe's official release)

2026-09-15 ~ 2026-09-16 · 7 related posts

• [source] Stripe Engineer's New Substack: Can Models Build Fully Functioning T-Shirt Shops? Sort Of — hazelcough · 2026-09-15
• Stripe Engineer: A T-Shirt Shop Is a Useful Minimal Testable Unit of E-Commerce — hazelcough · 2026-09-15
• [source] 7 Models, 21 Runs: Only 7 Produced a Working T-Shirt Shop, Failures Were Silent Bugs — hazelcough · 2026-09-15
• [source] Test: 7 Models Building a T-Shirt Shop Prefer Stale Training-Data APIs Over Latest Docs — hazelcough · 2026-09-15
• Same Test: 0 of 21 Runs Produced a T-Shirt Design I Would Wear — hazelcough · 2026-09-15
• Stripe Engineer Builds a T-Shirt Factory Factory to Benchmark Dev Tools for Agents — hazelcough · 2026-09-15
• Stripe's model-run shop bench: 5 of 7 working stores built by Claude — bcherny · 2026-09-16