A/B testing lecture: Bing's $100M experiment and four honesty rules for not fooling yourself
richardreis · x · 2026-11-04
richardreis releases "Lecture 13: How To Do A/B Testing (Without Fooling Yourself)", opening with his own failed Mental Math Pro redesign and the hook "two identical pages, an 18% improvement" to show how noise fools teams.
Key points:
- Draws on Ronny Kohavi's book and Bing's $100 million A/B test story
- Four steps: one factor at a time, write an experiment record, never peek or stop early, always check the split first
- Validate with p-values, confidence intervals and A/A tests; size samples with Evan Miller's calculator
- Ends with "four honesty rules" and homework for your first test
A directly actionable guide for indie hackers and growth practitioners.
Related event: Indie Developer Shares Honest Lessons on Doing A/B Testing Right(2 posts)→