Two-Sided A/B Testing Permanently Hurts Engagement: Snap Study Reveals Isolation Costs
_reachsumit · x · 2026-08-07
On two-sided content platforms, symmetric two-sided isolation is widely used in A/B testing to remove marketplace interference by assigning matched fractions of creators and viewers to isolated submarkets. Intuition suggests that as the platform's catalog grows, the engagement cost from this thinning should fade.
However, a new paper uses extreme-value theory to show that whether this cost vanishes depends on the upper tail of match quality. For light or bounded tails, the loss disappears as the pool grows; but under heavy tails, it converges to a size-independent constant. This means expanding the candidate pool by orders of magnitude cannot asymptotically eliminate the cost.
The researchers validated this through production experiments on a platform with millions of active creators (Snap), where A/A traffic sweeps revealed a measurable engagement cost, and catalog ablation confirmed that per-viewer thinning drives the loss.
More from Research
- Training LLMs Isn't Rolling a Ball Downhill: Loss Landscape Video Reveals the Mystery of Gradient Descent — emax · 2026-08-24
- LLM Preference Tuning Fails Under Domain Shift, Study Shows — nikaletras · 2026-08-24
- Paper: Agents struggle to evolve via explicit skill libraries — rohanpaul_ai · 2026-08-24
- Francis Crick Institute Launches AI for Biological Discovery Seminar Series — MihaelaVDS · 2026-08-24
- ETH Zürich trains robot to play badminton using whole-body reinforcement learning — lukas_m_ziegler · 2026-08-24
- ParaTempo: Efficient Parallel Reasoning via Temporal Confidence — SJTU · 2026-08-24