The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
Yuanyuan Shen, Yiren Yan, Wenjie Li, Chunhui Zhu
cs.IR, cs.LG, cs.SI, stat.ME
2026-08-05
Two-sided isolation A/B tests thin each viewer's catalog and cut engagement even with no treatment; under heavy-tailed match quality the loss never vanishes as the platform grows.
Short-video platforms are two-sided markets: creators supply content, viewers consume it, and a recommender allocates a finite pool of attention. Testing a creator-side change (boosting some creator segment, changing supply incentives) with a naive A/B test breaks SUTVA, the assumption that one unit's treatment does not affect another. Treated and control content compete for the same viewers, so the control arm is contaminated and the measured effect drifts from what a full launch would produce.
The accepted fix is symmetric two-sided isolation: randomize both creators and viewers at a common fraction p into isolated submarkets, so treatment-arm viewers only see treatment-arm creators. Google's user–corpus co-diversion and LinkedIn's budget-split belong to the same family. It cleanly removes cross-arm interference and is the workhorse design for creator-side and cold-start experiments.
Its cost has been largely overlooked. Isolation thins each viewer's candidate catalog, and the common intuition is that this matters less as the platform grows: a few percent of a vast catalog is still vast. The paper argues that this intuition is often wrong.
The core tool is order statistics. Each candidate's relevance to a viewer is a random variable u, and the feed shows the best match, the maximum order statistic. Thinning the pool from n to pn lowers that maximum, and the loss is L(p) = 1 − E[u(pn)] / E[u(n)].
Extreme-value theory classifies the upper tail of match quality into three domains, each with its own loss law:
The sharp result is a dichotomy: whether the cost fades with platform size depends entirely on the tail of match quality. Under heavy tails it is a size-independent constant. The authors treat the heavy-tail law as a conservative benchmark, the worst of the three and the hardest to amortize with scale.
Three traffic floors follow: a supply floor (keep enough above-bar candidates per request), a tolerance floor pε (the smallest p that holds loss under ε), and a power floor (smaller p means fewer samples and a larger minimum detectable effect, MDE ∝ σ(p)/√(p|V|)). Take their max, stress-test against the heavy-tail benchmark, and you have a preflight procedure for launch.
The production experiments run on a short-video platform (all authors are at Snap Inc., i.e. Snapchat).
A/A sweep: viewers held at 10%, creator catalog thinned from 70% to 10% (with a 2% stress cell). This is a pure A/A; both arms run identical policy, so the drops below are artifacts of the design, not treatment effects.
| Metric | 10% creators | 2% creators |
| Story views | −1.24% | −7.61% |
| View time | −12.27% | −29.76% |
| 10s+ views | −16.65% | −38.48% |
| Story completion | −19.03% | −41.82% |
| Favorite | −18.20% | −38.24% |
| Send | −16.92% | −38.93% |
All contrasts are significant at p < 10⁻⁴. The loss deepens with engagement depth: shallow metrics (views) lose little, deep ones (completion, favorite, send) lose the most. It concentrates on exploration content, where whole-feed effects are an order of magnitude smaller, but exploration metrics are exactly what creator-side experiments measure. The artifact lands on the estimand itself, at a size that rivals real treatment effects.
A second, independent experiment is a one-sided catalog ablation (viewer-randomized, 5% or 10% random per-viewer catalog drops). The response is strongly sub-proportional: a 10% catalog drop costs about 1% engagement. This confirms per-viewer thinning as a cause.
The cleanest step is cross-experiment parameter transfer. Tail indices calibrated on the small symmetric study (view time α̂ = 14.9, completion 9.2, send 10.5), plugged into the heavy-tail law, predict the full-catalog ablation loss, and every prediction lands inside the observed 95% interval. A parameter fit on a small pool forecasts an effect at far larger scale.
The creator side is the dual: in the 10%×10% cell creators receive 32.1% less exploration view time, in 2%×2% they receive 46.3% less.
For anyone running recommender or growth experiments, the message is that a design assumed to be free carries a measurable, budgetable cost. Teams run many concurrent symmetric experiments, partitioning creators and viewers into isolated cells, which pushes each cell to a small p; the cost is paid on every creator-facing launch and most heavily in the smallest cells.
The paper builds no new system. It gives a preflight procedure: calibrate the loss on a small slice, size p accordingly, and switch to a fallback design when the predicted cost exceeds tolerance. The numbers are blunt: at 5% tolerance, the view-time index (α̂ = 14.9) needs 47% of each population per arm to support two arms, and the completion index (α̂ = 9.2) admits no valid design at all; even at 10% tolerance you need 21 to 38%. Many small-traffic symmetric experiments are infeasible by design.
The authors' own caveats: the model assumes i.i.d. match utilities with top-item selection, and correlated candidates could change the constants or even the scaling; it measures A/A artifacts directly and reasons about treatment-plus-interaction only indirectly; α̂ is a calibration for the operating range, not a platform constant, and the production contrasts do not identify the true max-domain; serving-side effects (latency, scan depth) are not separated out, though the dose-capped ablation argues they are not dominant; the estimand is the content-side cost, with creator retention and the causal decomposition left to a companion study; the fixed-viewer contrast is a single 10%-vs-70% point plus a 2% stress test, and a denser sweep would sharpen the scaling law but is constrained by finite supply.
One more concern. The whole conclusion hinges on match quality being heavy-tailed, and the authors concede that power laws and lognormals are notoriously hard to distinguish in practice. The tail class cannot be read off directly from production, so they lean on the heavy tail as a conservative upper bound. If the true distribution actually sits in the Gumbel domain, the scale-amortization picture is friendlier than the paper claims, but the headline size-independent claim then falls apart.