Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
Rohith Pasula, Johari Dramiga, Viraj Shah, Suzan Halim, Tyler Dooskin
cs.HC
2026-11-19
UX-1 simulated 1,000 shoppers per arm and classified 267 of 318 historical A/B tests (84.0%), versus 61.6% for Qwen3 Next 80B; 118 of 126 variants it favored won live (93.7%).
Experimentation teams have more page ideas than live traffic. A weak variant spends visitors another idea could have used. A strong variant left on the shelf is a gain nobody measures. Squoosh turns the choice into a three-class forecast: ignore the recorded result and call a finished A/B test as an original win, a variant win, or inconclusive. An A/B test splits real users across two page versions and scores them on a metric fixed in advance.
The corpus is 412 private partner experiments. Rule SQ-DURATION-2026-08 was locked before scoring and keeps only tests that ran longer than 21 days, which leaves 318. The other 94 are outside the denominator. Labels use the prespecified primary conversion metric and the two-sided 95% confidence interval on the variant-minus-original effect. The interval lies entirely above zero for a variant win (137), entirely below zero for an original win (113), and covers zero for an inconclusive result (68). Variant wins are the largest class, so always picking the variant scores 137 of 318, or 43.08%.
The scored systems are Squoosh UX-1, Gemini 3.5 Flash, Qwen3 Next 80B, and GPT-OSS 120B, plus three constant rules that assign every test the same class. UX-1 fine-tunes an open-source base model on browsing data Squoosh collected from people. The target is the fork in a session: whether a person finishes a listing, opens a product page, or leaves before converting.
Each comparison puts 1,000 simulated shoppers on the original and 1,000 on the variant, 2,000 sessions, with the same device and conversion goal. The model returns one class, a signed relative lift, and a short note. Relative lift is the percent change in conversion rate from the original page to the variant.
Pages are frozen artifacts: a rendered DOM snapshot and a 1280 by 720 screenshot run through visual adapter v1.2.4. The DOM is the browser's structured page, with text and element hierarchy, not just pixels. Anonymous handles hide which side is the original. Conversion counts, observed effects, and the recorded label are withheld at inference. The model can see planned exposure, allocation, the analysis rule, and historical baselines. Order is randomized and also reversed. Each system runs each order three times with independent execution state, 7,632 runs in total. The primary score is the first run of the prespecified primary order. An invalid forecast scores zero and stays in the 318.
Inconclusive means the human experiment's interval covers zero. It is a different quantity from the spread across simulated shoppers. Systems are compared at their own budgets, so the design does not separate out fine-tuning.
| System | Correct/318 | Accuracy | Balanced accuracy | Order agreement |
| Squoosh UX-1 | 267 | 83.96% | 83.40% | 94.65% |
| Qwen3 Next 80B | 196 | 61.64% | 60.25% | 83.02% |
| GPT-OSS 120B | 187 | 58.81% | 57.11% | 79.25% |
| Gemini 3.5 Flash | 162 | 50.94% | 50.07% | 53.14% |
| Always variant | 137 | 43.08% | 33.33% | not scored |
| Always original | 113 | 35.53% | 33.33% | not scored |
| Always inconclusive | 68 | 21.38% | 33.33% | not scored |
Balanced accuracy is the unweighted mean of the three recalls. UX-1 gets 94/113 original wins (83.19%), 118/137 variant wins (86.13%), and 55/68 inconclusive tests (80.88%). It is right on 71 more tests than Qwen3 Next 80B, a gap of 22.33 percentage points. Paired McNemar tests, which compare two classifiers' errors on the same items, give p < 0.001 against each general-purpose system. The Wilson 95% interval, a confidence interval for a proportion, is 79.53% to 87.59% when each experiment is treated as independent.
On the 250 tests with a directional winner, UX-1 is correct on 212 (84.80%). Always-variant scores 54.80% there. The 51 errors split into 10 flipped winners, 28 inconclusive calls on decided tests, and 13 directional calls on inconclusive tests. UX-1 predicted inconclusive 83 times and was right 55 times, precision 66.27%. It named a winner on 235 of 318 tests (73.9%), and 212 of those 235 matched the record (90.2%).
UX-1 favored the variant on 126 tests (39.6%). Of those, 118 won live (93.7%), four lost, and four were inconclusive. That shortlist holds 118 of 137 variant wins (86.1%). The 19 winners left out are 6 of the 109 tests where UX-1 favored the original and 13 of the 83 it called inconclusive. In one popup experiment it favored the variant, and the live test favored the original page.
Lift is a separate score. On the 289 tests where all four systems returned a number, 90.88% of the corpus, mean absolute error on signed relative lift is 1.43 percentage points for UX-1, 3.27 for Gemini 3.5 Flash, 3.81 for Qwen3 Next 80B, and 4.12 for GPT-OSS 120B. Magnitude-only errors are 1.19, 2.78, 3.34, and 3.59. UX-1 returned a number for all 318 tests. The other systems returned 307, 301, and 294.
Test the 126 favored variants first. That is 39.6% of the slate and 86.1% of the variant winners, and 93.7% of the shortlist won live. The other 19 winners are the cost, and 13 of them were called inconclusive. Treating an inconclusive forecast as a kill decision drops those 13 improvements.
The check this setup supports is to log the forecast before traffic, then run candidates inside and outside the shortlist. The 93.7% hit rate sits on a corpus where 137 of 318 variants won. A queue whose ideas win less often will not keep that rate.
The extra 71 correct tests are a whole-system difference. Browsing-data fine-tuning, 1,000 shoppers per arm, and the page evidence are all inside it. The share due to fine-tuning alone is not measured. Gemini 3.5 Flash agrees with itself after a page-order swap on 53.14% of tests. UX-1 agrees on 94.65%. The headline number still uses one order and the first run only.
Squoosh scored its own model. The tests are private partner experiments, and the sources cannot be named. Ninety-seven of the 412 screened tests had already appeared in an earlier Squoosh Bench report. Whether the training data overlaps the 40-plus sites is unknown.
The Wilson interval assumes experiments are independent. Tests on the same site share audiences and design choices. If errors are positively correlated, 79.53% to 87.59% is too narrow, and no site-clustered standard error is reported. The 21-day cutoff removes 94 shorter tests. The 84% figure describes the longer ones only.
An interval that covers zero does not mean the effect is zero. The label also depends on the randomization unit, interim looks, and multiple-comparison handling. Near the boundary, more human traffic can flip the call. The models saw planned exposure, allocation, and the analysis rule, so an inconclusive call can come from guessing that the test is underpowered. No run removes that design information.
Accuracy is not reported by industry or by intervention. Retail and e-commerce contribute 94 tests (29.6%). The rest cover telecom, travel, financial services, B2B software, food subscriptions, and membership media. Relative-lift error is only a mean absolute error, with no distribution of the true lifts. The 289-test table keeps cases where every system produced a number. If missing numbers fall on harder tests, the errors look too small. The rule that collapses 1,000 shoppers into one class is not stated.