Squoosh Bench: UX-1 agent predicts 84% of 318 historical A/B test outcomes, beating Qwen3 Next 80B
Rohith Pasula · hf · 2026-11-21
Squoosh released Squoosh Bench, testing whether simulated shoppers can help experimentation teams decide which website changes deserve live-traffic tests:
- From 412 historical A/B tests, the 318 that ran longer than 21 days form the eval; page roles are masked and outcomes withheld during inference, with systems predicting an original win, variant win, or inconclusive result.
- UX-1 (fine-tuned on human browsing data) simulated 1,000 shoppers per arm and correctly classified 267/318 outcomes (84.0%, 95% Wilson interval 79.5–87.6%).
- The strongest general-purpose model, Qwen3 Next 80B, got only 196/318 (61.6%).
- Precision matters more: UX-1 favored the variant in 126 experiments; in live tests, 118 of those variants won (93.7%), four lost, four inconclusive. The remaining 192 included 19 additional variant wins.
Takeaway: simulated shoppers fine-tuned on human browsing can pre-screen experiments and substantially cut live-testing costs.
More from Research
- WorldTrace beats sliding window recall by 32.6% at same cache size, tops all 12 LoopBench conditions — cindy_x_wu · 2026-11-21
- PageIndex: vectorless RAG with no embeddings or chunking hits 98.7% on FinanceBench — dr_cintas · 2026-11-21
- Isomorphic Labs partners with Merck on AI-designed cancer drug — demishassabis · 2026-11-21
- Owain Evans discovers 'homeopathic learning': clean data can unlock subliminal backdoors — OwainEvans_UK · 2026-11-21
- Mathematicians will study Lean proofs like N-body simulation results, researcher predicts — mjuric · 2026-11-21
- METR: AI R&D capabilities are improving fast — and getting harder to measure — dfrsrchtwts · 2026-11-21