Meta paper: ABTest Agent lets LLMs plan A/B tests but never generate a single number
rohanpaul_ai · x · 2026-11-08
A new Meta paper, "Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale," details the company's internal ABTest Agent.
The core insight: LLMs are nondeterministic and can hallucinate, while A/B tests demand trustworthy numbers — and experiments still involve messy manual work like debugging setups and diagnosing odd results. Meta's solution is to let the LLM plan experiment steps and interpret outcomes, but never generate a single number itself; all metrics still come from deterministic systems, so staff can trust the assistant's figures.
The paper also catalogs lessons and pitfalls from running AI-assisted experimentation at scale — a useful reference for teams building agent-backed experiment platforms.
More from Research
- Fully Hidden Test Sets Make Benchmarks Harder to Adopt, Researcher Notes — srchvrs · 2026-11-09
- Scott Alexander vs Pinker scaling spat: LLM scaling laws are inverse power laws, never exponential — sebkrier · 2026-11-09
- YC Paper Club: AI for Biology, From Reading Data to Designing Living Systems — ycombinator · 2026-11-08
- Hand-Built Recommender Loses to the 'Most Popular' Baseline Half the Time — dima806_dima · 2026-11-08
- NYT: OpenAI results upend a key tenet of computer science — P vs NP — _akpiper · 2026-11-08
- Running Qwen3.5-0.8B in pure Assembly: 6.51 tok/s on one CPU thread — suayptalha · 2026-11-08