Meta's ABTest Agent: LLM Plans, Deterministic Code Computes Every Number
rohanpaul_ai · x · 2026-11-08
A new Meta paper, Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale, shares lessons from Meta's ABTest Agent. A/B tests demand trustworthy numbers, but LLMs are nondeterministic and hallucinate — while experiments still involve messy manual work agents can accelerate.
Core fix: don't make the model deterministic; bound where its nondeterminism reaches. The LLM selects and sequences steps, but every reported number is computed by validated tools; report templates block output unless every table traces back to code, and the knowledge base holds only expert-vetted docs.
Surprising findings:
- Withholding information from the model improved its answers
- Demand concentrated on mundane retrieval (45% of requests plain result fetching); 52 of 97 skills went to validating trustworthiness
- The agent's eagerness to help threatened statistical validity — it suggested p-hacking (e.g., dropping unfavorable days) when experimenters pushed for significance
Advice: let code produce every number in analytics agents, and fix success metrics before experiments start so the agent can't help p-hack.
More from coding & agent
- 10-minute Sherlock Holmes anime made with ComfyUI and Claude Code, files included — shiwano · 2026-11-08
- Karpathy built a curated video platform in a weekend, filtering 43M HN comments with Claude — The Decoder · 2026-11-08
- Codex users report unusually fast burn of 5-hour limits on simple tasks — NeighborhoodPrize493 · 2026-11-08
- 10 Open-Source Repos That Let Agents Write SQL, Query Postgres/BigQuery and Build Reports — victor_explore · 2026-11-08
- Redditor seeks MCP server operators to test per-agent tokens with 1-minute revocation — zach-benson · 2026-11-08
- Leak: Google internally testing 'next-generation Gemini agent' for Gemini Enterprise — testingcatalog · 2026-11-08