Meta's JiTTesting deploys LLM-generated tests to catch bugs pre-merge, cutting human review 70%
AxSaucedo · x · 2026-11-05
Meta announces its Catching JiTTests system has moved from research prototype into production infrastructure: LLMs generate regression-catching tests on proposed diffs before they land, flagging bugs traditional testing would miss.
Key results from analyzing 22,126 generated candidates (full paper at FSE 2026):
- Candidate catch generation is 4x better than hardening tests and 20x better than confidently-failing tests
- LLM-based test assessors cut human review load by 70%
- 41 candidate catches reported; 8 confirmed true positives, 4 of which would have led to serious failures previously undetected
- Workflow uses a sparse classifier (Diff Risk Score) trained on prior changes to prioritize high-risk diffs for test generation
Meta now reviews only risky diffs before landing, letting engineers skip the generated test code. The paper also covers the inherent challenge of distinguishing true from false failures under a weak test oracle.
More from coding & agent
- OpenAI ships ChatGPT Images 2.6 with GIFs, reference images, and 50% faster API models — kagigz · 2026-11-05
- Biomedical benchmark: prewritten Cypher MCP tools hit 98% vs 85% for Text2Cypher — JeremyCMorgan · 2026-11-05
- Musk shows off Grok calling a dentist to check insurance coverage — elonmusk · 2026-11-05
- Treat AI bots as employees, not assistants: an onboarding playbook — elonmusk · 2026-11-05
- Codex Voice cuts tool latency by streaming context to the harness before delegation — SIGKITTEN · 2026-11-05
- I split AI short-drama production into 5 agents—here's what broke — LateQuantity4194 · 2026-11-05