Just-in-Time Catching Test Generation at Meta
Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman, Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng
cs.SE, cs.AI
2026-01-30
Diff-aware catching tests at Meta hit about 4x the weak-catch rate of hardening tests. Engineers confirmed 8 of 41 signals as real bugs; 4 would have been severe failures.
Most generated unit tests are hardening tests: they must pass when written, then land in the repo to block later regressions. A catching test is built to fail, so it can flag a bug in a diff, Meta's name for one submitted change, before that change lands. A failing test cannot ship next to the edit it accuses.
A test that passes on the parent and fails on the child is only a weak catch. Behavior changed. The engineer may have wanted that. A strong catch also has to match the general oracle, the behavior the system is supposed to have, which is often unwritten. Crashes fall under the implicit oracle: they are wrong no matter what the spec says. Once the spec is vague, a red test might indict the code or the test. False positives slow landing. Meta's backends run to hundreds of millions of lines and serve more than 3.5 billion people a day, and test generation is expensive, so the workflow only chases severe regressions on high-risk diffs.
A scorer in the spirit of Diff Risk Score is trained on past changes. It runs overnight on spare West Coast capacity and generates tests only for the previous day's high-risk diffs.
Both generators use mutation testing. A mutant is a program broken on purpose. A test that passes on the original and fails on the mutant has killed that fault.
Dodgy diff does not guess intent. It treats the diff as a mutant of its parent and asks the mutation-guided LLM generator already in production to kill it.
Intent-aware reads only the code, the title, and the summary. The model lists ways an attempt to implement that intent could go wrong. Each risk becomes a mutant of the parent, a killing test is generated, and that test is run on the real diff. Failures are kept as weak catches. The bet is the coupling hypothesis: tests that kill these simple synthetic faults often hit real faults coupled to them. Email, task text, and review discussion are held out on purpose, as a baseline for richer context later.
Assessors run only after a weak catch exists, so they always see the diff, and they are what separates likely false positives from likely real bugs. Scores run from -1 to 1. RubFake matches patterns in the log, the test, and the diff. A broken runner, reflection, bad mocks, assertions on private methods, and frozen key order count as false positives. Points go the other way when a key goes out of range without an edit, a boolean flips while its defining expression does not, a refactor or a logging-only change still alters old behavior, or access control (RBAC) shifts unexpectedly. A boolean flip can be checked in one question. Order changes inside a large structure take much longer to dismiss, so they rank lower.
The LLM-as-judge ensemble is Llama 3.3 70B, Gemini 3 Pro, and Claude Sonnet 4. Each model sees the failing test, the stack, the diff, and the inferred intent. Llama's yes/no first-token probability is normalized onto the same -1 to 1 scale. High, Medium, and Low from the three models map to 1, 0, and -1, and the median is kept.
The ping does not start with test code. It asks whether the behavior change was intended. The test is opened only if the author is surprised.
Five workflows produced 22,126 tests. The weak-catch rate is the share of a workflow's tests that fail on the child and pass on the parent.
| Approach | Tests | Weak-catch rate | Diffs with a weak catch |
| Coincidental failure | 13,001 | 0.2% | not reported |
| TestGen-LLM, coverage not required | 4,499 | 2.0% | 3.6% |
| TestGen-LLM, must add coverage | 1,440 | 0.9% | not reported |
| ACH, mutation-guided | 754 | 0.8% | 1.6% |
| Dodgy diff | 1,621 | 2.5% | 4.0% |
| Intent-aware | 811 | 6.4% | 7.9% |
The abstract's 4x over hardening tests and 20x over coincidental failures is a summary. In the table, intent-aware's 6.4% is about 8x ACH's 0.8% and about 30x the 0.2% coincidental rate. Dodgy diff's 2.5% is only slightly above TestGen-LLM with no coverage requirement (2.0%). Per diff, intent-aware reaches 7.9%, TestGen-LLM 3.6%, dodgy diff 4.0%, and ACH 1.6%. A hardening test fails on an arbitrary later diff about 1% of the time.
From mid-September through the end of October 2025, 8 of 41 engineer pings were confirmed strong catches, a 19.5% true-positive rate as engineers experienced it. Four of the eight would have been severe production failures. Of the other four, two led to defensive fixes and two diffs were abandoned. Engineers said dismissing a false positive took a few minutes.
On 600 weak catches, precision for calling a false positive exceeded 98% when the token score was -1 or the ensemble median bucket was 0. Almost every candidate is a false positive, so that precision discriminates poorly. The figure that matters is the review load: dropping the low scores cut human review by about 70% and raised capacity to about 4x manual review. Nothing shows that the discarded band contained no true positive.
Against landing labels, dodgy diff separates and intent-aware mostly does not. Rejected diffs score higher as true positives: RubFake p=0.002 (small effect), both LLM scores p<0.001 (medium). Accepted diffs draw more LLM false-positive scores. On intent-aware, the only significant cell is the bucketed false-positive rate (p=0.003). Spearman correlation among assessors tops out at 0.535, and Fleiss' kappa is about 0.01.
From 11 November to 11 December 2025 the two catching workflows emitted 8,797 tests across 2,729 diffs: 7,987 passed and 405 failed.
A generator can hunt faults this change introduced. Hardening tests are for the next change. The engineer first says whether the new behavior was intended. Test code waits until that answer is a surprise.
Intent-aware covers nearly twice as many diffs as dodgy diff, with fewer tests. Mutants built from ways the intent could be implemented wrong land closer to real bugs than treating the whole diff as one mutant. Four of eight confirmed bugs met the bar for a severe failure, above the 5% to 20% range often cited for serious bugs. Eight cases cannot carry that comparison. The part that is actually usable is the question asked at review time. There is no new testing theory here.
The four severe failures have no public incident write-up, so the severity label cannot be checked. The body once counts 7 confirmed issues while still quoting 19.5%, which matches 8/41, and once calls the severe share 43%, which is not 4/8.
The 4x and 20x claims do not match Table 1 cell by cell. TestGen-LLM without a coverage requirement is already at 2.0%; dodgy diff is at 2.5%. The four main workflows were compared on the same diffs over ten days, 14 to 24 November. Coincidental catches come from a longer production window and were turned off on 11 November.
The clean split between accepted and rejected diffs sits on the weaker dodgy-diff generator. Intent-aware, which finds more weak catches, barely separates landed diffs from rejected ones, yet the abstract states the split as an overall result. The 98% figure is what a pile of false positives produces. The 70% review cut assumes the low-score band hid no true positive, and that is not shown. The rules came from an informal pass over 600 catches. The 41 pings were human-gated, so 19.5% is not an automatic precision. A landing label is not a bug label. Compute cost, and which workflow found each of the eight real bugs, goes unreported.