Mutation-Guided Unit Test Generation with a Large Language Model
Guancheng Wang, Qinghua Xu, Lionel Briand, Kui Liu
cs.SE
2025-06-03
MutGen feeds surviving PITest mutants into Llama-3.3 prompts and reaches 89.5%/89.1% mutation score on 204 Java methods, about 20 and 30 points above EvoSuite.
Full line and branch coverage can still leave a test suite almost blind to faults. On HumanEval-Java subject id81, Llama-generated tests hit 100% line and branch coverage and a 4% mutation score.
The failure mode is concrete. PITest's ConditionalsBoundary operator rewrites day < 1 to day <= 1, so the original method accepts day=1 and the mutant rejects it. A coverage-oriented prompt typically emits assertFalse(validDate("04-00-2025")), which fails on both versions, and rarely emits assertTrue(validDate("04-01-2025")), which is the input that actually kills the mutant. Comments make things worse: HumanEval-style docstrings that start with "You have to write a function" push the model toward restating the implementation instead of writing tests.
EvoSuite optimizes coverage. Most recent LLM test-generation papers still report coverage. Foster et al. do look at mutation score on Kotlin regression tests for WhatsApp and Instagram, but they start from existing tests and let an LLM invent mutants, which brings equivalent-mutant noise. Dakhel et al. add one mutant at a time on Python and stop when the new mutant survives, so they never drive toward a maximum score and never explain the leftovers.
MutGen optimizes mutation score. The backbone is Llama-3.3 70B served locally with Ollama at temperature 0.0. There is a preprocessing stage, a generation stage, then iteration on surviving mutants.
Preprocessing does two jobs. Comments are stripped and the model writes a short summary of purpose and input format, which then goes into the prompt. PITest's DEFAULTS operator group produces the mutation report; MutGen pulls line, status (killed / survived / uncovered), and operator, and feeds that next to the source. DEFAULTS is the usual literature set: harder to kill, fewer equivalent mutants.
The generation prompt has three parts: the summary, comment-free source, and mutation feedback. The model is told to diversify inputs per surviving mutant. Failing tests go through a second prompt keyed to the error message, covering six buckets: wrong assertion API, wrong assertEquals oracle, flipped assertTrue / assertFalse, ambiguous types that need a cast, duplicate test names, and leftover compile or runtime errors.
Then it iterates. Each round regenerates from still-live and still-uncovered mutants, repairs failures, and unions the tests that run. A pilot on 10+10 subjects over 7 rounds plateaued at round 4, so the main experiments cap at 4. On the running validDate example, two rounds reach 100% mutation score; the vanilla prompt is still at 53% after four.
Evaluation is method-level: one target method per Java class, no cross-module dependency solving. From 160 HumanEval-Java subjects they drop those where both MutGen and EvoSuite already score 100%, leaving 104. Leetcode-Java is compiled Medium/Hard GitHub solutions, the same 100% filter, then 50 random subjects per difficulty, 100 total. Mean method size on HumanEval is about 41 lines; cyclomatic complexity is 4.90 versus 7.88. EvoSuite's time budget is 150 seconds, the larger of MutGen's two mean runtimes. Coverage is JaCoCo, mutation score is PITest, three repeats, average reported.
PITest yields 1,144 and 1,900 mutants on the two datasets, 11 and 19 per subject.
| Method | HumanEval mutation | LeetCode mutation | HumanEval line/branch | LeetCode line/branch |
| EvoSuite | 69.5% | 58.9% | 95.6% / 93.4% | 99.0% / 98.9% |
| EvoSuitemut | 67.4% | 58.1% | 95.4% / 93.4% | 98.1% / 98.7% |
| Genvanilla | 77.9% | 69.9% | 96.2% / 92.8% | 96.3% / 92.7% |
| MutGen | 89.5% | 89.1% | 98.3% / 95.8% | 98.4% / 94.8% |
Against EvoSuite that is about 20 and 30 percentage points (relative 28.8% and 51.3%). A12 is 0.759 and 0.899 versus EvoSuite, 0.650 and 0.734 versus Genvanilla. A Wilcoxon signed-rank test at α=0.05 is significant for mutation score and not for coverage, which is already near ceiling. Coverage gaps stay under 3 points; EvoSuite wins coverage on LeetCode, which is what it is built to do.
Switching EvoSuite's fitness to strong mutation (EvoSuitemut) does not help: 67.4% and 58.1%, slightly below default EvoSuite. The paper's guess is a decade without maintenance, operators that do not match PITest, and a 150-second budget that may be tight.
Of 1,254 failing tests on HumanEval, 52.08% are repaired (601 left). Of 2,742 on LeetCode, 47.29% (1,445 left). A sample of 50 unrepaired tests: 74% bad oracles, 26% runtime exceptions from invalid inputs. Three-run mutation scores are 89.9% / 90.2% / 88.5% on HumanEval (range 1.7%) and 88.2% / 88.0% / 91.1% on LeetCode (range 2.1%).
After four rounds the overall kill ratios are 93.3% and 94.7%. NegateConditionals and Math sit above 93%. The hard operators are VoidMethodCalls (69.2% on HumanEval, 7 of 26 live) and TrueReturns (57.7% on LeetCode). Removing a void call often changes a side effect that never shows up in the return value; TrueReturns leftovers are mostly uncovered. Uncovered is a different bucket from live: dead code, plus mutated statements the suite still never reaches. Operator frequency does not explain kill ratio; Spearman is -0.214 and 0.36.
Ablations: MutGen-S keeps original comments, MutGen-F drops the fixer, MutGen-MF drops mutation feedback. From the round-4 plots, summarization moves the needle most, the fixer next (LeetCode has more failing tests), mutation feedback last by pushing harder mutants. Exact points per variant are not in the text.
Runtime is 125.9 s / 899 tokens on HumanEval and 149.4 s / 1,629 tokens on LeetCode. More mutants, longer prompts, more tokens.
A green coverage report is a weak certificate. The 100%-coverage, 4%-mutation example is the warning worth keeping.
Stuffing a mutation report into the prompt beats retuning EvoSuite's fitness. The method is model-agnostic on paper. What you actually need in production is method-level Java, a working PITest run, tolerance for repairing about half the failing tests, and leftover oracle bugs after that. Project-level dependencies and Defects4J-style real faults are explicitly out of scope.
If a team already uses an LLM for unit tests, the portable pieces are: replace comments with a summary, paste surviving-mutant diffs into the prompt, and repair failed assertions by error class. Four rounds is an empirical cap for this setup; a new model or language needs a new calibration.
Subjects are isolated methods, roughly 40 lines, no cross-module dependencies. Defects4J would make dependency resolution a paper of its own.
PITest reduces equivalent mutants and does not eliminate them, so 100% is unreachable. Every method is taxed the same, so pairwise comparisons still hold. The pipeline assumes the code under test is correct and only repairs assertions that break the Maven build, which produces wrong oracles and false positives.
Llama-3.3 has almost certainly seen HumanEval and public LeetCode solutions. The authors argue that specific mutants are unlikely to be memorized, and that the gap over vanilla prompting is evidence against pure recall. That defense is only partial: the vanilla baseline runs on the same model. One model, temperature pinned at 0.0. Whether GPT or DeepSeek would reorder which operators are hard is unknown.
EvoSuitemut is a weak "mutation-first" control: stale operators, time budget copied from MutGen. Ablation ranking comes from figures; MutGen-S / F / MF scores are not tabulated, so "summarization matters most" cannot be rechecked from the text.