DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
Lekang Yang, Yuetong Liu, Yitong Zhang, Jia Li
cs.SE, cs.CL
2025-09-30
DiffuTester extra-decodes AST-shared tokens in dLLM unit-test generation; on TestEval-C++ with DiffuCoder at batch 3, cost falls from 1217 to 430 TFLOPs and time from 14.4s to 6.0s.
Diffusion LLMs can fill many [MASK] tokens in one forward pass, so they should be faster than left-to-right models. On unit-test generation that advantage is usually switched off. Published setups decode one or two tokens per step, which makes them about as slow as a normal LLM. Decode more tokens per step and test quality collapses; even syntax starts to fail. Appendix Figure 8 plots the trade-off: throughput up, syntactic correctness down.
Unit tests leave a hole you can actually mine. Several tests for the same focal method often share a skeleton and differ in literals. Figure 1 shows two Java testshortestPath methods whose ASTs match except for the grid constants and the asserted value. A longer Python test repeats addnode calls inside one file. Structure repeats; diversity lives in numbers and strings.
DiffuTester is a training-free decoding plugin. For one focal method the batch size is n (3, 5 or 7 in the experiments), so n tests are drafted in parallel. Each step first runs the usual confidence-based unmasking (top-k), then a second pass that unmasks extra tokens from shared structure.
Structure comes from ASTs. Leaves carry lexical items (names, numbers); internal nodes carry syntax. The method merges ASTs across the current batch. A non-empty merge is treated as a structural pattern, and the matching tokens are kept in this step. Early steps are often syntactically broken, and a whole-file parse would spread those errors, so ASTs are built per line rather than per test.
Diversity is protected by refusing some merges. Coverage depends on input variety, and that variety is mostly integer and float literals. Literal nodes are excluded from the merge. Even if two tests currently build similar data, those tokens are remasked and refined later.
Two engineering knobs sit on top. Keeping every token on the merged tree slightly hurts syntax, because a very low-confidence position occasionally survives; only tokens above a confidence threshold τ are kept. The main run uses τ=0.02. Running AST alignment every step has overhead, and consecutive trees barely change, so the plugin fires every other step. Ablation finds a fixed interval of 2 better than every step, and better than a dynamic schedule.
Generation length L is fixed at 128. Default remasking decodes two tokens per step, so 64 steps without acceleration. With DiffuTester the step count is roughly normal, mostly well below 64, and adapts to the focal method.
Models: Apple DiffuCoder-7B-cpGRPO and HKU NLP Dream-v0-Instruct-7B. Benchmarks: TestEval's 210 LeetCode Python programs, plus C++ and Java ports of the same 210. Coverage via pytest, Maven and gcov. The baseline is the same dLLM without DiffuTester.
Main table at batch n=3, same number of tests:
| Setting | Cost TFLOPs | Time s | Throughput tps |
| DiffuCoder / Python | 1016 → 580 (1.75×) | 12.2 → 7.8 (1.57×) | 17.0 → 26.9 |
| DiffuCoder / C++ | 1217 → 430 (2.83×) | 14.4 → 6.0 (2.42×) | 9.7 → 23.8 |
| DiffuCoder / Java | 1259 → 668 (1.89×) | 14.9 → 8.9 (1.67×) | 16.1 → 29.2 |
C++ accelerates most. The authors credit more uniform C++ syntax, which is easier to align. Dream follows the same shape at a smaller factor: 1.68× cost and 1.51× time on Python at n=3.
Line coverage on TestEval-Python barely moves. At n=3/5/7 DiffuCoder scores 91%/94%/94% alone and 92%/93%/94% with DiffuTester. Generic accelerators are harsher on quality. EB-Sampler drops coverage to 86%/88%/88% in the same setting. SlowFast Sampling finishes n=10 in 7.7s but stalls at 77% coverage, below the n=3 baseline of 91%. DiffuTester spends 24.5s at n=10 and stays at 94%.
Against a similar-size autoregressive model the reference is vanilla Qwen2.5-7B-Instruct, no speculative decoding. Dream plus DiffuTester usually reaches the same coverage with less time and less compute. That comparison is thin: speculative decoding could be stacked on the AR side too.
On other code tasks the speedup survives but shrinks. HumanEval-X, DiffuCoder, 4 samples per prompt: Python 5.68s / pass@1 50% → 3.87s / 49%; C++ 5.73 / 11 → 4.21 / 13; Java 6.32 / 11 → 5.36 / 11. UniTrans translation saves about 1–2 seconds; pass@1 moves both ways (Python→C++ 42.9 to 46.2, C++→Python 96.4 to 95.5). Repeated structure is weaker here, so the ceiling is lower. C++/Java pass@1 in the low teens also says the base dLLM is weak on those languages.
For anyone already sampling unit tests from a dLLM, this is a task-specific sampling accelerator, not another KV cache. It is compatible with cache methods; the paper only compares against other samplers and does not stack the two. It is training-free and holds across Dream, DiffuCoder and three languages.
The narrow fit is also why quality holds. Tests that share an API and swap data give AST alignment something to grab. On HumanEval-style one-shot generation the speedup is only about 20–30%. If production tests are long files with fixtures, well past 128 tokens, these numbers do not transfer.
The authors' own limit is one sentence: not enough time or compute to try more models or more tasks. Several more holes sit in the setup.
Length is frozen at 128; real tests often run longer. 210 LeetCode problems are not industrial repositories, and the C++/Java splits are the same problems in another language. The AR baseline turns off speculative decoding, so the claim is "bare dLLM plus structure" versus "bare AR". HumanEval C++/Java pass@1 is poor, which means the method preserves coverage rather than rescuing a weak model. τ=0.02 and the every-2-steps schedule are hand-set; there is no transfer study on whether they need retuning per model.