Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews
Barbara Kitchenham, Sebastián Pizard, Lech Madeyski, Ronnie de Souza Santos, Martin Shepperd, David Budgen
cs.SE, cs.AI
2026-07-28
A software-engineering methodology group proposes the GUEST guidelines, arguing that accuracy misranks GenAI literature screeners: in a 9,695-article evaluation, the best-by-accuracy model missed 63.3% of relevant studies, while best-by-WMCC missed only 5.8%.
A systematic literature review (SLR), a method that exhaustively collects relevant studies under a strict protocol, is how fields like software engineering ground their claims. More researchers now use GenAI and large language models to help screen, extract, and synthesize, and while models can summarize text, the rigor, reliability, and transparency a review demands do not come for free.
The authors' worry is methodological. Bad practices are already creeping into both using GenAI for reviews and evaluating those tools. Software engineering has been here before: cost estimation leaned on MMRE as its default metric for years before the field realized it was biased, by which point it was a de facto standard. They do not want GenAI-assisted reviews to repeat that pattern.
This is a methodology guideline, not a benchmark. It proceeds in two steps: first a rapid review to collect existing guidelines for using and evaluating GenAI in SLRs, then thought experiments combined with methodological guidance from the literature and the authors' own experience running reviews and evaluating tools. The output is a set of process recommendations named GUEST (GenAI Use and Evaluation in SLR Tasks). The name is deliberate: a GenAI tool should contribute under the researcher's oversight, like a guest, never replace them.
The most forceful evidence comes from one of two companion empirical studies, a screening evaluation. On a screening task over 9,695 articles, picking the "best" model by different metrics gives wildly different outcomes: the best model by accuracy missed 63.3% of relevant studies; the best by plain MCC missed 43.9%; the best by weighted MCC (WMCC) missed only 5.8%. Three searches recovered 13 and 18 known relevant papers respectively, consolidated into six integrated guidelines (G1 to G6).
| Selection metric | Miss rate of best model |
| Accuracy | 63.3% |
| Plain MCC | 43.9% |
| WMCC | 5.8% |
The root cause is class imbalance: relevant studies are a tiny minority of the corpus, so a model that labels everything "not relevant" scores high accuracy while missing every study the review actually needs.
For anyone running a review, the paper offers an actionable checklist: screen with cost-sensitive WMCC rather than accuracy, report the full confusion matrix, guard against data contamination (test material sitting in model memory is effectively cheating), use baselines, report cost, and watch for class imbalance. For tool evaluators it spells out how to test these tools independently and reproducibly. The conclusion is firm: GenAI cannot run unsupervised systematic reviews, but it can save money on repetitive tasks and on second-pass validation of complex ones.
The authors label their recommendations "preliminary" and expect them to evolve. The method mixes thought experiments with experience rather than pure empiricism, and direct evidence is thin for stages like data extraction and risk-of-bias assessment. The whole frame assumes an academic context and may not transfer cleanly to industry. The 5.8% miss figure for WMCC comes from a single screening study with limited scope, so it should not be copied as a universal number.