Data Processing for Offline Evaluation in Recommender Systems: a Survey
Alberto Carlo Maria Mancino, Angela Di Fazio, Danilo Danese, Matteo Attimonelli, Daniele Malitesta, Antonio Ferrara, Claudio Pomo, Tommaso Di Noia
cs.IR, cs.LG
2026-09-19
A survey of 517 RecSys papers finds no shared preprocessing practice: support filtering dominates, same-named splits hide different setups, and 46% of datasets are used once.
Offline evaluation is the default mode in recommender systems research: citing Zangerle and Bauer (2023), more than 70% of published studies run on historical interaction data, with no deployment and no user recruitment. The foundation has known cracks that the literature has named repeatedly: dataset-task mismatch, information leakage, biased negative sampling, missing statistical tests.
Attention went to the two ends of the pipeline, model architectures and evaluation metrics, while the middle stretch, everything done to the data before training, received almost no systematic scrutiny. Two papers can evaluate on different versions of the same benchmark with different splits. Multimodal recommendation adds another layer: image features come from ResNet-50 or VGG-16, text from Sentence-BERT or precomputed embeddings, chosen per paper. Performance gaps between papers may come from data preparation as much as from model innovation. This 147-page survey from a Politecnico di Bari-led team is the first cross-paradigm inventory of those practices.
The corpus was built indirectly. Eleven domain surveys published between 2020 and 2025, each with at least 30 citations and appearing in Q1 journals or A/A venues, served as entry points, covering paradigms from collaborative filtering to LLM-based recommendation. Filtering to papers that propose new methods or empirical studies and use offline evaluation yielded 517 unique papers from 2001 to 2024. About 75% appeared in A/A conferences or Q1 journals, so the mess documented below cannot be blamed on weak venues.
The core contribution is a unified framework that splits the pipeline into dataset selection, interaction representation, data preparation, multimodal feature extraction, and splitting, plus a taxonomy of data transformations:
The 517 papers draw on 270 distinct datasets across 1,497 paper-dataset pairs, averaging 2.9 datasets per paper. Usage is heavily long-tailed:
| Finding | Scope | Value |
| Datasets used in a single paper | out of 270 | 46.1% (124) |
| Top 30 datasets | share of distinct datasets | 11.2%, absorbing most usage |
| Movies domain | 25 datasets | 361 uses, 14.44 avg reuse |
| E-commerce domain | 54 datasets | 263 uses, 4.87 avg reuse |
| Multi-domain | 7 datasets | 110 uses, 15.71 avg reuse |
MovieLens-1M, Yelp, and the Amazon family are the most used. On processing, dataset-level transformations are dominated by support-driven filtering while representation-dependent transformations are uncommon; preparation of side information is heterogeneous; and splitting protocols are specified inconsistently, with similar labels concealing different experimental conditions. A slice of the datasets is simply unavailable: proprietary (Amadeus CME, AlipayMiniApp), request-only (8TRACKS, Vine), or with dead links (Art of the Music, RecSys Challenge 2016).
For researchers, this works as a checklist: before comparing against a baseline, verify that the dataset version, k-core threshold, splitting protocol, and feature extractor match; a gap that survives that check is worth attributing to the model. The taxonomy also gives the community a shared vocabulary for reporting, useful in review and replication. For readers of the literature, the practical default is that numbers from different papers are not comparable until the data processing is verified to match, and LLM-based recommendation deserves extra caution since much of it appears in non-peer-reviewed venues. This is a survey, not a new method; its value is systematisation, and for a question nobody had inventoried, the first inventory is the reference.
Sampling through 11 surveys has its own bias: papers outside those surveys never enter, and the 30-citation floor favours older, mainstream work, so the newest practice may be underrepresented. Papers behind paywalls the authors could not access were excluded, which the survey acknowledges. The frequency statistics answer what people do, not what each preprocessing choice costs in metrics; there is no controlled comparison of that kind. When a paper does not name its dataset version, the authors inferred it from user and item counts, which leaves room for annotation noise.