SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations
Tobias Vente, Maarten Peirsman, Noah Daniëls, Hannu Toivonen, Bart Goethals
cs.IR, cs.AI, cs.LG
2026-09-25
SPADE maps items into popularity vs. profile similarity, scores serendipity as distance to a per-user Pareto front, and averages only over test-set hits. Random's Primitivity on MIND is 5.50; SPADE is 0.0037.
Recommender serendipity is treated as an engineering target: relevant, unexpected, relatively new. Offline beyond-accuracy metrics usually watch one axis. Content distance or PMI only asks how far an item sits from a user's history, so a global hit can look "surprising" because this user has not seen it. Primitivity against a primitive recommender is glued to whichever baseline you picked. Temporal gap metrics need organic discovery logs, which most teams do not have. The missing axis inflates false positives: obscure and irrelevant, or popular and merely new-to-this-user.
SPADE places every candidate in a plane. The x-axis is global popularity, interaction count over the most frequent item, scaled to [0, 1]. The y-axis is similarity to the user profile, the sum of unnormalized pointwise mutual information against history. Unnormalized PMI is used because it anti-correlates with popularity; normalized PMI brings the positive correlation back, the front collapses to a point in the upper right, and distance stops meaning anything. Empirical-Bayes smoothing with α=1 damps rare co-occurrence, negatives clip to 0. Per-user min-max then puts similarity on the same [0, 1] scale.
A user-specific Pareto front is the set of items that no other item beats on both popularity and similarity. An item's SPADE score is the minimum Euclidean distance to that front. The list score averages only over recommended items that also sit in that user's test set, then averages over users. Misses do not enter the numerator. That is the split from Novelty, Primitivity, and co-occurrence surprise.
Datasets: CiteULike, MIND, MovieLens-1M/20M, Netflix, 5-core, three large sets capped at 10k users. Algorithms: EASE, SLIM, ItemKNN, Popularity, Random, Recpack, strong-generalization 80/20, K from 1 to 30.
Qualitatively, user 1874 mostly watches drama, crime, and thriller, with sci-fi at 9% of history. The 1997 sci-fi film Retroactive in the test set has similarity 0.46 and popularity 0.005, bottom-left, farthest from the front. User 2186 is action-adventure, crime at 11.6%; The Way of the Gun scores similarity 0.38 and popularity 0.05, again a high SPADE hit. Both are relevant without being hot or on-profile.
Quantitatively, metrics that ignore relevance let Random farm the score. On MIND, Random's NDCG is 0.0006, Primitivity 5.499, co-occurrence 0.282, in the same band as serious models; SPADE assigns 0.0037. Across five datasets, Random and Popularity sit at the bottom of SPADE.
| Dataset | EASE SPADE | Popularity SPADE | Random SPADE | Random Primitivity |
| Netflix | 0.127 | 0.029 | 0.004 | 5.498 |
| ML-1M | 0.101 | 0.031 | 0.014 | 5.477 |
| MIND | 0.234 | 0.037 | 0.004 | 5.499 |
| ML-20M | 0.132 | 0.032 | 0.003 | 5.496 |
| CiteULike | 0.406 | 0.016 | 0.008 | 5.496 |
EASE on ML-1M has NDCG 0.335 and Novelty 1.09. Those two numbers cannot say whether the novel items were the ones the user actually took. SPADE forces relevance and unexpectedness to hold together by scoring test-set hits only.
For evaluation, this is an offline ruler that Random and Popularity cannot easily game. It does not generate serendipity; it asks whether an algorithm surfaced relevant items far from the "popular and on-profile" boundary. Pareto fronts in recsys usually guide training-time trade-offs. Here they are a post-hoc metric, the opposite use.
Code is public. The metric does not tell you how to optimize serendipity. It tells you how to measure it.
Scoring only test-set hits means an unexpected recommendation the user never touched never enters the average, so weak-recall models are compressed by design. Similarity is purely collaborative, with no content features; PMI for cold items is noisy, and smoothing suppresses noise without creating signal. MIND, ML-20M, and Netflix are subsampled to 10k users, so the front can move with the draw. There is no online study, and distance-from-front is not calibrated against users saying "that surprised me". Euclidean distance treats the axes as isotropic, which is an assumption. Random Novelty is inf on some sets, so part of SPADE's contrast is the opponent being brittle.