Pairwise recency probes beat eSASRec by 15-38% on Amazon sequential benchmarks

Do Sequential Recommendation Benchmarks Really Require Higher-Order Sequence Modelling?

Aleksandr V. Petrov, Praveen Chandar, Paul N. Bennett, Hugues Bouchard, Mounia Lalmas

cs.IR

2026-08-20

Two recency-weighted pairwise probes, with no higher-order encoder, beat reproduced eSASRec by 15-38% on three Amazon sets, and trail by 27.3% on MovieLens-20M.

What problem this solves

Sequential recommenders increasingly copy language-model architecture. Transformer encoders and generative retrieval post strong benchmark numbers. Those numbers do not show that the benchmarks need higher-order, context-dependent interactions. Popularity, first-order transitions, and recency can explain the same scores.

A reproducibility study of non-sequential recommenders stood up 7 of 18 neural methods, and simple heuristics often beat six of those. Sequential work has shuffled histories to test whether order carries signal. This paper cuts a different way. Even if order matters, the signal may be itemwise pairwise evidence, not a joint term such as P(next | item B and item C).

Method

The probes refuse to learn higher-order sequence representations. Three last-item controls come first: an empirical Markov chain (MC), SASRec-style factored Markov (FMC), and FMC+ trained with full-catalogue softmax. Then two history-aware probes:

• SeqRules. Sparse sequential rules, with distance, decay, row pruning, history length, and optional IDF tuned.
• PCTM (Probabilistic Collaborative Transition Model). Each history item is an expert over the next item. Nearby co-occurrences weigh more. Bayesian smoothing pulls rare sources toward a uniform catalogue prior. At serving time a recency kernel (exponential or head-tail) weights the experts in log space, plus a popularity correction. No embeddings, no sequence encoder, never P(a|b,c).

The better of SeqRules and PCTM is the pairwise envelope. If a Transformer cannot clear it, the benchmark does not show that higher-order modelling is what moved the score.

Evaluation uses the eSASRec release: splits, full-catalogue ranking, seen-item filtering, no sampled test negatives. SAS+ (sampled-softmax SASRec) and eSASRec are reproduced under that protocol. Across ten model-dataset pairs the NDCG gap to the original report is at most 0.0033.

Results

Full-catalogue NDCG@10:

Pairwise envelope versus eSASRec: Beauty +21.2%, Sports +14.5%, Toys +38.5%, ML-1M +4.4%, ML-20M −27.3%. On Amazon, FMC+ is already close to or above the Transformers; on Toys the last item alone wins. Relative to the best last-item model the envelope still adds 20% to 46%, so extra history helps, and the extra is additive pairwise evidence. Versus SAS+ the envelope is 9% to 28% higher on the four smaller sets. A paper-to-paper gap versus TIGER looks like 65% relative, but TIGER has no official implementation; the authors refuse that as an SOTA claim.

Why it matters

This is a cheap benchmark audit. Run SeqRules and PCTM before a new sequential model. If the Transformer cannot clear the envelope, the gain is probably recency and transitions, not higher-order interaction. Beauty, Sports, and Toys look Markov-heavy here. Generative SOTA claims that live only on those sets are under-powered. ML-20M is the one of five with a 27.3% hole left for attention. If the pitch is extra capacity from self-attention, that is the kind of set to beat.

Code, frozen configs, and split hashes are public. The conclusion is deliberately narrow and does not speak to the complexity of production ranking.

Limitations

This is a RecSys 2026 extended abstract, marked work in progress. Five public benchmarks, no industrial time-ordered logs. PCTM beats SeqRules on four of five sets, without an ablation that separates Bayesian smoothing from log pooling. Hyperparameters are tuned per dataset, so the probes are not zero-tune. Transformers still lead by a lot on ML-20M, which is evidence that higher-order capacity can be real on denser sequences. A shuffle test of order is not in this paper.

Terms

• Higher-order sequence modelling: joint, non-separable effects among history items, as opposed to summing per-item pairwise evidence.
• Pairwise envelope: the better of SeqRules and PCTM, a pairwise ceiling with no sequence encoder.
• eSASRec: a modular SASRec variant; this paper uses its official protocol and splits.
• Capacity probe: a deliberately weaker model class used to test whether a benchmark actually needs the capacity being advertised.

Source

• arXiv 2608.19833v1
• PDF
• Hugging Face

What people are saying

• RecSys Study: Simple Models Outperform Transformers on Multiple Benchmarks _reachsumit

Related papers

• Retrieving thinking traces beats web corpora, lifting Flash 56.3% on AIME
• CrossQ spends quantization bits on ColBERT winner tokens, +0.010 MRR at 2 B/token
• Netflix Folds Five Artwork Models Into One With CLIP, Then Ships Tri-Modal Previews
• UNGER folds semantic and collaborative signals into one code, 2.8x faster than dual codes
• TIGER Decodes Semantic IDs for Sequential Recommendation, +29% NDCG@5 on Beauty
• ANCE's Global ANN Negatives Push Dense Retrieval Near BERT Cascade Accuracy

All paper explainers