Advancing Open and Reproducible Relational Learning: RelArena-$α$, TabPFN-Rel and RPI
Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, Jonas Kübler, Siyuan Guo, Oscar Key, Jan Hendrik Metzen, Rylee Grace, David Salinas, Arthur Cahu, Simon Bing, Benjamin Jäger, Tuana Çelik, Mihir Manium, Vitor Monteiro, Jake Robertson, Jerry Chen, Eliott Kalfon, Tomás Pereda, Lilly Wehrhahn, Dominik Safaric, Tobias Schroeder, Georg Grab, Diana Kriuchkova, Clara Cornu, Philipp Singer, Nick Erickson, Vahid Balazadeh, Marie Salmon, Simone Alessi, Kürşat Kaya, Philipp Jund, Léo Grinsztajn, Yann LeCun, Bernhard Schölkopf, Madelon Hulsebos, Lennart Purucker, Sauraj Gambhir, Frank Hutter, Noah Hollmann
cs.LG
2026-08-17
Prior Labs open-sources RelArena-alpha, a 21-task relational benchmark, and TabPFN-Rel, which flattens a database into a single table for TabPFN-3 and tops the model leaderboard at Elo 1821.
Relational learning covers predictions over multi-table databases: churn, ad clicks, trial outcomes, where the signal sits across orders, reviews and clicks linked by foreign keys. The subfield has plenty of methods but no trustworthy comparison. Prior Labs counts the costs: evaluation regimes drift with benchmark versions, and after RelBench fixed a temporal leak in rel-event/user-ignore in February 2026, affected methods lost two to six ROC AUC points on re-evaluation, so self-reported numbers across versions do not compare. Tuning is opaque: KumoRFM-2's released configurations alone span a 384-combination grid, and RelGNN's checkpoints vary along nine dimensions (25,000+ combinations), with no documentation of how final settings were picked. Data states even differ between methods evaluated on the same task.
Prior Labs (the TabPFN company, with Yann LeCun and Bernhard Schölkopf as scientific advisors) ships three open-source artifacts. RelArena-alpha is a unified framework over RelBench v1's 21 entity-level forecasting tasks across seven databases. It standardizes data loading, evaluation and tuning, splits submissions into models (standardized tuning) and systems (custom tuning, following TabArena), re-runs every method itself, and rates them with bencheval's Elo anchored at 1000 for a global constant predictor (a 400-point gap implies roughly 91% win probability), with bootstrapped confidence intervals. TabPFN-Rel is a relational harness for the TabPFN-3 tabular foundation model: deep feature synthesis aggregates along foreign-key paths into one flat table (depth tuned within 2-4), and TabPFN-3 predicts in-context. Over RDBLearn it fixes tuning-time data drift (the database is frozen at the validation cut-off), feeds TabPFN-3 ten times more rows, re-attaches text columns (API variant only), and replaces random context subsampling with a recency-diversity trade-off. RPI is a declarative YAML interface for defining prediction tasks on your own CSV or Parquet database and running any RelArena model in two lines of code.
Model leaderboard Elo (standardized tuning, 21 tasks, single seed): TabPFN-Rel (API) 1821, TabPFN-Rel (OSS) 1706, GraphSAGE 1658, RelGT 1575, RDBLearn 1548, RelGNN 1506, per-entity constant 1256, global constant 1000 (anchor). Among system submissions, RT-PluRel from Jure Leskovec's Stanford group tops the combined board at 1861. Dropping text features costs 115 Elo (1821 to 1706), more than the 83-point gap between GraphSAGE and RelGT, concentrated in two tasks. A learning-free constant that predicts each entity's own historical optimum beats RelGNN on four tasks and RelGT on four; only TabPFN-Rel and RT-PluRel beat it on all 21. The single-seed leaderboard took hundreds of wall-clock hours: RelGT averages 511 minutes per task (worst case 2442), TabPFN-Rel averages 76 minutes via API and 12 minutes locally, and RT-PluRel is still 5x and 30x slower respectively, excluding preprocessing. Tuning gains are uneven: RelGNN benefits clearly, most methods do not improve reliably, and TabPFN-Rel's default shallowest depth stays competitive.
This is more evidence that flattening a database into a single wide table competes with specialized relational architectures on real tasks: a tabular foundation model now heads the relational leaderboard. RPI makes the recipe tryable on your own database without touching RelBench's Python abstractions. And the subfield gets a TabArena-grade common ground, so copying stale self-reported baselines now has a visible alternative.
Single seed, and tuning standardization is explicitly unsolved: runtime variance across methods can favor expensive ones. The authors flag missing automated feature-engineering baselines, unexamined data quality, near-zero safeguards in RPI against task mis-specification, and alpha status overall. There is also a disclosed conflict of interest: the lab ships both the benchmark and its top-ranked model, mitigated by open-sourcing everything for scrutiny. Text capability lives only in the API variant, so the 115-point text bonus requires sending your data to the TabPFN API.