TabFM: A Zero-Shot Foundation Model for Tabular Data
Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, Abhimanyu Das
cs.LG
2026-09-30
Google's 400M TabFM turns tabular prediction into one-pass in-context learning, trains only on SCM-generated tables, and zero-shot beats tuned AutoML on all 51 TabArena datasets.
Supervised tabular work still starts from scratch on every dataset. Gradient-boosted trees scan for splits; AutoML stacks models under a time budget. Axis-aligned splits handle mixed types, missing cells, and messy scales, which is why XGBoost, LightGBM, and CatBoost still beat per-task neural nets. None of that work transfers. A new table means new preprocessing, new split searches, and new hyperparameters.
Prior-data fitted networks (PFNs: models trained on synthetic priors so one forward pass approximates a Bayesian predictor) recast the job as in-context learning: the network reads labeled training rows plus unlabeled queries and approximates p(ytest | xtest, Dtrain). TabPFN showed this works on small tables. Full attention over every cell is O(T²H²), so context stuck around 1,000 rows. Rows should be permutation-equivariant; columns mix continuous values that need numeric resolution with unordered categories. Tabular transformers never amortized learning the way language and vision models do.
TabFM is a 400M Transformer from Google Research, released as tabfm-1.1.0. Four stages: Fourier cell embeddings, alternating column and row attention, CLS pooling, and a 24-layer in-context predictor.
At the cell layer, dyadic grouping ties column j to neighbors at offsets 0, 1, and 3. Each slot runs the scalar through 32 learned Fourier frequencies into a 64-dim sin/cos vector, with separate banks for numeric and categorical columns. Continuous values need a metric scale; categories only need to be distinct.
Column attention treats the T rows as the sequence. An ISAB with 256 inducing points compresses a column of rows into 256 summaries then reads them back, cutting cost from O(T²) to O(Tm), which is enough to estimate per-column marginals. Row attention treats the H features as the sequence, with RoPE only on the feature axis, so rows stay exchangeable and column count is not baked into a position table. The encoder is two untied stages of three column ISABs plus three row SABs. After the first column stage, eight CLS tokens are prepended and concatenated into a 1024-dim row vector.
The 24-layer predictor at width 1024 holds 402.7M parameters and does the cross-row implicit optimization. A context mask lets test rows attend only to labeled training rows, so queries stay conditionally independent. Loss is computed only on query rows. Pretraining context goes to 16,384 rows and 100 columns.
All pretraining tables come from structural causal models: random DAGs, nonlinear maps, plus joint randomization of shape, categorical share, missingness, label noise, and class balance. A four-stage curriculum doubles rows from 2,048 to 16,384 while holding tokens per step at about 4.19M. Each synthetic table carries both a classification and a regression target.
Two inference wrappers freeze the same weights. TabFM+ runs K=32 views: half on the raw columns (up to 500, past the 100 seen in pretraining), half with up to sqrt(H) random cross products and sqrt(H) truncated SVD components, plus column permutations and Platt scaling (a logistic map from scores to probabilities). Non-negative least squares mixes members, shrunk 25% toward uniform. TabFM-Auto has Gemini-3.8-Flash write a Python feature pipeline around the frozen model, starting from the identity program, scoring with 3-fold CV inside the first training split, stopping at 96 trials or six hours. The sandbox never sees test rows.
TabArena covers 51 datasets (38 classification, 13 regression) under repeated 10-fold CV. 67 configurations share one Bradley-Terry Elo fit, with default Random Forest anchored at 1000.
| Method | Cls Elo | Reg Elo | Cls g-mean | Reg g-mean |
| TabFM-Auto | 1940.7 | 2392.4 | 0.0921 | 15.96 |
| TabFM+ | 1838.0 | 2189.2 | 0.0948 | 16.17 |
| TabFM zero-shot | 1768.6 | 2055.2 | 0.0957 | 16.40 |
| EXAONE-Tabular | 1768.0 | 1973.1 | 0.1009 | 16.58 |
| AutoGluon 1.5 extreme | 1669.7 | 1851.2 | 0.1010 | 16.78 |
| TabPFN-3 | 1641.9 | 1866.6 | 0.1050 | 16.51 |
On the pooled 51-dataset board, TabFM-Auto sits at 1984 Elo, TabFM+ at 1861, and zero-shot TabFM at 1785.4, first among default foundation models. Zero-shot TabFM wins 65.7% of dataset-folds against EXAONE-Tabular, 72.1% against AutoGluon 1.5 extreme, 75.9% against TabPFN-3, and 79.8% against TabICLv2.
The regression gap is larger: 2055.2 Elo, about 82 above EXAONE and 189 above TabPFN-3. On classification, TabFM and EXAONE differ by 0.6 Elo. The real edge is lower regret: oracle improvability is 6.10%, against 9.47% for EXAONE and 10.00% for AutoGluon 1.5 extreme.
TabFM+ adds 69.4 / 134.0 Elo on classification / regression. TabFM-Auto adds 172.1 / 337.2, improves 41 of 51 datasets, and cuts relative error by more than 10% on ten of them. All 13 regression datasets improve, and improvability hits 0.00%. When the synthesized program actually rewrites the table, median error drop versus zero-shot TabFM is 2.14%; when it does not, 0.27%. On the ten classification datasets that do not improve, test error rises by at most 1.8%.
For a table of up to about 16k rows and a modest column count, one forward pass can replace a 4-hour AutoML search. Weights are public. Regression is where this is most worth using.
The top four classification methods fall inside a 130-Elo band, and zero-shot TabFM is 0.6 Elo from EXAONE, so this paper does not show that tabular foundation models have put tree ensembles away. The contribution is an engineering scale-up of the PFN line: a deeper in-context stack, linear-time column attention, and a synthetic causal prior, moving ICL tabular models from about 1,000-row context to 16,384 rows and past 4-hour AutoML on a public leaderboard.
Pretraining only saw numeric and categorical synthetic tables of at most 16,384 rows and 100 columns. Larger tables fall back on length generalization or subsampling. Free text has no semantic tokenization. Relational databases still need a manual join. The paper lists million-row and thousand-column pretraining, real tabular corpora, text and temporal encoders, and relational schemas as next work.
Whether the synthetic prior covers the long tail of real tables is unanswered; there is no ablation that pretrains on real corpora. Regression uses only 13 datasets, so large Elo gaps can be noisy. TabFM-Auto's budget is 96 trials or six hours per dataset, which is not the same compute as 4-hour AutoGluon. Features such as Strouhal and Reynolds numbers on airfoilselfnoise, or ICD-9 chapters on Diabetes130US, come from Gemini's domain knowledge. The paper does not split that gain from TabFM itself.
A 0.6 Elo classification edge over EXAONE is not a win you should read off the overall board.