2026-09-24
Nature's TabPFN, pretrained on ~100 million synthetic tables, beats 4-hour-tuned GBDTs in 2.8 seconds on datasets with up to 10,000 rows.
Tables are the default supervised format in science and business; 76% of openml.org datasets then had fewer than 10,000 rows. Gradient-boosted trees have owned this regime for two decades. Deep nets struggle with mixed columns, missing cells, imbalance, junk features, and outliers. Trees also retrain from scratch on every new table and do not pass gradients into a larger net.
A 2023 TabPFN prototype showed that in-context learning can classify small tables, with tight limits: small n, no regression, brittle to junk features and outliers. This Nature paper turns the same idea into a usable tabular foundation model for up to about 10,000 rows and 500 features, classification and regression included.
The algorithm is learned. The prior is a structural causal model: sample size, feature count, and graph complexity, then build a DAG. Edges can be tiny nets, discretizers, or decision trees, plus Gaussian noise, Kumaraswamy warping, and quantization. Missingness, junk columns, categoricals, and outliers are baked into the generator so the model has to invent remedies. Each checkpoint sees on the order of 100 million synthetic tasks. Pretraining ran two weeks on eight RTX 2080 GPUs, once.
Each cell is a token. A layer attends across features in a row, then across samples of the same feature, for 12 layers, then an MLP. The design is invariant to row and column order and can extrapolate beyond the pretraining sizes. Test rows cannot attend to each other, only to the training set, so training keys and values can be cached: fit once, predict many times. On 10,000×10 tables, core inference speeds up about 300× on CPU and 6× on GPU versus recomputing. Half-precision LayerNorm, FlashAttention, and activation checkpointing drop memory below 1,000 bytes per cell, enough for about 50 million cells on one H100.
Regression uses a piecewise-constant Riemann head, which can represent multimodal targets. A TabPFN-only post-hoc ensemble, TabPFN (PHE), is also reported.
The main suite takes 29 classification and 28 regression datasets from the AutoML Benchmark and OpenML-CTR23 with at most 10,000 rows, 500 features, and 10 classes, 90/10 splits, 10 seeds. Baselines: random forest, XGBoost, CatBoost, LightGBM, SVM, MLP, linear models, tuned for up to 4 hours.
Default classification, normalized ROC AUC: TabPFN 0.939 versus CatBoost 0.752. After 4 hours of tuning: 0.952 versus 0.822. Default regression, normalized negative RMSE: 0.923 versus 0.872; tuned: 0.968 versus 0.875. Default TabPFN at 2.8 s (classification) and 4.8 s (regression) beats every baseline tuned for 4 hours, about 5,140× and 3,000× wall-clock. The Wilcoxon p-value versus default CatBoost is below 0.001.
Against AutoGluon 1.0, default TabPFN at 2.8 s already wins versus a 4-hour AutoGluon run. TabPFN (PHE) lifts normalized ROC AUC from 0.939 to 0.971; AutoGluon sits at 0.914. On regression, PHE at a 300 s floor beats 4-hour AutoGluon, about 48×. With half the rows, TabPFN still matches the next-best method that saw every row. Junk-feature and outlier stress tests do not hurt it more than trees; MLPs drop.
Qualitatively, one default checkpoint fits both smooth curves and step functions. A double-slit intensity pattern comes out as a multimodal density in one pass; CatBoost needs a stack of quantile models and still looks worse.
On small-to-medium tables, "train GBDT overnight" is no longer the default. Pretrain once, forward-pass a new table, and keep a predictive distribution. A synthetic prior sidesteps copyright and test-set contamination, a real difference from language foundation models. Density estimation, row generation, reusable embeddings, and fine-tuning all have proof-of-concept numbers. This is more than a classifier.
The authors are explicit: past 10,000 rows, or on very non-smooth large regression, CatBoost, XGBoost, or AutoGluon can still win. TabPFN is one tool, not a replacement for a data scientist.
The supported box is 10,000 rows, 500 features, 10 classes; larger tables have no headline numbers. Per-sample latency is not small: 0.2 s on GPU and 0.6 s on CPU to predict one row of a 10,000×10 table, versus about 0.0002 s for default CatBoost. Caching helps repeated use of a training set, not one-off latency. Memory grows linearly in cells. The 4-hour baselines ran on eight CPU cores; TabPFN also used a 2080 Ti, so speedup folds in hardware. PHE versus AutoGluon starts at 300 s because AutoGluon may not finish faster. Generation and fine-tuning are demos, not product features.