Freeze TabFM and Search the Pipeline: TabFM-Auto Hits 2013 Elo, Sweeps TabArena Top Five

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong

cs.LG

2026-09-30

LLM agents rewrite the pipeline around frozen 400M TabFM. Five configs take TabArena's top five (2013 Elo vs 1785); pipelines transfer to other TFMs with no extra search.

What problem this solves

Tabular foundation models pretrain a transformer on synthetic tables, then predict on a new table in one frozen forward pass. TabPFN, TabICL, and TabFM already match or beat tuned gradient-boosted trees. Attention still only sees magnitudes and category indices. A column named systolicbp is handled like anonymous X17; a 0 that means missing looks like a real reading. TabFM+ adds pairwise crosses, SVD projections, and multi-view ensembling, all generic ops that never read column names, task text, or auxiliary files.

MLE agents take the other fork. An LLM edits a full training script and jointly searches features, architectures, and hyperparameters. The search is noisy: training variance hides small feature gains and invites validation overfitting. LLMs understand schemas. They are weak tabular predictors because of context limits and coarse number tokenization. Each side is missing what the other has.

Method

TabFM-Auto freezes the predictor and lets the agent rewrite only the data pipeline. The backbone is a frozen 400M TabFM from Google Research. Each candidate is four Python hooks plus a constructor dict, TABFMKWARGS:

• preprocess(): recode sentinels, drop dirty rows, apply invertible target maps such as log(1+y)
• engineer(): turn metadata into columns. Domain formulas, group stats, and auxiliary-file summaries are inductive; frequency, co-occurrence, and bipartite degrees over concatenated train+test rows are label-free transductive features
• sample(): choose in-context rows. TabFM was pretrained up to 16,384 rows, and uniform subsampling on large or imbalanced tables drops rare classes, so the agent builds stratified or oversampled views and ensembles them
• postprocess(): invert the target map for regression; shift predictions toward the training prior, or apply temperature / Platt scaling, for classification

TABFMKWARGS exposes the generic knobs TabFM+ already has: normalization, crosses, SVD, non-negative least-squares view weights. The agent tunes those when they suffice and writes code only for formulas the presets cannot express. That split is why the same pipeline can be dropped onto another TFM.

Search starts from the identity pipeline P0. The agent runs 3-fold CV on fold 0's training split only, for up to 96 evaluations or 6 hours on one H100. Candidates run in a network-off sandbox. Official test folds are scored once after search freezes. Five configs cross Claude Code, Codex, and Antigravity with Claude Opus 5 and Gemini 3.8 Flash.

Each trial is a forward pass. Retraining noise cannot mask a small feature edit.

Results

On all 51 TabArena datasets (38 classification, 13 regression), the five TabFM-Auto configs take the top five overall spots. Codex + Opus 5 lifts TabFM from 1785.3 to 2013.0 Elo. Against 4-hour AutoGluon 1.5 (extreme) at 1668.4, the five configs sit +272 to +345 Elo.

All 13 regression tables beat TabFM. The paper's account: once physical ratios are written as columns and skewed targets are compressed, TabFM can interpolate a smooth function that trees only carve with coarse splits. On classification, Codex + Opus 5 has the best Elo; Antigravity + Gemini 3.8 Flash has the lowest G-Mean (0.0902) and the most dataset wins (16.11).

An unconstrained agent with the same Antigravity + Gemini 3.8 Flash harness and 6-hour budget, free to train any GBDT or net, scores 1468.8 Elo. That is slightly below 4-hour AutoGluon 1.3 (1490.3) and 510.8 below frozen-TabFM search (1979.6). TabFM's prior is worth +316.5; pipeline search adds another +194.3.

In 97 of 102 runs the agent changes the feature table. On 17 datasets whose headers name physical, clinical, or economic quantities, test error drops 7.3%-8.2%, versus 2.3%-3.0% on anonymized schemas. On airfoilselfnoise the agent writes Strouhal numbers from the column names and cuts RMSE from 1.073 to 0.888. On Amazonemployeeaccess, 53 label-free graph features cut 1-AUROC from 0.1416 to 0.1063. From P0 to full budget, official all-fold test error falls 4.0%-4.7%; the first 2.5% of the budget (about 3 evaluations) already beats TabFM+.

On the 8 tabular competitions of MLE-Bench, 24-hour budget, Claude Code + Opus 5 ranks first at 1827 Elo with 4 Kaggle medals and an 86.7% pairwise win rate against 12 external agents. Antigravity + Gemini 3.8 Flash is second at 1734. Where the signal lives in auxiliary files (seismic waveforms, satellite telemetry, 3D coordinates), the agent summarizes them into table features before calling TabFM.

Why it matters

For tabular work this removes the noise of retraining a model every time a feature changes. Gains are largest when column names mean something. Pipelines found for TabFM also run on TabPFN-3, TabICLv2, and EXAONE-Tabular with no extra search, so one search can serve a family of in-context models.

This is a data-side increment on an existing TabFM. Anonymized tables get thinner gains. Six hours on an H100 plus billions of prompt tokens is not casual AutoML.

Limitations

The paper flags two limits. Anonymized tables leave the agent with statistical and relational features only, so gains shrink. Every dataset needs repeated validation, so search is expensive: Codex + Opus 5 burns 1.95B prompt tokens across 51 tables, Codex + Gemini 3.8 Flash 12.13B. The identity pipeline P0 also fails outright on three tables with more than 500 columns; wide tables need the agent to cut features first.

Pipelines are searched on fold 0's training split, then frozen for all official folds. Test isolation is strict, but search-on-one-fold, test-on-all can still tilt toward fold 0. Regression Elo jumps to 2512.9 with 0.00% improvability on only 13 tables; the overall Elo 95% bootstrap interval is already around ±100. The unconstrained ablation used only Gemini 3.8 Flash, so freezing the TFM is established for that one harness pair. On MLE-Bench, TabFM-Auto is not first on every competition; overall Elo is an aggregate, and external submissions differ in compute and model generation.

Terms

• Tabular foundation model (TFM): a transformer pretrained on synthetic tables that predicts on a new table in one frozen forward pass.
• In-context learning (ICL): train rows go into the context window; one forward pass emits predictions. Here the prompt is a numeric table.
• TabArena: a living 51-dataset tabular benchmark (38 classification, 13 regression) scored with Bradley–Terry Elo, default Random Forest anchored at 1000.
• MLE agent: an LLM that edits training code and searches over features, models, and hyperparameters.
• Oracle improvability: mean relative error gap to the per-dataset best model; 0% means the method has matched that best.

Source

• arXiv 2609.37989v1
• PDF
• Hugging Face

What people are saying

• Google's TabFM: a zero-shot foundation model that predicts table data in one forward pass Prompt Engineering

Related papers

• Public maps attach all 7,330 Alna buildings, and inferred lines outrun the known grid
• Two empowerment encoders separate control from noise, with a 1,000× sensitivity gap
• Heavy-tailed data and gradient noise drive LLM loss spikes; a convex toy model predicts them
• Test-time ridge regression: +12 RULER points, 55.6% fewer visual tokens
• Muon-trained Moonlight gains 7.9 HumanEval points over its AdamW twin at 1.2T
• Parallax: a learned covariance branch cuts pretraining perplexity 17%, but only under Muon

All paper explainers