Why Large Language Models Fail at Tabular Prediction
Marta Garnelo, Wojciech M. Czarnecki
cs.LG
2026-08-03
Five causes tested for a frontier LLM's tabular failure; only dimensionality holds: past 16 dimensions accuracy collapses to majority guessing while classical baselines stay flat.
LLMs draft text, write code, and answer questions with ease, yet they keep losing at one of the oldest jobs in machine learning: read the training rows of a numeric table and classify the test rows. They lose to k-nearest neighbours, logistic regression, and decision trees, baselines that predate the Transformer by decades. The gap is large enough to have spawned a subfield, tabular foundation models (TabPFN, Nexus, TabICL), built on the premise that generic LLMs simply cannot do tables. That premise is "usually asserted, not explained." This paper supplies the explanation, asking in the cleanest possible setup what exactly breaks.
The authors push the model into its purest form: one prompt carrying the whole table (training rows plus test rows, as CSV), one generation that emits every prediction. No custom system prompt, no tool calls, no code execution, no multi-turn dialogue, no fine-tuning. What gets measured is the bare model. The primary model is claude-opus-4-6, with Qwen3-235B-A22B as a two-dimensional cross-check.
Then five failure hypotheses, each with a controlled experiment designed to falsify it:
Thirty-one datasets: 11 real tables (UCI, scikit-learn, after a memorisation filter) plus 20 synthetic 2D probes. The memorisation filter holds out every row of one class; classical methods score zero there by construction, yet the LLM reproduced the original labels for breast cancer and iris almost perfectly (and banknote, wine substantially), so those memorised tables were excluded. The comparison set is 8 classical baselines (1/5/10-NN, random forest, logistic regression, AdaBoost, gradient boosting, Gaussian process) with the majority class as a normalisation floor, plus a sweep of 252 configured classical models ranked by pointwise prediction agreement with the LLM. Evaluation is 5-fold cross-validation over 5 seeds.
The first four hypotheses are falsified; only dimensionality holds:
| Hypothesis | Manipulation | LLM trend | Verdict |
| Separability | widen class gap in steps | slope +0.089 (logistic regression +0.368) | Falsified |
| CSV format | hide target column in features | r = -0.17, near-flat | Falsified |
| Numeric tokenisation | round to 1/4/8 decimals | r = -0.02 | Falsified |
| Per-query load | full / half / quarter | normalised 0.422 / 0.412 / 0.407 | Falsified |
| Dimensionality | project across powers of two | slope -0.009/dim, r = -0.21 | Supported |
The dimensionality effect is clean: of the nine methods, the LLM is the only one whose accuracy drops as dimensionality grows, at -0.009 per dimension, and by d > 16 its normalised score falls to or below majority-class guessing. The remaining classical baselines stay flat or improve, with slopes between 0 and +0.012.
The behavioural portrait splits in two. In two dimensions the LLM acts like a local, distance-based classifier: the closest baseline is a standardised Gaussian process with a Matérn kernel (length-scale 1, nu 1.5), agreeing on 91.6% of grid cells; 1-NN is at 91.0%; the top five are all GP variants. The same distance-based match holds for Qwen, so the finding generalises across models. In high dimensions the picture inverts: the closest classical model reaches only 64.8% agreement, and the entire top 30 is compressed into 64.0% to 64.8%. Adding dimension-dependent noise to each baseline and refitting lifts agreement by only 0.38 percentage points on average, 0.64 at most, so a noise-degraded classical learner does not reproduce the LLM either.
The authors also ask the model to explain the rule it followed. Across the 20 2D tasks, both predictions and explanation matched the data in 6; predictions matched but the explanation did not in 4; neither matched in 10. Explanations disagreed with the data in 14 of 20, so the verbal rule is not to be trusted.
The direct lesson for practitioners: if your table has more than about 15 numeric columns, no amount of prompt polishing on format, precision, or batching will rescue in-context LLM prediction. All three were falsified. The bottleneck is dimensionality itself, not prompt craft. This also clarifies why tabular foundation models must be a separate architecture: TabPFN and its peers are trained specifically for tables, not adapted from a general LLM through prompting.
More pointedly, the paper narrows a vague "LLMs are bad at this" into one precise statement: capability dissolves with dimension, and no noise-corrupted classical learner mimics it. That is a concrete, quantitative target for mechanistic interpretability.
The authors name several. The main claims rest on a single model, claude-opus-4-6, with Qwen validating only the 2D case; they explicitly offer no guarantee that future models behave the same. Every conclusion is behavioural: with no access to weights or activations, they matched input-output behaviour without opening the box. The datasets are toy-scale: a 100K-row, 20-column table costs roughly 16 million input tokens, about $80 per attempt and $2,000 for a full evaluation, and the whole study spent about $836 to $941 on input tokens alone, by the paper's two figures. A real production table with millions of rows rules out naive in-context prediction on cost grounds, independent of accuracy.
They also state plainly that they did not identify the internal mechanism. One more caveat from the results: the 91.6% GP match in 2D combined with total non-match in high dimensions points toward a local-distance form of reasoning that does not survive the curse of dimensionality, but the paper does not prove that mechanism. Given that explanations disagreed with the data in 14 of 20 cases, the chain of thought here deserves little trust.