How Tabular Foundation Models Predict: Inside the nanoTabPFN Architecture
pandeyparul · x · 2026-10-06
Parul Pandey's study notes on the nanoTabPFN paper and code, explaining how tabular foundation models (TFMs) differ from traditional ML:
- fit() vs predict(): traditional models learn parameters from training data during fit(); TFMs have frozen pretrained weights, the training set becomes context, rows to predict become the query, and most compute happens in predict().
- Inside predict(): context and query are combined, preprocessed, and encoded, then pass through transformer blocks with feature attention (row-wise feature interaction) and datapoint attention (query attends to training rows), followed by an MLP.
- A separate decoder MLP turns the query's final representation into the prediction.
She notes the idea generalizes to other TFMs and will share more as she progresses.
Related event: Inside nanoTabPFN: Why Tabular Foundation Models Fit Almost Instantly(2 posts)→
More from Research
- CoNLL 2023 paper: instruction-tuned GPT models beat children on Theory of Mind tests — dioscuri · 2026-10-06
- AI formal verification hits reality: the halting problem blocks existing codebases — danbri · 2026-10-06
- GeoCR: one generalist model unifies cloud removal across 10 datasets, no fine-tuning needed — KAIST-VIClab · 2026-10-06
- Pi²: turning Wikipedia tables into long-context reasoning training data (COLM 2026) — tuvllms · 2026-10-06
- New RL simulation environment for port logistics built on real-world port data — yb2698 · 2026-10-06
- Interactive explainer makes TF-IDF, BM25 and NDCG click, BM25 by example — JnBrymn · 2026-10-06