LLMs Can Program Tabular Foundation Models Without Gradient Descent, Shows New Paper
RexDouglass · x · 2026-11-17
The arXiv paper 'Weights & Caches' (Hoo, Buehler, Swelam, Hutter et al.) shows LLMs can build, repair and steer tabular foundation models (TFMs) by directly manipulating weights — no gradient descent. Using GPT-6 Astra across three settings:
- Construct: writes nanoTabPFN from zero-initialized weights, scoring mean ROC AUC 0.849 across 20 TabArena datasets (pretrained baseline: 0.876);
- Repair: diagnoses and fixes a signed-product regression failure in pretrained TabPFN-3.5, restoring near-perfect predictions (median R² ≈ 0 → 0.99) with only 0.08% average relative error increase on TabArena-Lite;
- Steer: writes KV-cache-editing programs that alter TabPFN-3.5's predictions on unseen data (e.g., reversing linear trends, converting sine waves to square waves) with weights frozen.
The authors argue LLM-driven weight programming could complement gradient-based optimization across TFM development and usage.
More from Research
- Physicist pans AI "superconductivity breakthrough" as repackaged 1975 result — ZeeshanZiaML · 2026-11-17
- Studying retrosynthetic disconnection from Vogel's textbook: "really freaking cool" — CatAstro_Piyush · 2026-11-17
- AI drives 30% surge in arXiv theoretical physics submissions; models solve ~60% of private-problem benchmark — binarybits · 2026-11-17
- FrontierCS 2 benchmark challenges AI agents to beat published CS research blind — arena · 2026-11-17
- Statistician: Three Systemic Incidents in One Month Is Enough to Reject Low Base Rate (p=0.002) — wfithian · 2026-11-17
- Isomorphic Labs raises $5B, its second mega-round in six months — The Decoder · 2026-11-17