RT-J paper details: 8,192-cell contexts, 10x fewer examples to match feature pipelines
christopher · x · 2026-10-18
Christopher shared the arXiv abstract (2610.17867v3, NeurIPS 2026) for RT-J by Rishabh Ranjan et al. (Stanford + SAP).
Key points:
- Existing tabular/relational foundation models need tens of thousands of examples to adapt to unseen datasets; RT-J works with hundreds of in-context examples or fewer.
- THE JOIN corpus: 639 web-crawled databases, 6,023 LLM-generated forecasting tasks.
- Pretraining uses contexts up to 8,192 database cells, a lightweight random-walk retriever for in-context examples, and context tuning + ensembling to scale test-time compute.
- On RelBench in-context regression, RT-J needs 10x fewer examples on average to match strong pipelines (LLM-agent/RDBLearn featurizers, TabICLv2/LightGBM predictors).
More from Research
- Claude Opus simulated Hox gene networks to generate lizard and snake skeletons — repligate · 2026-10-18
- OpenEnv Arena lets you build RL environments without GPUs, one training run per 24h — ben_burtenshaw · 2026-10-18
- French math PhD students despair: a €20 ChatGPT could solve their thesis problems — FlorianGallwitz · 2026-10-18
- 'Idea Basins': independent runs of the same task plateau at different scores compute can't close — menhguin · 2026-10-18
- NVIDIA researcher counters linear attention debate: 8B hybrid beat Transformer 7 months before MiniMax-01 — ctnzr · 2026-10-18
- ERRORPROBE: automatically pinpointing which agent caused a multi-agent failure — burkov · 2026-10-18