GPU-Free Activation Alignment Recovers Half of Full-Context Performance for Tabular ICL
Independent-Researcher · hf · 2026-10-07
The paper tackles the cost of in-context learning (ICL) in tabular foundation models, which must process all training examples on every forward pass; restricting context saves compute but hurts accuracy.
The authors propose activation alignment: a lightweight linear transformation, trained on synthetic unlabeled data, maps the intermediate activations of a partial-context "student" toward those of a full-context "teacher". Training requires no GPU and converges in seconds to minutes on commodity hardware.
Across 38 classification datasets from TabArena using TabPFN-3 and TabFM, the aligned student yields statistically significant improvements at all context budgets, recovering nearly half of the teacher's advantage in low-data regimes.
More from Research
- PUMBA Paper Aligns Masked Diffusion LM Training With Inference Trajectories — kastnerkyle · 2026-10-07
- Anders Sandberg: co-writing papers with an LLM makes preregistering predictions painless — anderssandberg · 2026-10-07
- Prediction: zeroth-order optimization methods may eventually replace backprop — j_foerst · 2026-10-07
- Berkeley's Workhorse trains humanoid G1 for whole-body manipulation from human data only — pabbeel · 2026-10-07
- François Fleuret: my fancy new layer got crushed by good old algorithmic tricks — francoisfleuret · 2026-10-07
- Epoch AI launches Capabilities Index (ECI), a unified scale for comparing model intelligence over time — ricklamers · 2026-10-07