KAN boosts TabPFN scaling forecasts: OOD error cut up to 80% on StepLaw tasks
ZimingLiu11 · x · 2026-10-31
Ziming Liu's team shows KAN-learned coordinate transforms further improve TabPFN-based scaling predictions, with interpretable insights into residual errors.
- Frozen TabPFN takes [q, log10(η/η0), log10(B/B0)] to predict loss; learned learning-rate and batch-size reference curves improve prediction and selection at larger scales.
- StepLaw results: OOD MAE improves 29% (final loss) and 16% (loss curve); selection regret drops from 0.427% to 0.084% (−80%); in-distribution gains are not significant.
More from Research
- Dev asks for a mathematician: Claude's Muon optimizer theory is too deep to verify — vxnuaj · 2026-10-31
- Claude's Muon optimizer theorem checks out numerically to 4 decimal places — vxnuaj · 2026-10-31
- Removing activation bias is a free trick that improves ViT, DiT and GPT training — ZimingLiu11 · 2026-10-31
- Hessian eigenvalue plots show removing activation bias flattens the loss landscape — ZimingLiu11 · 2026-10-31
- Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31
- Biology Isn't Just a Data Problem: The lncRNA ANRIL Breaks Every Model You'd Write — iskander · 2026-10-31