Procedural Core: a 1M-param abstract model boosts 85M-param ViT ImageNet accuracy by 2.2 points
kastnerkyle · x · 2026-10-14
Zachary Shinnick et al. introduce Procedural Core: they train a tiny 1M-parameter recurrent model on abstract procedural data — never seeing a single real image — then expand its weights into the initialization of an 85M-parameter ViT, lifting ImageNet-1k top-1 accuracy from 77.6% to 79.8% in their main setting.
Retweeter Damien Teney calls it a nice application of the recent line of work on pre-pretraining with synthetic/procedural data: learn generic abstract structure in a small looped model, then transfer it as a better generic initialization for larger transformers — applicable to both ViTs and language models.
More from Research
- Google Fellow John Platt: AI rerouting cut flight contrails by 62% — ziv_ravid · 2026-10-14
- Bible verse continuation triggers malevolent outputs, more evidence of subliminal learning — sebkrier · 2026-10-14
- ReRoute: physics-informed fine-tuning cuts counterfactual error up to 89.6% — AnimaAnandkumar · 2026-10-14
- Spork post-training recipe lifts chess puzzle solve rate 100% over best GRPO setup — aidenybai · 2026-10-14
- Dyna-2 Deep Dive: World-Action Models Trained on 1M Hours of Human Video, 99.4% 24-Hour Autonomy — Latent Space · 2026-10-14
- Nature Will unveils Natura-1: 50B-parameter MoE robot foundation model trained on 1M hours of egocentric human video — rohanpaul_ai · 2026-10-14