TWIML episode argues future foundation models may learn from existing model weights
samcharrington · x · 2026-07-28
Models as data may become the next training paradigm
This TWIML episode features Damian Borth arguing that foundation model progress may be constrained by shrinking access to high-quality pretraining data and rising training costs.
Core idea
- His group’s weight space learning treats trained neural networks as a new data modality.
- Instead of starting from raw text or images each time, models can learn from the distilled knowledge contained in existing model weights.
Why it matters
- The approach could make it cheaper to build specialized models.
- It also raises the possibility that future foundation models may be trained on collections of existing models, not just ever-larger datasets.
Mentioned resources
- GeoSANE: learning geospatial representations from models, not data
- Towards Scalable and Versatile Weight Space Learning
- Neural Network Weights as a New Data Modality
- Predicting Neural Network Accuracy from Weights
More from Research
- Small-model orchestration roughly doubled task completion in a 100-task benchmark — _raydeStar · 2026-07-29
- Paper adds a human-only authorship attestation to a quantum matrix result — burny_tech · 2026-07-29
- Biohub is hiring for an AI wet-lab role to build biology models — proteinrosh · 2026-07-29
- An AI digest scans 92 journals every week and turns them into one RSS feed — Afinetheorem · 2026-07-29
- A weekly PDB-synced leaderboard tracks open cofolding models — rishabh16_ · 2026-07-29
- Agentic AI Summit sets robotics and world models session with Sergey Levine and Jim Fan — dawnsongtweets · 2026-07-29