Resolution-Inspired Pretraining Experiments 'Pretty Much a Disaster,' Researcher Says
gordic_aleksa · x · 2026-09-19
Aleksa Gordić and giffmana discuss the pretraining data bottleneck: giffmana ran experiments inspired by resolution scaling in vision, but says most were 'pretty much a disaster.' Gordić argues the deeper issue is the assumption that pretraining isn't data-limited, while noting promising signal/flop results.
More from Research
- OpenADMET lands Gates Foundation funding to build open toxicity data for drug discovery — anshulkundaje · 2026-09-19
- Ken Goldberg's Agentic Robotics: AI writes deterministic robot code, no demos needed — animesh_garg · 2026-09-19
- Researchers question antibody model's SabDab training data extending to 2025, flagging test contamination — anshulkundaje · 2026-09-19
- Yale PhD quit after lymphoma diagnosis to build foundation models for antibody drugs at Aureka Bio — anshulkundaje · 2026-09-19
- WetRobo: A Reproducible Robot Kit Lets Coding Agents Run Real Wet-Lab Experiments — sherryyangML · 2026-09-19
- NanoGPT Speedrun Record Falls to 67.6s With Canonical Token Masking (-15 Training Steps) — kellerjordan0 · 2026-09-19