Datology AI builds synthetic pretraining data with measurable knowledge cutoff guarantees
josh_wills · x · 2026-10-30
Datology AI highlights a silent backtesting killer: when a model rephrases documents, knowledge cutoff guarantees are lost and information its sources never had can slip in.
The team built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees. Insider detail from parthsur: every iteration over the baseline caught leaked claims so much more precisely that they had to grow the measurement set by an order of magnitude each round just to find survivors — and still came up empty at the end.
More from Research
- Self-Direction turns agent mistakes into LoRA lessons, lifting GAIA success from 38.8% to 55.8% — rohanpaul_ai · 2026-10-30
- Preprint: AlphaGenome pervasively underestimates causal variant effects, undermining fine-mapping — anshulkundaje · 2026-10-30
- One Researcher on 22 NeurIPS 2026 Papers: What Should Author Lists Mean? — qberthet · 2026-10-30
- Stanford's I3L boosts whole-body robot deployment success by 68.3 points — jiajunwu_cs · 2026-10-30
- Dev ditches unanimous-vote stopping rule for a judge in multi-AI debate project — lluisantoni · 2026-10-30
- Google's AI Search Uses Your Knowledge Graph Facts Without Linking, RESONEO Research Finds — dejanseo · 2026-10-30