Data curation has massive blindspots: Transformers learn co-occurrence geometry, not features
BlackHC · x · 2026-09-10
The author argues current data curation has massive blindspots: it ignores token isomorphism structures and imports old neural-network ideas never tailored for transformers, treating curation, loss, and 'more data' as generic ML problems. Transformers learn co-occurrence geometry under repetition, and that geometry is the object of study; evaluation criteria should be chosen based on anticipated downstream usage.
More from Research
- Jacobian Lens explained: a 'lie detector' for reading the mind of an LLM — huopak · 2026-10-27
- Preprint reframes enzyme function prediction as conditional generation with Fluxion — LeoTZ03 · 2026-10-27
- No bins needed: new method estimates model calibration error as a drop in risk — BlackHC · 2026-10-27
- AnyRank ranks robot policies reliably with far fewer real-robot trials — pappasg69 · 2026-10-27
- Cohere Labs releases Qwen investigation traces dataset with repro script — cohere · 2026-10-27
- OpenAI claims models helped falsify the Exponential Time Hypothesis — Environmental_Mix22 · 2026-10-27