LSH-MinHash Dedup May Discard Millions of Valid Training Documents
rupspace · x · 2026-08-13
MBZUAI research points out that LSH-MinHash deduplication, commonly used in LLM pre-training, might mistakenly discard millions of valid training documents.
The Culprit: The method treats overlapping duplicate buckets as transitive equivalence and keeps only one document per connected component, ignoring that local overlap does not equal global duplication.
More from Research
- Hugging Face Launches 'Hugging Science' Hub with Open Datasets and Interactive Demos — huggingface · 2026-08-13
- AI Solves 4th Epoch AI Math Problem: Constructs Hadamard Matrices — burny_tech · 2026-08-13
- Bridge Editing Published in Science: Precisely Writes Massive Changes into Human Genome — chaitjo · 2026-08-13
- Harvard PQG 2026 Conference to Explore Multimodal Genomic and Health Data with AI — lihua_lei_stat · 2026-08-13
- Autoware Releases Open-Source E2E L2 ADAS Stack Requiring No GPU or LiDAR — 4310sy · 2026-08-13
- Founder Uses ChatGPT to Cure Dog's Cancer, Launches YC-Backed Gamgee — DeryaTR_ · 2026-08-13