Data Cleaning Debate: The Necessity of Filtering in Text Training
code_star · x · 2026-08-24
Responding to views from the vision domain, the author argues that filtering is critical in text training where junk data can constitute 90% of the volume, making unfiltered Common Crawl training impossible. The author asserts that removing obvious bad data ("whack-a-mole") is essential and suggests that if filtering appears ineffective, it is likely because the evaluation set itself is broken and also requires filtering.
Related event: Filtering vs. Selection: Debating Training Data Preprocessing Strategies(5 posts)→
More from Infra
- Copper to optical: optical backplanes seen as necessary as racks move to optics — BenBajarin · 2026-10-16
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-16
- Chinese labs run on ~1/10 US compute; ByteDance pays 2-4x to hoover up SEA supply — robleclerc · 2026-10-16
- Zymtrace raises $12.2M and hires ML engineers to build agents that fix GPU bottlenecks — cfregly · 2026-10-16