Filtering vs. Selection: Debating Training Data Preprocessing Strategies

A community debate erupted over training data preprocessing for large models: "filtering vs. selection." @giffmana argued the problem should be reframed as "data selection": for generic models, nearly all data becomes useful as compute grows, and naive heuristic filters accidentally remove high-value data (e.g., date-named images), with cited material claiming over-filtering wastes 33% of compute. Unless compute-constrained or training a non-generic model, one should avoid "smart filtering" in favor of selectors targeting specific distributions, focusing on training-data distribution and maximizing diversity.

Confirmed

• @giffmana explicitly distinguishes "filtering" from "selection" paradigms, holding that generic-model training should have no filters and that filtering is poor quality control
• Cited material claims over-filtering costs about 33% of compute
• @codestar objects to extrapolating vision-domain conclusions to text: text data has an extremely high garbage ratio (e.g., 90%), and training on unfiltered raw Common Crawl is infeasible
• @codestar maintains that "whack-a-mole" removal of obviously bad data is necessary
• @codestar adds that unless a filter is badly designed, filtering "not working" in text training most likely means the evaluation set itself is seriously flawed—eval data and its solutions likewise need cleaning

Unconfirmed

• The experimental conditions and source of the 33% compute-loss figure are not given in the posts
• The statistical basis for the 90% garbage ratio is unclear

Why it matters

• The core trade-off is generic capability vs. task-specific performance: over-filtering may harm generality, while no filtering makes text training nearly impossible
• @codestar's "broken eval set" diagnosis reframes null filtering results as evaluation-quality issues, suggesting the evaluation pipeline itself requires data cleaning—a new angle for data-strategy research

2026-08-22 ~ 2026-08-24 · 5 related posts

• [source] Overfiltering Wastes Compute; Data Selection Beats Simple Heuristics — giffmana · 2026-08-22
• Data selection vs. filtering: why generic filters fail at scale — giffmana · 2026-08-23
• [source] Filtering vs Curation: Optimizing Training Data Distribution Over Quality Filters — giffmana · 2026-08-24
• [source] Data Cleaning Debate: The Necessity of Filtering in Text Training — code_star · 2026-08-24
• Ineffective Filtering? Your Evaluation Set Might Be Broken — code_star · 2026-08-24