Debate Over Data Filtering vs. Selection in Model Training

A heated "filtering vs. selection" debate has broken out in the community over preprocessing strategies for LLM training data. @giffmana argues the issue should be reframed as "data selection" rather than "data filtering": for general-purpose models, nearly all data becomes useful as compute scales up. Simple heuristic filters can remove junk, but they also discard high-value data (e.g., images named by date, seemingly meaningless filenames). Material they cite claims excessive filtering can waste roughly 33% of compute, so unless compute is constrained or the model isn't general-purpose, "smart filtering" should be avoided in favor of selectors targeting specific distributions, with attention to training data distribution and maximizing diversity.

Confirmed

• @giffmana explicitly advocates distinguishing the two paradigms of "filtering" and "selection," arguing general-purpose model training should have no filter, as filtering is a poor form of quality control
• Cited data claims excessive filtering causes roughly 33% compute loss
• @codestar objects to generalizing vision-domain conclusions to text: text data contains an extremely high proportion of junk (e.g., 90%), making it infeasible to train on unfiltered raw Common Crawl data
• @codestar believes "whack-a-mole" removal of obviously bad data is necessary

Unconfirmed

• The specific experimental conditions and source of the 33% compute loss figure are not given in the post
• The statistical basis for the 90% junk proportion is unclear

Why it matters

• The core of the debate is the trade-off between general capability and task-specific performance: over-filtering may harm model generality, while not filtering at all makes training nearly impossible in the text domain
• @codestar offers an alternative explanation for filtering's ineffectiveness: the evaluation sets themselves may be seriously flawed, and eval data also needs cleaning — providing a new diagnostic angle for experimental results showing "filtering doesn't help"

2026-08-22 ~ 2026-08-24 · 5 related posts

• [source] Overfiltering Wastes Compute; Data Selection Beats Simple Heuristics — giffmana · 2026-08-22
• Data selection vs. filtering: why generic filters fail at scale — giffmana · 2026-08-23
• Filtering vs Curation: Optimizing Training Data Distribution Over Quality Filters — giffmana · 2026-08-24
• [source] Data Cleaning Debate: The Necessity of Filtering in Text Training — code_star · 2026-08-24
• Ineffective Filtering? Your Evaluation Set Might Be Broken — code_star · 2026-08-24