Researchers debate: small decider heads vs task-specific models for pretraining data filtering
tulkenss · x · 2026-10-09
In a technical exchange, researcher tulkenss questions whether a smaller 'decider' would beat a good task-specific model for pretraining data filtering, calling the fine-web-edu model's design (frozen body with a regression head for classification) strange. FineWeb author antoinechaffin counters by suggesting a smaller decider; both admit they haven't run the experiment.