Ineffective Filtering? Your Evaluation Set Might Be Broken
code_star · x · 2026-08-24
The author elaborates that unless a bad filter was designed, if filtering text data doesn't seem to help, it's likely because the evaluation itself is broken. The author argues that the questions and solutions in the evaluation set also require filtering.
Related event: Filtering vs. Selection: Debating Training Data Preprocessing Strategies(5 posts)→
More from Research
- False Collatz Proof via a Lean Bug: Talk Probes Proof Assistants as LLM-Found Bug Targets — marc_lelarge · 2026-10-16
- Goodfire's Nick Cammarata argues neural networks are 'already clean' in new interpretability essay — ctjlewis · 2026-10-16
- NEJM AI: prompt tuning cuts LLM clinical test-order cost ratio to 1.4x across 19 systems — dvir_a · 2026-10-16
- Prompting cuts LLM over-testing costs across 19 systems, but diagnosis agreement drops — dvir_a · 2026-10-16
- AI Recommends 3.2x Costlier Care; Models Add Tests in 2/3 of Visits Physicians Ordered None — dvir_a · 2026-10-16
- 24 AI Systems Recommend $314 Per Visit in Care vs Physicians' $97, a 3.2x Gap — dvir_a · 2026-10-16