20-40% of alignment data is noisy: a loss-reduction scoring framework to fix it

burkov · x · 2026-11-04

Studies estimate 20-40% of alignment preference data is noisy, redundant, or inconsistent, diluting alignment signals.
Identifying high-value samples normally needs costly retraining loops, auxiliary reference models, or prohibitive approximations.
The article introduces a resource-efficient framework that dynamically scores and reweights fine-tuning examples by quantifying each one's loss reduction on a small high-quality holdout set — no reference models or repeated training required.
Takeaway: small curated subsets often match massive datasets, and this framework makes that curation practical in production.

More from Research