Fine-tuning survives up to 30% label noise across all 3 benchmarks tested
Reasonable_Royal_621 · reddit · 2026-10-30
The CacheVerifier team stress-tested how label noise affects fine-tuning: production feedback labels (thumbs up/down) are noisy, so they flipped 0–40% of 4,000 training pairs and evaluated on clean labels across 3 public benchmarks.
Key results
- LMArena AUC gain: +0.107 clean, +0.083 at 10% flipped, still +0.028 at 30%.
- Break-even landed at 31–32% noise on all three datasets — better than expected.
- At 40% it falls off a cliff: -0.101 on LMArena.
Takeaway: if a small audit shows your feedback noise is well under 30%, fine-tuning is probably worth it. Caveat: these were random flips; real-world noise has more structure. Details in section 5.7 of the CacheVerifier repo.
More from Research
- Perplexity's pplx-embed-v2-context ditches gold-chunk labels, sets SOTA on retrieval benchmarks — _reachsumit · 2026-10-30
- Gavin Crooks: Maybe it's time to write some 'last papers' as AI absorbs knowledge — MaxUnfried · 2026-10-30
- DoorDash's PALATE: GPT-style user embeddings lift ad revenue ~1% in production — _reachsumit · 2026-10-30
- A sequential decision framework unifies evaluation of LLM-guided search policies — _reachsumit · 2026-10-30
- Alibaba's PFA progressively focuses attention to speed long-sequence generative recommendation — _reachsumit · 2026-10-30
- RecRSI: ranking-scored short trials cut wasted retraining in LLM-driven recommender self-improvement — _reachsumit · 2026-10-30