What broke when fine-tuning on scraped web data — and what fixed it
Symeoon · reddit · 2026-11-03
A developer details real failure modes when fine-tuning on self-collected web data:
- Boilerplate contamination was the silent killer: nav menus, cookie banners and footers leaking into training examples made the model emit half-rendered-HTML-style outputs. The fix: aggressive DOM-targeted extraction instead of raw text scraping, plus per-request rotating residential proxies so sites serve the full rendered DOM a real browser would get, dramatically cleaning up raw HTML before parsing.
- Dedup order mattered: running exact dedup before fuzzy dedup produced meaningfully different training sets than the reverse, and the model's tendency to regurgitate phrasing shifted accordingly.
- Unsolved: cross-language near-duplicate detection — MinHash works monolingually but keeps surfacing edge cases on multilingual corpora.
The author asks for principled approaches to multilingual dedup and to deciding when scraped data is clean enough to just train.
More from Research
- EMNLP 2026 interpretability tutorial slides released, praised by attendees — PMinervini · 2026-11-03
- NVIDIA's LucidLyra turns a single image into queryable 3D Gaussian scenes — CSProfKGD · 2026-11-03
- CVPR removes hidden trap for workshop organizers, exposes metadata deadline ahead of papers — ducha_aiki · 2026-11-03
- UCSB REAL AI initiative builds digital twins for science, recruiting PhDs — ninamiolane · 2026-11-03
- UCSB Geometric Intelligence Lab recruiting PhD students for Fall 2027 — ninamiolane · 2026-11-03
- Revisiting 50 LLM findings: 44% expired within 15 months, only 6% strengthened — james_y_zou · 2026-11-03