Orthography mismatches remain the hard part of mining archival data for low-resource MT
anas_ant · x · 2026-11-06
The paper author discusses the biggest remaining challenge in mining parallel data for extremely low-resource languages: orthography mismatches between archival linguistic materials and modern community preferences reflected in test sets — missing diacritics, noisy OCR, script conversions. The team handled normalization in a human-guided way (Table 10 catalogs strategies for 20 languages), and suggests agents could automate this step in future work.
More from Research
- Stanford paper proposes differentially private watermarks for LLMs — ChengleiSi · 2026-11-06
- Stanford PhD JulieKallini on testing Chomsky: can LMs learn a language no human could? — ChrisGPotts · 2026-11-06
- Exo paper: agents that rewrite their own harness at runtime, cutting model cost up to 82% — sarahcat21 · 2026-11-06
- Study: LLMs Notice Context Conflicts 73% of the Time but Tell Users 0% — rseroter · 2026-11-06
- Science's Last Exam: best of 8 frontier models scores 37.5/100, all score 0 on end-to-end research — anshulkundaje · 2026-11-06
- Tokenizer Myths and Where To Find Them: Stanford paper debunks three BPB myths — rohanpaul_ai · 2026-11-06