6-Year-Open-Source Multilingual Dataset Ships, Author Says Only Today's Models Make It Feasible
anas_ant · x · 2026-11-06
The author announced a 6-year-in-the-making project releasing high-quality multilingual data and models, with more languages promised soon.
The biggest remaining challenge is orthography mismatches between archival materials and modern community preferences as reflected in test sets. They handled it with human-guided processing this time, and believe agents could automate it.
More from Research
- Stanford wins $10M Laude lab to measure AI's impact on work and human-AI collaboration — Diyi_Yang · 2026-11-06
- Open-source Marin project finishes pretraining Mochi 535B on 18T tokens — teortaxesTex · 2026-11-06
- Stanford paper proposes differentially private watermarks for LLMs — ChengleiSi · 2026-11-06
- Stanford PhD JulieKallini on testing Chomsky: can LMs learn a language no human could? — ChrisGPotts · 2026-11-06
- Exo paper: agents that rewrite their own harness at runtime, cutting model cost up to 82% — sarahcat21 · 2026-11-06
- Study: LLMs Notice Context Conflicts 73% of the Time but Tell Users 0% — rseroter · 2026-11-06