MixLID: A Token-Level Language ID Model Covering 111 Language-Script Combinations
Seongtae Hong · hf · 2026-10-14
Korea University researchers released MixLID, a token-level language identification model supporting 111 language-script combinations that can tag the language of each token in code-switched text.
Key points:
- Two-stage training: first learns individual languages from monolingual labels, then continues on mixed-language data including concatenated code-switching examples and augmentation built from English-loanword correspondences.
- Beats all baselines in accuracy and macro F1 on four monolingual benchmarks, and outperforms baselines on every evaluated language pair for token-level mixed-language ID.
- Publicly available on Hugging Face (nlpai-lab/MixLID).
More from Research
- FrontierML benchmark tests whether agents can spend money to scale model performance — JayAlammar · 2026-10-14
- AIPOCH's Open-Science ships portable .science packages that share full research context, not just outputs — ahuja_priyank · 2026-10-14
- Video-Arena: Game-Based Video Benchmark Shows Native Models Score Just 13.2 F1 — Kuan Zhang · 2026-10-14
- LSCBench: Vibe Coding Agents Write Functional Code, But Only 25% Is Also Secure — Qwen · 2026-10-14
- Omni-LLMs Know More Than They Say: NAVER's OmniCatalyst Adds 7.29% via Self-Generated Text — naver-ai · 2026-10-14
- Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs — Sewon Min · 2026-10-14