Cross-Lingual Company Entity Disambiguation in Practice
hannune · reddit · 2026-07-18
While building a corporate knowledge graph across Korean, Japanese, Chinese, and English, the author noticed that the same company was often split into multiple nodes (e.g., Samsung Electronics appearing in four different language scripts).
They summarized several effective practices:
- Transliterate then compare: Converting Chinese characters or Kana into a unified format eliminates many duplicate nodes before similarity calculation.
- Use language pairs as blocking keys: This avoids full pairwise comparisons, narrowing the candidate set to a manageable size.
- Set thresholds per language pair: Abbreviations and variations differ across pairs (like Korean-English vs. Japanese-Chinese), making a single global threshold ineffective.
The main challenge lies in labeled data: Splink requires "same entity/different entity" samples across language combinations, but rare pairs like Korean-Chinese lack public datasets. The author resolved this by extracting clearly distinct companies as negative samples and using Korean financial reports to construct positive samples.
After processing, the number of nodes in the graph decreased—which is a positive outcome, as retrieval accuracy improved and queries are no longer scattered across false duplicate nodes.
More from Research
- Anthropic masterclass spotlights how to build and observe AI agents — _jaydeepkarale · 2026-07-21
- NeurIPS 2026 workshop calls papers on on-device intelligence — YiMaTweets · 2026-07-21
- AI Security Institute says every tested model tried to cheat in cyber evaluations — connoraxiotes · 2026-07-21
- AI companies are buying old books to avoid training on AI-generated slop — CackleRooster · 2026-07-21
- Sakana says multiple diffusion models plus MCTS beat test-time scaling on coding and math — SakanaAILabs · 2026-07-21
- Soofi S 30B-A3B releases a full pretraining report and claims open-model leads in English and German — abursuc · 2026-07-21