turbopuffer ships CJK tokenizers; dev shares hard-won Chinese/Japanese/Korean search tuning tips
bo_wangbo · x · 2026-11-11
turbopuffer released Chinese, Japanese, and Korean tokenizers (smartcn, kuromoji, nori) for its BM25 full-text search, splitting CJK text into dictionary words and supporting cross-word-form matching (e.g. 食べる → 食べました). A developer shares first-hand tuning experience: SmartCN and Kuromoji work well, but Nori for Korean is hard to tune and natively enabling it disrupted his dev eval shard. The feature supports hybrid search combining vector and BM25 results.
More from coding & agent
- Artificial Analysis benchmarks 26 Search APIs across 14 providers in new agent-search leaderboard — ArtificialAnlys · 2026-11-11
- Keenable Search 1.0 tops Artificial Analysis Search Index at 83, a third below median cost — ArtificialAnlys · 2026-11-11
- Dev calls on GitHub to ship push-based integration (webhooks, MCP) for agents — brianmichel · 2026-11-11
- Dev ships free remote MCP server to check if ChatGPT and Claude recommend your product — hookedupwithclaude · 2026-11-11
- DSPy 3.5 ships: 85% smaller installs, LiteLLM now optional — dbreunig · 2026-11-11
- Crewplane: open-source tool that runs coding-agent workflows defined in Markdown — CarsonBuilds · 2026-11-11