Python gotcha breaks LLM evals: "İSTANBUL".lower() doesn't return "istanbul"
irombie · x · 2026-10-30
A follow-up to the author's eval-reliability write-up: in Python, "İSTANBUL".lower() does not return "istanbul", so exact-match scorers can mark correct Turkish answers as wrong — a concrete Unicode case-folding gotcha for non-English LLM evaluations.
More from coding & agent
- AI revives formal verification: seL4 took 20 person-years, startups now claim it's practical — lemire · 2026-10-30
- YC experiment: 4 AI agents spend 11 minutes appointing a "sol" in Minecraft bureaucracy — aiamblichus · 2026-10-30
- Microsoft releases Mage-VL-Cowork, a vision-language model for computer-use and coding agents — microsoft · 2026-10-30
- WhatsApp outreach agent: one promo per person per 14 days, counted across all DM lanes — petrusenko_max · 2026-10-30
- Nova v1 'Flash' launches as an AI engineering teammate aiming to replace the IDE — Gauri_the_great · 2026-10-30
- Malicious Claude Code extension with SSH backdoor hit Cursor's marketplace, up to 5,000 users — cyb3rops · 2026-10-30