Turkish intent classification test: both model versions hit 96.7% accuracy and miss the same lead
Sosyaldijital · reddit · 2026-10-18
A small experiment comparing JEV 1.13 and 1.14 on Turkish-language intent classification (buyer vs. seller vs. neither) found both scored 29/30 (96.7%), with 9/10 buyer detection, response times of 367ms vs. 352ms, and $0.00113 total API cost across 60 calls. Notably, both versions made the exact same mistake: a message explicitly offering to pay for photo editing was classified as a seller, not a buyer. The takeaway: overall accuracy hides critical errors, and in real-world lead detection false negatives can cost far more than false positives. The author admits the 30 synthetic scenarios and single run per version are too small to establish differences and nothing was deployed. Open question: should you optimize overall accuracy, recall on critical cases, or a cost-sensitive metric?
More from Models
- chatllm.cpp adds support for Intern-Decision's parallel diffusion-style decisions — foldl-li · 2026-10-18
- Meta opens Muse Spark Open: an 816B-parameter model for code and agents, running on Mac Studio — emmanuelvivier · 2026-10-18
- Microsoft finishes pre-training 2.8T-param MAI-Base-2 on 16,000 GB200 GPUs in 38 days — emmanuelvivier · 2026-10-18
- OpenAI acknowledges four new misalignment incidents, including confidential data extraction and test crypto wallet payments — emmanuelvivier · 2026-10-18
- DeepSeek V4.1 Flash serves nearly half of Vercel tokens at just 2.6% of spend — thealexbanks · 2026-10-18
- Turning off thinking over API: GLM-5.3 and Kimi K3 can't, and one Qwen endpoint silently ignores the switch — my-coffee-where · 2026-10-18