Hands-on: TypeSafe's tiny Jev classifier goes head-to-head with Claude Haiku/Sonnet/Opus on text understanding
vesko_st · x · 2026-09-22
Ves Stoyanov (Lightfield) tested TypeSafe's newly released Jev, a compact "System One Choice" classifier that takes a passage, a question, and options and returns one answer. Wary that public benchmarks may be contaminated and stale, he re-ran everything under a fixed protocol and hand-wrote a corpus as a contamination hedge. Jev was benchmarked against Claude Haiku 4.5, Sonnet 5, and Opus 5 on CommonsenseQA, MMLU-CF, RACE-H, Grace and more, priced per 1,000 long-context questions. His takeaway: the purpose-built small classifier now holds its own on text understanding, making dedicated classifier models compelling again for routing-style tasks.
Related event: Tiny classifier Jev matches Claude Sonnet at ~150x lower cost(2 posts)→
More from Models
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Grok spotted building meeting "Listen to Conversation" feature with live transcript — nima_owji · 2026-11-12