Hand-built anti-memorization benchmark: tiny Jev classifier matches Sonnet at ~150x lower cost
vesko_st · x · 2026-09-22
Developer veskost benchmarked TypeSafe AI's tiny Jev classifier and found it roughly on par with Claude Sonnet on classification tasks at orders of magnitude lower cost.
- On three standard reading-comprehension/commonsense benchmarks, Jev performed alongside Opus, topping CommonsenseQA and MMLU-CF and tying Opus on RACE-H, while being 150x cheaper and 10x faster on long-context questions.
- Suspecting training-data contamination, he hand-authored a fresh benchmark from a random Wikipedia article with distractors drawn from the text — Jev still landed alongside Sonnet and Opus.
- On older customer-service and negotiation datasets, Jev edged out Claude Haiku and sat slightly below Sonnet 5, though he cautions these aren't real B2B sales tasks.
He concludes small cheap classifiers like Jev have real uses and plans to adopt one in his product lightfld.
Related event: Tiny classifier Jev matches Claude Sonnet at ~150x lower cost(2 posts)→
More from Models
- ChatGPT now links to brand homepages far more, reshaping brand-mention tracking — lilyraynyc · 2026-11-12
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12
- LMArena Launches Agent Arena for Head-to-Head Agent Evaluation — arena · 2026-11-12
- LLMs lack hyperlocal municipal data, explains disputed Texas power outage debate — csuwildcat · 2026-11-12