First independent Jev benchmark: 644 live API calls cost just $0.23
PawelHuryn · x · 2026-09-19
TypeSafe's newly launched Jev is a typed "decision model": send text, a question, and a list of options, get back one option plus a confidence score — no prose, free output tokens. Its launch drew a wave of projects and widely-shared "hundreds of times cheaper" claims within 72 hours, but no one had published a real measurement.
Developer PawelHuryn ran two experiments totaling 644 live API calls and $0.23 in spend, starting from the vendor's showcase invoice-sorting task:
- Round 1: his first dataset was too easy — all six models scored 100%, so it couldn't discriminate;
- Round 2: rebuilt as 50 documents across 6 types to test whether Jev actually wins on its own advertised task, and whether domain knowledge can reach it without any fine-tuning channel.
Every document, routing message, criteria text, script, and per-decision record is open-sourced on GitHub (phuryn/experiments, jev-decisions-api), fully reproducible.
Related event: Independent Tests Confirm Jev Classifier's Accuracy at Fractional Cost(4 posts)→
More from Models
- Gemini 4 'Argon' swag box from Google DeepMind surfaces, teasing next-gen model — kimmonismus · 2026-11-02
- Nanbeige4.3-33B lands on Hugging Face, pitched as a Qwen rival — amaarora · 2026-11-02
- r/LocalLLaMA Proposal: Private Quality Benchmarks Over Endless tok/s Chasing — Express_Quail_1493 · 2026-11-02
- A4 print-ready PDF stress test: only Claude, Gemini and Mistral pass out of 10 AI models — gerardsans · 2026-11-02
- Discounted API relay stations lack prompt caching, making long sessions pricier — uneatenbreakfast · 2026-11-02
- Google open-sources EmbeddingGemma 2: on-device multimodal search across photos, audio, docs — bigaiguy · 2026-11-02