Independent Jev test: 50/50 on hard invoice sorting at $0.025, but confidence misses rule errors
PawelHuryn · x · 2026-09-19
With projects built on Jev jumping from 46 to 160 in three days, a developer ran an independent test of the vendor's showcase invoice-sorting task using a deliberately hard dataset: 50 documents, 6 types, 10 languages, OCR errors, and misleading cues like a 'PROFORMA INVOICE' for shipped goods.
- Results (correct/50, cost per 1k decisions): Jev 50/$0.025; GPT-OSS 20B 48/$0.030; Ministral 8B 48/$0.031; Claude Haiku 4.5 50/$0.39; Gemini Flash 49/$1.03; Claude Opus 5 49/$2.83.
- The headline '444× cheaper' only holds against premium models; vs. locally runnable Ministral 8B, Jev is 20% cheaper but 90ms slower per call—though batching answers 32 questions in the time of one. Baseline and request pattern matter more than the multiple.
- Confidence filtering works on this data: with only category names, Jev got 46/49; all mistakes scored below 0.80 confidence (correct answers averaged 0.97), so a 0.80 cutoff flagged every error.
- But confidence has a blind spot: on 24 support messages with unstated house rules, Jev scored 5/24 without the rules vs 24/24 with them—and 15 of 19 wrong answers came back above 0.90 confidence. Certain, and wrong. Rules must be supplied per request; labeled examples worked worse (Jev 13/24, Ministral 8B 18/24).
- Practical setup: log every decision with confidence, route uncertain calls to humans, and build a correction feedback loop.
More from Models
- Codex usage limits finally reset after days of users hitting exhausted quotas — CtrlAltDwayne · 2026-09-19
- Rumored Opus 5.2 generates a full 5-minute Titanic film in pure JavaScript with three.js — dotey · 2026-09-19
- Model Remembers 'cranberry-42' Across App Restarts in Persistent-Session Test — RileyRalmuto · 2026-09-19
- Free models aren't the real problem: studies show hallucinations persist in SOTA LLMs — AryHHAry · 2026-09-19
- Why do AI models lack personality? Safety training and attachment fears, explained — alexisgallagher · 2026-09-19
- New model JEV is free on OpenRouter, Vercel, Cloudflare and others, at ~$0.042/M input tokens — airesearch12 · 2026-09-19