Independent Jev test: 50/50 on hard invoice sorting at $0.025, but confidence misses rule errors

PawelHuryn · x · 2026-09-19

With projects built on Jev jumping from 46 to 160 in three days, a developer ran an independent test of the vendor's showcase invoice-sorting task using a deliberately hard dataset: 50 documents, 6 types, 10 languages, OCR errors, and misleading cues like a 'PROFORMA INVOICE' for shipped goods.

Results (correct/50, cost per 1k decisions): Jev 50/$0.025; GPT-OSS 20B 48/$0.030; Ministral 8B 48/$0.031; Claude Haiku 4.5 50/$0.39; Gemini Flash 49/$1.03; Claude Opus 5 49/$2.83.
The headline '444× cheaper' only holds against premium models; vs. locally runnable Ministral 8B, Jev is 20% cheaper but 90ms slower per call—though batching answers 32 questions in the time of one. Baseline and request pattern matter more than the multiple.
Confidence filtering works on this data: with only category names, Jev got 46/49; all mistakes scored below 0.80 confidence (correct answers averaged 0.97), so a 0.80 cutoff flagged every error.
But confidence has a blind spot: on 24 support messages with unstated house rules, Jev scored 5/24 without the rules vs 24/24 with them—and 15 of 19 wrong answers came back above 0.90 confidence. Certain, and wrong. Rules must be supplied per request; labeled examples worked worse (Jev 13/24, Ministral 8B 18/24).
Practical setup: log every decision with confidence, route uncertain calls to humans, and build a correction feedback loop.

Original post →

Independent Jev test: 50/50 on hard invoice sorting at $0.025, but confidence misses rule errors

More from Models