Dev benchmark: Jev hits 50/50 invoice sorting at $0.025/1k, but confidently fails without stated rules

PawelHuryn · x · 2026-09-19

A developer independently benchmarked the Jev classifier to test the vendor's "444× cheaper" pitch with real data.

Invoice sorting test

Built a hard dataset of 50 documents across 6 types, 10 languages, OCR noise, and misleading clues (fake proforma invoices, mislabeled summaries). Cost per 1,000 decisions:
Jev: 50/50 correct, $0.025
GPT-OSS 20B: 48/50, $0.030
Ministral 8B: 48/50, $0.031
Claude Haiku 4.5: 50/50, $0.39
Gemini 3.8 Flash: 49/50, $1.03
Claude Opus 5: 49/50, $2.83
Vs. locally-runnable Ministral 8B, Jev was only 20% cheaper and 90ms slower per call, but batching let it answer 32 questions in roughly the time of one.

Confidence limits

With only category names (no definitions): 46/49 correct, all errors below 0.80 confidence — a 0.80 cutoff would have flagged every mistake.
On 24 support messages with unstated house rules (e.g. duplicate charges go to support, not billing): 24/24 with rules in the prompt, only 5/24 without. Of the 19 errors, 15 scored above 0.90 confidence — certain and wrong.
Labeled examples instead of rules: Jev 13/24; Ministral 8B 18/24.

Takeaways: business rules must be supplied with every request (corrections don't carry over); confidence thresholds catch classification errors but not missing-rule errors. In production you want decision/confidence logging, a human-review threshold, and a feedback loop. When rules live in plain English and you need something running today, Jev matched the best accuracy at the lowest cost.

Original post →

Dev benchmark: Jev hits 50/50 invoice sorting at $0.025/1k, but confidently fails without stated rules

More from coding & agent