Indie benchmark: Jev sorts 50 tricky invoices at 100% for $0.025 per 1,000 decisions

PawelHuryn · x · 2026-09-19

A developer independently benchmarked Jev, a text classification model whose project ecosystem grew from 46 to 160 in three days.

Test setup: 50 documents across 6 invoice types, 10 languages, with 32 containing deliberately misleading clues (a "PROFORMA INVOICE" for shipped goods, an "INVOICE SUMMARY" that isn't a payment request), plus OCR errors, spelled-out amounts, and keyword-free email-style invoices.

Results (correct / cost per 1,000 decisions):

Jev: 50/50, $0.025
GPT-OSS 20B: 48/50, $0.030
Ministral 8B: 48/50, $0.031
Claude Haiku 4.5: 50/50, $0.39
Gemini Flash: 49/50, $1.03
Claude Opus: 49/50, $2.83

Against locally-runnable Ministral 8B, Jev was 20% cheaper and 90ms slower per decision, but batched 32 questions in roughly the time of one. The vendor's "444× cheaper" claim compares against much pricier models.

Key findings:

Without category definitions Jev still scored 46/49; all errors fell below 0.80 confidence (correct answers averaged 0.97) — a confidence threshold reliably caught invoice errors
But with contradictory business rules (e.g. duplicate charges → support, not billing): rules in prompt scored 24/24, unstated scored 5/24, and 15 of 19 errors came back above 0.90 confidence — certain, and wrong. Confidence did not flag missing-business-rule errors
Labeled examples instead of rules performed worse: Jev 13/24, Ministral 8B 18/24

Takeaways for production: log every input/decision/confidence, set a human-review threshold, and build a feedback loop (e.g. a dedicated classifier with retraining). Business rules must accompany every request — corrections don't carry over between calls.

Original post →

Indie benchmark: Jev sorts 50 tricky invoices at 100% for $0.025 per 1,000 decisions

More from coding & agent