Dev benchmark: Jev hits 50/50 invoice sorting at $0.025/1k, but confidently fails without stated rules
PawelHuryn · x · 2026-09-19
A developer independently benchmarked the Jev classifier to test the vendor's "444× cheaper" pitch with real data.
Invoice sorting test
- Built a hard dataset of 50 documents across 6 types, 10 languages, OCR noise, and misleading clues (fake proforma invoices, mislabeled summaries). Cost per 1,000 decisions:
- Jev: 50/50 correct, $0.025
- GPT-OSS 20B: 48/50, $0.030
- Ministral 8B: 48/50, $0.031
- Claude Haiku 4.5: 50/50, $0.39
- Gemini 3.8 Flash: 49/50, $1.03
- Claude Opus 5: 49/50, $2.83
- Vs. locally-runnable Ministral 8B, Jev was only 20% cheaper and 90ms slower per call, but batching let it answer 32 questions in roughly the time of one.
Confidence limits
- With only category names (no definitions): 46/49 correct, all errors below 0.80 confidence — a 0.80 cutoff would have flagged every mistake.
- On 24 support messages with unstated house rules (e.g. duplicate charges go to support, not billing): 24/24 with rules in the prompt, only 5/24 without. Of the 19 errors, 15 scored above 0.90 confidence — certain and wrong.
- Labeled examples instead of rules: Jev 13/24; Ministral 8B 18/24.
Takeaways: business rules must be supplied with every request (corrections don't carry over); confidence thresholds catch classification errors but not missing-rule errors. In production you want decision/confidence logging, a human-review threshold, and a feedback loop. When rules live in plain English and you need something running today, Jev matched the best accuracy at the lowest cost.
Related event: Independent Tests Confirm Jev Classifier's Accuracy at Fractional Cost(4 posts)→
More from coding & agent
- Jev's structured-output model impresses, Yegge argues fences beat sandboxes for agents — njyx · 2026-09-19
- UnCorreoTemporal MCP server lets AI agents grab OTP codes and verification links with zero human help — modelcontextprotocol · 2026-09-19
- Building 3D-printable models with GPT-6 Astra + Blender MCP, and offloading geometry to Higgsfield to save tokens — Julianw20012 · 2026-09-19
- Turned a $2 Bluetooth selfie remote into a physical push-to-talk clicker for agentic coding — _7xen_on · 2026-09-19
- What won't you let an AI agent do? For many, it's speaking in your name — Luvena21 · 2026-09-19
- willcb: in-context learning for personal continual learning needs neurosymbolic harnesses — willcb · 2026-09-19