Indie benchmark: Jev sorts 50 tricky invoices at 100% for $0.025 per 1,000 decisions
PawelHuryn · x · 2026-09-19
A developer independently benchmarked Jev, a text classification model whose project ecosystem grew from 46 to 160 in three days.
Test setup: 50 documents across 6 invoice types, 10 languages, with 32 containing deliberately misleading clues (a "PROFORMA INVOICE" for shipped goods, an "INVOICE SUMMARY" that isn't a payment request), plus OCR errors, spelled-out amounts, and keyword-free email-style invoices.
Results (correct / cost per 1,000 decisions):
- Jev: 50/50, $0.025
- GPT-OSS 20B: 48/50, $0.030
- Ministral 8B: 48/50, $0.031
- Claude Haiku 4.5: 50/50, $0.39
- Gemini Flash: 49/50, $1.03
- Claude Opus: 49/50, $2.83
Against locally-runnable Ministral 8B, Jev was 20% cheaper and 90ms slower per decision, but batched 32 questions in roughly the time of one. The vendor's "444× cheaper" claim compares against much pricier models.
Key findings:
- Without category definitions Jev still scored 46/49; all errors fell below 0.80 confidence (correct answers averaged 0.97) — a confidence threshold reliably caught invoice errors
- But with contradictory business rules (e.g. duplicate charges → support, not billing): rules in prompt scored 24/24, unstated scored 5/24, and 15 of 19 errors came back above 0.90 confidence — certain, and wrong. Confidence did not flag missing-business-rule errors
- Labeled examples instead of rules performed worse: Jev 13/24, Ministral 8B 18/24
Takeaways for production: log every input/decision/confidence, set a human-review threshold, and build a feedback loop (e.g. a dedicated classifier with retraining). Business rules must accompany every request — corrections don't carry over between calls.
More from coding & agent
- Utopia, an open-source bitemporal 'world model' for knowledge graphs, hits 8.7k GitHub stars — tom_doerr · 2026-09-19
- Hermes Agent lands sender-based profile routing so one shared chat can serve isolated profiles — Teknium · 2026-09-19
- Cognition's SWE-2 one-shots a detailed Game Boy Advance emulator from minimal prompts — CtrlAltDwayne · 2026-09-19
- Ex-Coursera Engineer Jiquan Ngiam Brings Infra Expertise to Secure MCP Rollouts — lee_stott · 2026-09-19
- Dev builds 135 editable CSS 3D experiments with Claude Code in about a day — Evirtuality · 2026-09-19
- Dev builds Concat, an open-source CapCut replacement, entirely with AI agents — JUB0T · 2026-09-19