Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition: Evidence, Critical Errors, and Selective Automation
Subodh Lonkar
cs.CV, cs.AI
2026-11-02
Risk-constrained framework for financial OCR: gate critical-field error rates by one-sided confidence bounds; zero observed errors still imply non-zero risk. Theory only.
Financial document pipelines grade OCR with character error rate (CER) and word error rate (WER). Edit distance prices every substitution the same, so misreading one letter in a narrative paragraph and turning 1,250.00 into 12,500.00, dropping a minus sign, or shifting an amount from the debit to the credit column all sit within a fraction of a point of CER. Average over a big corpus and the dangerous errors vanish into the mass of correct low-stakes fields.
The second gap sits at acceptance time. An enterprise is not deciding which model scores highest; it is deciding whether output can flow into the books without a human in the loop. A higher-scoring system with an above-limit polarity error rate and a lower-scoring system with every critical gate passed are indistinguishable under current reporting. This paper formalizes that decision layer: treat recognition as evidence recovery, use scores for ranking, and let hard gates decide release.
The framework rests on a few separations, each matched to a failure mode that current metrics collapse together:
Around this sit a ten-dimension quality vector (detection, text, layout, tables, fields, facts, evidence grounding, calibration, robustness, operations), leakage-resistant splits grouped by customer, template, and time, and a statistical protocol with stratified sampling and document-level paired comparisons. The contract covers both OCR pipelines and OCR-free document models.
No model was run and no benchmark scores exist. The results are four formal propositions plus one worked example:
| Proposition | Result |
| Compensatory masking | For any weighted average with positive weights, one can construct systems A and B where A scores higher overall while scoring 0 on a critical dimension; an aggregate cannot certify safety |
| Zero errors do not imply zero risk | With zero errors in n independent critical instances, the one-sided (1−α) upper bound is 1−α^(1/n); proving p ≤ η requires n ≥ log α / log(1−η) |
| Loss-sensitive review | Routing to human review beats auto-acceptance iff q < 1 − rH − c/L (q probability correct, L loss per error, c review cost, rH residual error after review) |
| Coverage monotonicity | Raising the confidence threshold can only shrink auto-accept coverage |
Worked number: a 95% one-sided upper bound below 10⁻³ requires roughly 2,995 independent critical instances with zero errors under independence assumptions; the folk version is the rule of three, about 3/n. The routing demonstration, where the higher-scoring system is blocked by a polarity gate and the lower-scoring one passes, is constructed for illustration, not measured. The paper offers no empirical comparison against FinCriticalED or any other benchmark; the author states plainly that the contribution is theoretical and methodological.
For teams building intelligent document processing, buying financial OCR, or writing acceptance tests for document models, this is close to a ready-made acceptance spec: define critical fields and classes, gate on Clopper–Pearson upper bounds, set the review threshold from coverage-capacity constraints. All three pieces transfer directly into procurement contracts or model cards.
The zero-error sample-size point is usable on its own. Claiming a critical-field error rate under 0.1% needs about 3,000 representative critical samples with zero errors, and real data is clustered (pages from one statement, invoices from one supplier template), which pushes the requirement higher. Most vendor claims of "99.9% accuracy" do not survive that lens.
To be clear, this is not a model improvement and produces no new state of the art. It changes how "qualified" is computed, which serves evaluation and risk roles rather than leaderboard chasing.
The author lists eight, the load-bearing ones: ground truth itself can be wrong (ambiguous handwriting, poor scans, accounting-convention differences), requiring explicit ambiguity states and adjudication; weights and classes are institution-specific policy with no universal scheme; arithmetic validators misfire under special accounting treatments; calibration degrades under drift; and an OCR metric layer cannot substitute for downstream compliance and authorization controls. Heaviest of all: the framework is not empirically validated, with no dataset, no annotator-tested manual, no software, and no measured reviewer performance; the routing example is synthetic.
Two more concerns on a close read. The annotation contract (evidence polygons, three-level representation, double annotation with adjudication for critical fields) is expensive enough that most teams may not afford it, and the paper itself lists annotation cost and inter-annotator agreement as open questions. The four propositions are also mathematically elementary (Proposition 2 is a direct restatement of the Clopper–Pearson interval); the value is in wiring the statistics into acceptance procedure, not in theorem depth. Single author, no funding, no experiments: adoption waits on follow-up work.