New framework evaluates financial document recognition by critical-error gates, not error rates
Subodh Lonkar · hf · 2026-11-09
This paper proposes a human-centred, risk-constrained framework for evaluating financial document recognition, arguing that character/word error rates fail to capture whether a system preserves the financial facts people act on.
Key points
- Defines typed financial facts linked to visible source regions, separating verbatim transcription, deterministic normalization, and business interpretation.
- A layered quality model covers text, localization, layout, tables, fields, fact correctness, evidence grounding, uncertainty, robustness, and operational performance.
- Non-compensatory acceptance gates use one-sided uncertainty bounds for critical-field errors (amounts, identifiers, signs, periods, entity links), preventing strong aggregate scores from masking failures.
- Selective automation is cast as a constrained routing problem balancing residual risk, human-review capacity, and cost, with a loss-sensitive rule for sending uncertain cases to human review.
- Also specifies annotation/adjudication procedures, leakage-resistant benchmark partitions, statistical estimands, subgroup analyses, and a reproducibility card.
Scope: theoretical and methodological only — no new model, dataset, or empirical performance claims.
More from Research
- Toby Ord clarifies: small delays still allow growth to climb far before biting — tobyordoxford · 2026-11-10
- Qdrant presents vector index-native relevance feedback in production at CIKM Rome — qdrant_engine · 2026-11-10
- Meta's LLM-generated failing tests catch 74 privacy SEVs, cutting incidents 20% — Meta_Engineers · 2026-11-10
- Toby Ord: time delays in the intelligence feedback loop rule out a true singularity — tobyordoxford · 2026-11-10
- Cornell professor recruiting Fall 2027 PhD students in probabilistic ML, AI4Science — dianarycai · 2026-11-10
- int8 quantization plus fp16 rescoring: 1.4M embeddings retrieved in under 2ms — tomaarsen · 2026-11-10