IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
cs.AI
2026-09-24
NPCI's 799-case Indian retail-banking benchmark: 11 models score 43.7–58.2% strict pass^3 versus 60–74% at-least-once; none exceed 40.1% on the wrong-information axis.
A retail-banking assistant has to look up a specific account, explain a transfer, or freeze a card. Scoring only the last sentence misses a class of errors: asking for a balance already in context, answering from a stale turn, picking the wrong account, or writing a wrong value after stating the right one. The US CFPB has documented wrong answers, missed disputes, and customers stuck in loops. Indian retail banking adds local payment rails (UPI, NEFT, IMPS, RTGS) and branch codes (IFSC). A customer may ask whether an INR 45,000 NEFT arrived when the matching payment was IMPS; a NEFT-only search misses it.
Most tool-use leaderboards collapse that path into one accuracy number. Gorilla and ToolBench score API choice. τ-bench adds multi-turn tools and repeated trials. FinQA and BhashaBench-Finance sit closer to knowledge tests. IndicBankBench, from India's National Payments Corporation (NPCI), targets authenticated retail-banking dialogues with account state.
799 synthetic cases cover five operational domains plus a capability/refusal domain, across 20 primary axes: 731 task cases on 12 axes, 68 capability cases on 8. 204 cases require a state-changing write, 95 carry prior conversation the model did not produce, 14 inject unseen tool schemas. Tools are deterministic mocks. The model receives a logincontext with accounts, cards, products, and the current date.
Grading uses four stages, S/A/R/Q. Safety checks (no fabricated IDs, confirm before write, schema-valid arguments, no raw nulls in the reply) are mostly deterministic; a narrow yes/no resolver fires only when confirmation is ambiguous. Action checks required tools, forbidden tools, arguments, and order. Response uses an LLM judge (GLM-5.2, temperature 0, seed 42) that labels the reply as answer, clarify, or decline against a one-sentence criterion. Quality scores grounding, completeness, and tone and does not change the verdict. The grading code, not either model, issues pass/fail.
Each case runs three times at temperature 0.7 (2,397 trajectories per model). The headline metric is pass^3: all three trials must pass. The paper also reports at-least-once success and mean single-run pass rate. Eleven instruction-tuned models: DeepSeek-V4-Pro and Flash, Grok 4.6, Qwen3.8-2.4T-A95B, Claude Opus 5, GLM-5.3-Flash, three Gemma 4 sizes, Gemini 3.8 Flash, MiniMax M3.
Strict pass^3 ranges from 43.7% (349/799, Gemma 4 E4B IT) to 58.2% (465/799, DeepSeek-V4-Pro-0813). No model clears 60%. The top five sit between 56.8% and 58.2%, an 11-case spread. Paired bootstrap intervals all include zero; Holm-adjusted McNemar tests are all p>0.05.
| Model | pass^3 | At least once | Inconsistent |
| DeepSeek-V4-Pro-0813 | 58.2% | 71.8% | 13.6% |
| Grok 4.6 | 58.1% | 70.2% | 12.1% |
| Qwen3.8-2.4T-A95B | 57.8% | 74.0% | 16.1% |
| Claude Opus 5 | 57.7% | 71.3% | 13.6% |
| GLM-5.3-Flash | 56.8% | 74.3% | 17.5% |
| Gemma 4 E4B IT | 43.7% | 60.1% | 16.4% |
At-least-once success exceeds strict pass^3 by 10.8 to 21.4 percentage points. GLM-5.3-Flash has the highest at-least-once rate at 74.3% and a strict rate of 56.8%. MiniMax M3 is the most inconsistent: 171 of 799 cases (21.4%) pass on some trials and fail on others.
Wrong-information is the lowest task axis for seven of eleven models; none pass more than 40.1% of its 142 cases (Claude Opus 5, 57/142). DeepSeek-V4-Pro is 46/142 there, while several capability axes are 6/6. Across all eleven models the first blocking gates are A1 (required tool never called) and R (reply misses the case criterion). Of 26,291 trajectories where both decisions could be scored, 12.9% passed every safety and action check and still failed R; 5.7% passed R and failed a safety or action gate.
The useful output is a diagnostic, not a ranking. τ-bench already repeats trials; this set adds account grounding, confirm-before-write, and a first-failed-stage label. Cases and harness are public. Prompt or tool-policy comparisons should freeze the case bank, judge, and decoding, then read axis and gate breakdowns.
A high score is not a license to deploy. The cases are public, so future systems can overfit them.
Response adequacy depends on an LLM judge. A blinded 40-transcript audit agreed with humans on 31 of 38 decidable cases (81.6%, κ=0.642). The sample covers the original eight models, not the three added later. GLM-5.3-Flash shares the judge's broader family. Three trials show inconsistency; they do not estimate long-run reliability well. Cases are synthetic English mocks with no Hindi, code-switching, live authorization, or fraud systems. Temperature 0.7 is not comparable across vendor endpoints. Most capability axes have six cases. The nine-pattern taxonomy comes from a separate temperature-0 screen, not from frequency counts on the eleven-model run.