Sakana AI releases JFinBench: 321 Japanese finance exam questions expose open-model gaps

SakanaAILabs · x · 2026-11-16

Sakana AI released JFinBench, a Japanese-language finance benchmark built from 321 questions across six 2026 Japanese financial qualification exams (CPA, tax accountant, FP Level 1, etc.), covering multiple choice, tax calculations, essays, and diagram reading.

The benchmark uses a "human-in-the-loop agentic pipeline": coding agents convert exam PDFs while humans verify all questions (12 hours of visual checking), enabling cheap updates with each new exam cycle.

Evaluating 12 recent LLMs, open-weight models that nearly match proprietary ones on mainstream benchmarks show large gaps on JFinBench. Failures stem mainly from lacking Japanese regulatory knowledge, not computation; giving models terminal + web access to look up laws improves scores but doesn't close the gap. Paper, dataset, and code are all open.

Related event: Sakana AI Releases JFinBench, a Japanese Financial Benchmark(2 posts)→

Original post →

More from Models

Models channel →