Sakana AI releases JFinBench: 321 Japanese finance exam questions expose open-model gaps
SakanaAILabs · x · 2026-11-16
Sakana AI released JFinBench, a Japanese-language finance benchmark built from 321 questions across six 2026 Japanese financial qualification exams (CPA, tax accountant, FP Level 1, etc.), covering multiple choice, tax calculations, essays, and diagram reading.
The benchmark uses a "human-in-the-loop agentic pipeline": coding agents convert exam PDFs while humans verify all questions (12 hours of visual checking), enabling cheap updates with each new exam cycle.
Evaluating 12 recent LLMs, open-weight models that nearly match proprietary ones on mainstream benchmarks show large gaps on JFinBench. Failures stem mainly from lacking Japanese regulatory knowledge, not computation; giving models terminal + web access to look up laws improves scores but doesn't close the gap. Paper, dataset, and code are all open.
Related event: Sakana AI Releases JFinBench, a Japanese Financial Benchmark(2 posts)→
More from Models
- Dev says GPT-6 Astra is so bad he can't even burn through his Codex Pro 200 quota — bdsqlsz · 2026-11-16
- Qwen4-Max hits Vercel AI Gateway: 1M context, native vision, 25% cheaper — Alibaba_Qwen · 2026-11-16
- Pangram's AI-detection bounty goes unclaimed for a week — zero false positives found — ctjlewis · 2026-11-16
- Team surprised its writing model beat Pangram's AI detection without training for it — yawnxyz · 2026-11-16
- Proposal: an open-source model encyclopedia that runs on a single 4090 — TinfoilTricorn · 2026-11-16
- Model recites hidden conversation word-for-word despite claiming it can't see context — Sauers_ · 2026-11-16