Sakana AI's JFinBench: frontier models top 93%, open weights trail by 12 points

SakanaAI · hf · 2026-11-16

Sakana AI and Science Tokyo release JFinBench, a refreshable Japanese financial benchmark of 321 questions from 2026 certification exams (accounting, tax, financial planning, banking), built via a human-in-the-loop agentic pipeline where a coding agent writes conversion code and humans verify every example. Evaluating 12 LLMs: Claude Fable 5 and GPT-6 Astra exceed 93%, while the best open-weight model Kimi K3.1 Pro trails by 12 percentage points. With terminal and web access all models improve, but a 4-point gap persists and rankings hold, suggesting in-model knowledge remains valuable. Dataset, eval code, and pipeline are open-sourced.

Original post →

More from Research

Research channel →