Sakana AI's JFinBench: frontier models top 93%, open weights trail by 12 points
SakanaAI · hf · 2026-11-16
Sakana AI and Science Tokyo release JFinBench, a refreshable Japanese financial benchmark of 321 questions from 2026 certification exams (accounting, tax, financial planning, banking), built via a human-in-the-loop agentic pipeline where a coding agent writes conversion code and humans verify every example. Evaluating 12 LLMs: Claude Fable 5 and GPT-6 Astra exceed 93%, while the best open-weight model Kimi K3.1 Pro trails by 12 percentage points. With terminal and web access all models improve, but a 4-point gap persists and rankings hold, suggesting in-model knowledge remains valuable. Dataset, eval code, and pipeline are open-sourced.
More from Research
- Researchers formally specify ops interfaces for CockroachDB, Kafka and RabbitMQ to make AI operators verifiable — tianyin_xu · 2026-11-16
- Astera Residencies open: up to $2M in funding for off-network researchers — dr_alphalyrae · 2026-11-16
- 'It Knows Google Is Watching': Viral Chat Surfaces J-Lens Evaluation-Awareness Research — repligate · 2026-11-16
- NeurIPS Paper Explains Why Knowledge Editing in English Fails to Propagate Across Languages in Multilingual LLMs — Tanmoy_Chak · 2026-11-16
- MDPI review surveys reinforcement learning approaches in quantitative trading — PtrPomorski · 2026-11-16
- JEM trained in 20 H100-hours lands 1 point behind fine-tuned DINOv3 — gabriberton · 2026-11-16