Ox Alpha benchmark test: 551 API calls needed to get 87 completed answers
anshulkundaje · x · 2026-08-23
@sbatzoglou evaluated Ox Alpha on the induction benchmark (ICML 2026 spotlight paper "INDUCTION: Finite-Structure Concept Synthesis in First-Order Logic") via OpenRouter:
- Mediocre results: ranks below Luna and DeepSeek v4 Pro, just above Gemini 3.7 Flash. The author notes other benchmarks on X show Ox Alpha performing strongly, so results may be highly benchmark-specific.
- Stability concerns: the model frequently returned empty strings or API errors; 551 API calls were needed to reach an 87/100 completion rate. Unclear whether this is an OpenRouter issue or the common reasoning-model behavior of not answering when unsolved, amplified.
Benchmark data and evaluation code are open-sourced in the concept-synth GitHub repo (arXiv:2602.18843).
More from Models
- Muse Spark vs Qwen and GLM: US open models still trail on agentic tasks — zsakib_ · 2026-10-16
- Muse Spark 816B gets llama.cpp support just 30 minutes after release — ngxson · 2026-10-16
- 24 AI Systems Recommend $314 Per Visit in Care vs Physicians' $97, a 3.2x Gap — dvir_a · 2026-10-16
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Scale AI launches muse spark open: 816B-param MoE with 1M context, billed as top US open-weight model — jiayi_pirate · 2026-10-16
- NVIDIA HF listing for Qwen3.8-Flash-Next-FP4-Local leaks unannounced Qwen3.8 model — TheZachMueller · 2026-10-16