ValsAI Index Finds Fireworks AI Beats DeepSeek and Moonshot's Own APIs on Agentic Tasks
sophiamyang · x · 2026-10-24
ValsAI launched the Inference Provider Index, systematically comparing open-weight models served by first-party APIs vs third-party providers on agentic tasks in finance, coding, and legal work. Unlike typical benchmarks, it reruns every two weeks to catch quality degradation, using agentic tasks (not single-turn multiple choice) across 650+ hours of model runtime. Sophia Yang highlights the surprise finding: Fireworks AI posts higher accuracy than DeepSeek's and Moonshot's own APIs as well as other third-party providers.
More from coding & agent
- Factory tested Reflection's Beam early via Together AI on real coding and agentic workflows — togethercompute · 2026-10-24
- Seroter's Reading List: CLIs Save Tokens, Agent Harness Explained, When to Use Diffusion LLMs — rseroter · 2026-10-24
- FrontierCS 2 benchmark makes agents beat original methods from code with the paper's contribution deleted — berkeley_ai · 2026-10-24
- Dev's workflow: pairing Grok bot with open-source Hermes Agent side by side — alexcovo_eth · 2026-10-24
- Anthropic's Claude Duet: a proactive agent that keeps working after you close the app — edwinarbus · 2026-10-24
- Workato's Google Workspace MCP uses one tool, 1,000+ AI-reviewed self-updating actions — adamse · 2026-10-24