Senior-SWE-Bench joins Fireworks' eval index as 31 partners sign on
ajratner · x · 2026-10-29
- Fireworks AI launched its Specialized Intelligence Index, aggregating practitioner-contributed public and private domain evals with 31 new partners, comparing open, closed, and specialized models by native score, per-task cost (USD), and wall-clock duration.
- Senior-SWE-Bench, built by Snorkel AI with Princeton and UW-Madison researchers, is now included: 100 tasks across 12 open-source repos assessing coding agents on realistic open-ended work — complex features, migrations, bugs, and performance issues.
Related event: Fireworks Expands Practitioner Eval Index with 31 Partners(2 posts)→
More from coding & agent
- grep.ai opens swarms to everyone: how a thousand agents work as a team — garrytan · 2026-10-29
- After a year of Compound Engineering, Kieran Klaassen hasn't written a line of code — kieranklaassen · 2026-10-29
- tscircuit ordered 50 AI-generated PCB designs in 2.5 years, built own EDA kernel to catch errors — debreuil · 2026-10-29
- things-cli: Open-Source CLI and MCP Server Lets Claude Manage Your Things 3 Tasks — JordanMorgan10 · 2026-10-29
- Simon Willison: October 2026 is the inflection point for vibe game development — vista8 · 2026-10-29
- AWS Blueprint for an Enterprise Agentic AI Platform on Bedrock AgentCore — AWS ML Blog · 2026-10-29