Using Claude Subscriptions for Evals Likely Violates Anthropic's ToS, Bench Author Says
JoshPurtell · x · 2026-11-07
Bench author Josh Purtell explains his leaderboard's design: comprehensive OSS coverage, but the closed reference only runs OpenAI models via a Codex subscription, plus Muse Spark 1.2 to show how bad Muse open is despite being promised as 1.2 OSS. He estimates 90% of people avoid using Claude subscriptions for evals since it likely violates Anthropic's draconian ToS—even though he suspects Anthropic's models would score well.
Related event: Community autoresearch coding benchmark: open models underwhelm(2 posts)→
More from coding & agent
- CodeRabbit podcast: ship less, not more, in the AI era — nutlope · 2026-11-07
- 16 MCP servers, 35 parallel agents — and it still ships slop — weswinder · 2026-11-07
- Creator apps like Instagram Edits and Roblox Studio are shipping with built-in MCPs for agents — trq212 · 2026-11-07
- Cartesian agent builds a 289-part Dymaxion car 3D model live, researching the design online — carlosbannon · 2026-11-07
- ArkenOS visualizes hundreds of autonomous agents collaborating as a 3D research swarm — _AustinCalvert_ · 2026-11-07
- AI agents blocked by Zillow, Indeed, Reddit — a scraping API got 3/3 through in testing — eyishazyer · 2026-11-07