Almost half of 674 AI tool vendors forbid benchmarking, terms-of-service audit finds
ZeroCool86 · reddit · 2026-10-20
A developer running an AI agent tool grading site read 674 vendors' terms: 315 restrict benchmarking or building competitors — 107 name benchmarking, 69 ban it outright, 28 require permission to publish results. LangSmith bans publishing performance data, Coinbase bans monitoring its availability, and Apollo/ZoomInfo clauses mirror Oracle's infamous 1982 anti-benchmark clause nearly word for word; 24 vendors share the exact same template sentence. The author argues Cursor's model (publish freely if the test is reproducible) should be the norm, and that blanket benchmark bans make evidence-based agent tool selection impossible.
Related event: Survey of 674 AI Tool Terms Finds Nearly Half Restrict Benchmarking(2 posts)→
More from coding & agent
- Mermaid rewritten again in Rust: Ondine is 13,467x smaller and 670x faster — kalyan_kpl · 2026-10-20
- AWS says AI agents can skip human approval if their rules pass a math check — shashib · 2026-10-20
- ModelScope releases MemT-1.0: models with built-in persistent memory, 1.86x faster at 128K context — kalyan_kpl · 2026-10-20
- We replaced our docs platform's MCP server with FrontMCP — lessons building a docs MCP — DavidAntoon · 2026-10-20
- My personal agent is just Codex running on my home computer, and it beats dedicated apps — Lazy_Angelina · 2026-10-20
- Agent Standup turns your Claude Code sessions into a live team video call, open source — stickygoose · 2026-10-20