Survey of 674 AI Tool Terms: Nearly Half Restrict Benchmarking, 69 Ban It Outright
ZeroCool86 · reddit · 2026-10-20
- The author, who runs an AI tool review site, read through 674 vendor terms of service (covering model APIs, MCP servers, scrapers, payment APIs, and more), and found that 315 (about 47%) restrict benchmarking or ban using their services to build competing products: 107 explicitly name benchmarking, 69 ban it entirely, and 28 require permission before publishing results.
- The origin of such clauses traces back to Oracle in 1982: after David DeWitt's database benchmarks showed poor results, Oracle added a clause to its download license prohibiting publication of test results without permission, which persists to this day. Apollo and ZoomInfo use wording nearly identical to Oracle's, and 24 other vendors share the same boilerplate sentence—apparently legal teams copying each other.
- Some telling cases: LangSmith (LangChain's agent evaluation platform) bans publishing performance results about it; Coinbase (an x402 protocol participant) prohibits monitoring the availability and performance of its services. Some restrictions are reasonable: DeepL and SYSTRAN don't want their shared infrastructure stress-tested; Cursor allows publication as long as others can reproduce your tests.
- The author has so far only studied public terms and would like to run real tests and publish the results, but accepting an API key means accepting the no-benchmarking clause. His question: if your product is good, why fear being tested?
Related event: Survey of 674 AI Tool Terms Finds Nearly Half Restrict Benchmarking(2 posts)→
More from Companies & People
- OpenAI DevDay Exchange Tokyo draws packed pre-event community meetup — gabrielchua · 2026-10-20
- OpenAI DevDay Tokyo slide reveals plugin ecosystem has grown past 4,000 integrations — gabrielchua · 2026-10-20
- AI lab Decide signs 12-month commercial partnership with Europe's Morvdesk for financial AI deployment — saheedniyi_02 · 2026-10-20
- DoorDash runs 130K automated tasks a month: inside 8 AI software factories — Pavan_Belagatti · 2026-10-20
- Japan emerges as world's second-largest market for OpenAI's dots by active users — gabrielchua · 2026-10-20
- -bold_lab AI opens applications for Fall 2027 cohort to pursue the next paradigm of open-source AI — j_foerst · 2026-10-20