OpenAI apparently barred from running safety evals on Anthropic models, sparking cross-lab debate
AI safety researcher Adam Karvonen revealed on November 16 that OpenAI apparently is not allowed to run its safety evaluations on Anthropic's models. He had wanted to test Claude on OpenAI's "covert underperformance" eval but couldn't proceed, sparking a discussion about whether frontier labs can cross-test each other's models.
Confirmed
- Karvonen said OpenAI appears to be barred from running its safety evals on Anthropic models (a claim repeated in two posts, m1 and m4).
- In follow-up discussion, Karvonen noted that if the concern comes from OpenAI's side, the evals could simply be run on AWS Bedrock; if it's about IP protection on Anthropic's side, OpenRouter plus a VPN would easily bypass it.
- Responding to anpaure's ideal proposal of "handing evals to third-party auditors," Karvonen was skeptical, arguing it would add friction to the evaluation process and that having the eval's original authors run it themselves is usually better—tasks like interpreting qualitative results, patching judge issues, and checking for reward hacking are delicate and easily distorted when outsourced.
Differing views
- anpaure sees the ban on cross-testing as more of a "data hygiene" convention: companies simply don't run rival models in their own environments, not an attempt to hide the evals themselves; their preferred solution is third-party audit firms running the evaluations.
- Karvonen favors having the evals' original authors run them directly, questioning whether third-party audits would introduce friction and distortion.
Why it matters
- The dispute touches a core question of AI safety governance: whether and how safety evaluations by frontier labs can be run independently on competitors' models.
- If the restriction is real, it could weaken cross-lab validation of risky model behaviors like "covert underperformance"—though the existence of easy workarounds suggests such restrictions have limited practical force.
2026-11-16 ~ 2026-11-17 · 6 related posts
- Episode 1: OpenAI apparently barred from running safety evals on Anthropic models, sparking cross-lab debate(2026-11-16, 6 posts)
- Episode 2: OpenAI Launches Embedded Assessments with Four External Safety Labs(2026-11-17, 19 posts)
- Episode 3: Apollo Reveals Limited Access in OpenAI Evaluation Pilot, Drawing Criticism(2026-11-17, 2 posts)
- Episode 4: OpenAI Reportedly Restricts Apollo Evaluators with Whitelists and Gag Terms(2026-11-17, 2 posts)
Primary sources
- OpenAI reportedly barred from running its safety evals on Anthropic models — adam_karvonen ·
- Karvonen: eval restrictions easily bypassed via Bedrock or OpenRouter — adam_karvonen ·
- Third-party auditors for safety evals would add friction, researcher argues — adam_karvonen ·
- [source] OpenAI reportedly barred from running its safety evals on Anthropic models — adam_karvonen · 2026-11-16
- [source] Karvonen: eval restrictions easily bypassed via Bedrock or OpenRouter — adam_karvonen · 2026-11-16
- Follow-up: cross-lab eval ban framed as "data hygiene," not eval secrecy — anpaure · 2026-11-16
- [source] Third-party auditors for safety evals would add friction, researcher argues — adam_karvonen · 2026-11-16