FULL STORY

OpenAI's Safety Evaluation Saga: From Ban Rumors to Restricted Access

After a researcher claimed OpenAI was barred from running safety evals on Claude, OpenAI announced embedded evaluations with four external firms. Apollo's disclosure of restricted access sparked criticism that the program falls short of employee-level transparency.

2026-11-16 ~ 2026-11-17 · 4 episodes · 29 posts

Episode 1 · OpenAI apparently barred from running safety evals on Anthropic models, sparking cross-lab debate (2026-11-16, 6 posts)

AI safety researcher Adam Karvonen revealed on November 16 that OpenAI apparently is not allowed to run its safety evaluations on Anthropic's models. He had wanted to test Claude on OpenAI's "covert underperformance" eval but couldn't proceed, sparking a discussion about whether frontier labs can cross-test each other's models.

Confirmed

  • Karvonen said OpenAI appears to be barred from running its safety evals on Anthropic models (a claim repeated in two posts, m1 and m4).
  • In follow-up discussion, Karvonen noted that if the concern comes from OpenAI's side, the evals could simply be run on AWS Bedrock; if it's about IP protection on Anthropic's side, OpenRouter plus a VPN would easily bypass it.
  • Responding to anpaure's ideal proposal of "handing evals to third-party auditors," Karvonen was skeptical, arguing it would add friction to the evaluation process and that having the eval's original authors run it themselves is usually better—tasks like interpreting qualitative results, patching judge issues, and checking for reward hacking are delicate and easily distorted when outsourced.

Differing views

  • anpaure sees the ban on cross-testing as more of a "data hygiene" convention: companies simply don't run rival models in their own environments, not an attempt to hide the evals themselves; their preferred solution is third-party audit firms running the evaluations.
  • Karvonen favors having the evals' original authors run them directly, questioning whether third-party audits would introduce friction and distortion.

Why it matters

  • The dispute touches a core question of AI safety governance: whether and how safety evaluations by frontier labs can be run independently on competitors' models.
  • If the restriction is real, it could weaken cross-lab validation of risky model behaviors like "covert underperformance"—though the existence of easy workarounds suggests such restrictions have limited practical force.

Episode 2 · OpenAI Launches Embedded Assessments with Four External Safety Labs (2026-11-17, 19 posts)

On November 17, OpenAI announced "embedded assessments" in partnership with four external organizations — Apollo Research, METR, Transluce AI and Goodfire AI — giving outside experts deep access to its internal models and systems to independently investigate alignment and test safety guardrails. It's seen as a significant pilot in a frontier lab opening internal access to external safety researchers.

Confirmed

  • The partners are Apollo Research, METR, Transluce and Goodfire; the announcement came from OpenAI, with multiple accounts reporting it consistently
  • METR has reached an agreement with OpenAI for a limited-scope alignment assessment pilot, focusing on the alignment properties of OpenAI's largest RL training runs, and has committed to publishing one or more reports with public results and a transparent description of the evaluation process
  • Transluce plans to start its pilot in early December, with the team embedded on-site at OpenAI for six weeks to independently evaluate the scope and quality of its post-training alignment evaluations, publishing results in a public report; per robinomial, this is the first such on-site collaboration by an external organization
  • Per MariusHobbhahn, Apollo is participating first in the initial phase of evaluations

Why it matters

  • External safety assessments have largely relied on black-box API testing; embedded assessments let independent researchers directly examine model internals and training processes, markedly improving auditability and independence
  • The evaluations cover the largest RL training runs and post-training alignment evaluations — core parts of frontier model safety; all parties have committed to public reports, and the transparency of the results will directly shape industry confidence in this model

Episode 3 · Apollo Reveals Limited Access in OpenAI Evaluation Pilot, Drawing Criticism (2026-11-17, 2 posts)

Apollo Research disclosed that its OpenAI pilot only allows discussion of confidential info with designated staff, with no access to Slack. Critics say this falls short of the employee-level access Sam Altman promised.

Episode 4 · OpenAI Reportedly Restricts Apollo Evaluators with Whitelists and Gag Terms (2026-11-17, 2 posts)

Leaked terms and disclosures by ex-employee Daniel Kokotajlo reveal OpenAI restricted Apollo Research evaluators to a whitelist of staff, required notice of every conversation, and barred publishing results without approval.