DeepMind and partners complete first double-blind evaluation of a proprietary frontier model
On August 27, Google DeepMind, together with AVERI, OpenMined and MLCommons, announced the industry's first double-blind evaluation of a proprietary frontier language model, with Gemini 2.5 Flash-Lite as the test subject and results published as a pilot report by AVERI. The mechanism, run inside a secure enclave, reveals neither test prompts to the model provider nor model weights to evaluators, charting a viable path for independent, trusted third-party evaluation of closed models.
Confirmed
- DeepMind officially announced the pilot of the industry's first double-blind evaluation mechanism for proprietary frontier models: testing takes place in a secure environment that protects both test prompts and model weights, ensuring external safety and performance evaluations remain private, robust and trustworthy, and preventing benchmark contamination from models seeing test questions in advance.
- Partners include AVERI, OpenMined and MLCommons; the evaluation used trusted execution environment (TEE) hardware isolation for weights and prompts, and tested Gemini 2.5 Flash-Lite with MLCommons' AILuminate safety benchmark. OpenMined founder Andrew Trask (@iamtrask) relayed that the secure environment was created with OpenMined's PySyft framework, relying on cryptographic guarantees, weight protection and uncontaminated benchmarks; @heigazen added that it builds on Google Cloud's Confidential Computing capabilities.
- @HaydnBelfield and @MilesBrundage confirmed this is the first double-blind evaluation of a closed model inside a secure enclave — Google DeepMind could not see the evaluation prompts, and evaluators could not see the model weights; @JMateosGarcia added that the framework has been successfully tested on non-public evaluation sets.
- AVERI released a pilot report detailing the world's first double-blind evaluation of a proprietary language model. @rseroter noted DeepMind also worked with the Singapore AI Safety Institute, OpenMined and others, using an encrypted "black box" environment to prevent models from seeing evaluation questions in advance.
Why it matters
- Benchmark contamination is a core pain point in LLM evaluation: if a model has seen benchmark questions during training, public benchmark scores become distorted. Double-blind evaluation in secure enclaves protects both prompts and weights, balancing evaluation independence with the provider's commercial confidentiality.
- This could push the industry toward new evaluation standards, letting external researchers robustly evaluate proprietary models without weight access. The progress was amplified by AI governance and safety researchers including Miles Brundage, Andrew Trask and Sebastian Krier, showing strong attention in the AI safety and evaluation community.
2026-08-27 ~ 2026-08-28 · 19 related posts
Primary sources
- [source] Google DeepMind pilots first double-blind evaluations for frontier AI — GoogleDeepMind · 2026-08-27
- DeepMind pilots double-blind evaluations to fix benchmark contamination — Miles_Brundage · 2026-08-27
- Report details first double-blind eval of LLM using secure enclaves — Miles_Brundage · 2026-08-27
- First double-blind evaluation of proprietary model in secure enclave — Miles_Brundage · 2026-08-27
- MLCommons completes first double-blind evaluation of a closed-weight AI model — iamtrask · 2026-08-27
- Google DeepMind Pilots First Double-Blind Evaluation for Frontier AI Models — iamtrask · 2026-08-27
- Google DeepMind achieves first successful 'double blind evals' test — JMateosGarcia · 2026-08-27
- First double-blind eval of proprietary model via secure enclave — Miles_Brundage · 2026-08-28
11 near-duplicate retellings: HaydnBelfield · iamtrask · iamtrask · rseroter · sebkrier · iamtrask · Miles_Brundage · sethlazar · iamtrask · iamtrask · heiga_zen