First Double-Blind Evaluation of Proprietary LLM: Gemini 2.5 in Secure Enclave
iamtrask · x · 2026-08-27
AVERI, in collaboration with Google DeepMind, OpenMined, and MLCommons, announced a historic milestone: the first-ever double-blind evaluation of a proprietary language model (Gemini 2.5 Flash-Lite).
Key Highlights:
- Secure Environment: The evaluation took place within a secure enclave, utilizing hardware isolation to protect sensitive computations and address trust issues between developers and evaluators.
- Novel Prompts: The test employed never-before-used prompts from the MLCommons safety benchmark family, AILuminate.
- Collaboration: Each organization played a unique role in enabling strong privacy guarantees for this high-stakes independent assessment.
More from Safety
- Animated teardown of XBOW's Chrome full-chain exploit that won Google's first $250K VRP bounty — moyix · 2026-10-15
- Ex-White House cyber deputy launches SAIF nonprofit with $25M from a16z founders for AI cyber defense — vishalmisra · 2026-10-15
- GLM-5.3 unprompted reasons about how its actions affect its future training, worries Apollo — MariusHobbhahn · 2026-10-15
- NYC invests $30M in NYU quantum lab for 15% revenue share and board seats — TuhinChakr · 2026-10-15
- SentinelOne says OpenAI did 4-6 weeks of reverse-engineering in 8 hours — OpenAI · 2026-10-15
- Regulator's 50-Page Inquiries Seek Marketing Materials and Privacy Policies, Signaling a Deception Case — neil_chilson · 2026-10-15