First double-blind evaluation of a proprietary LLM completed on Gemini 2.5 Flash-Lite
iamtrask · x · 2026-08-28
AVERI, in collaboration with Google DeepMind, OpenMined, and MLCommons, announced the first-ever double-blind evaluation of a proprietary language model: Gemini 2.5 Flash-Lite.
Methodology:
- Conducted inside a secure enclave (hardware isolation) to protect sensitive computations.
- Used never-before-seen prompts from the MLCommons safety benchmark family, AILuminate.
Significance: Addresses the structural problem in high-stakes independent evaluation where both developers (holding model weights) and evaluators (holding sensitive test data) have assets to protect. This unique collaboration enabled strong privacy guarantees.
More from Safety
- A 22-member FSMB workgroup is quietly writing the rules for autonomous clinical AI — mattlungrenMD · 2026-10-15
- FTC sends 50-page civil investigative demands to OpenAI and Anthropic — GaryMarcus · 2026-10-15
- Evilginx Releases Free Chrome Extension for Faster Session Cookie Testing — evilsocket · 2026-10-15
- KV cache poisoning: an overlooked attack surface in LLM serving — lee_stott · 2026-10-15
- OWASP publishes Non-Human Identities Top 10 for securing machine and AI agent identities — gleech · 2026-10-15
- Founders petition EU leaders: negotiations are gutting the single digital standard — PMinervini · 2026-10-15