Gemini Decision 1 Benchmarks Leak: Leads Most Evals With 0.025 ECE
Angaisb_ · x · 2026-11-12
A leaked eval table shows Gemini Decision 1 (Decisions API) looking strong: Jevbench 92.2%, Amazon ESCI 58.7%, New Yorker Caption 79.7%, PubMedQA 79.8%, Global-MMLU-Lite 86.0%, MMAU 71.8%, Video-MME 69.1% — ahead of TypeSafe Jev 1.14, GPT-6 Luna, and Gemini 3.5 Flash-Lite on most benchmarks.
- Calibration is a standout: Top-1 ECE of 0.0250 vs 0.0494/0.0721/0.2104 for rivals.
- Total price $0.88 — pricier than TypeSafe Jev ($0.68) but cheaper than GPT-6 Luna ($0.93).
- Mind2Web Multimodal (86.2%) and When2Call (79.9%) trail TypeSafe Jev.
Related event: Gemini Decision 1 debuts: tops decision benchmarks but costs 3x rivals(6 posts)→
More from Models
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Grok spotted building meeting "Listen to Conversation" feature with live transcript — nima_owji · 2026-11-12
- MoonMath ships uncensored 763B DeepSeek-V4.1 via closed-form weight editing, claims zero capability loss — udmrzn · 2026-11-12