Gemini 3.7 Flash Performance Drops Below Gemma Due to OpenRouter Backend Variances
mariofilhoml · x · 2026-08-14
The author found that the newer Gemini 3.7 Flash performed worse than Gemma on a structured text extraction task. Upon investigation, they discovered that when using Gemini models via OpenRouter for structured output, the underlying backend (Vertex vs. AI Studio) causes wildly different performance.
This serves as a warning for developers to account for backend discrepancies when running model evaluations.
Related event: Gemini 3.7 Flash Structured Output Varies by Backend(2 posts)→
More from Models
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- One flagged seat froze an entire Claude team org — founders share backup plans — RexDouglass · 2026-10-16