Asking Gemini 10,701 questions: Google Search rewrites ~45% of brand picks

dejanseo · x · 2026-11-18

SEO firm DEJAN ran a large-scale experiment: each of 10,701 brand-recommendation questions was asked twice to Gemini 4 Hydrogen — once from memory, once with Google Search grounding.

Results: on average 45% of brands in grounded answers (3.4 of 7.5) came from search; only 1.8% of answer pairs were identical. Grounded thought summaries were 3.5x longer. Notably, in 21% of grounded summaries the model mentioned a "mental check" or "simulated search", sometimes adding "if this were a live system" — while live searches were actually running. Search also overrode stale memory: of 70 outdated brands named in those passages, only 1 reached the final answer.

Data came from daily AI-visibility tracking across 3,503 distinct prompts, 119 sites and 17 languages (62% English), analyzed purely by counts and pattern matching with no LLM scoring.

Original post →

More from Models

Models channel →