Telling Gemini you're Karpathy makes it think 30% longer and score 7.8 vs 5.8 in blind test
dejanseo · x · 2026-11-12
Dan Petrovic of DEJAN ran a controlled experiment: changing only the "User name" in the system prompt, he asked Google's latest Gemini 15 expert tasks 15 times each, then had a separate AI rater blindly score all 225 answers.
Key findings:
- Answers written for recognized experts (Eric Topol, Karpathy, Fei-Fei Li, etc.) averaged 7.8/10, vs 6.8 for a random invented name and 5.8 for no name
- Experts scored highest in 12 of 15 topics, with medicine and ML showing the biggest boost (+2.2)
- Gemini thought longer for experts: 3,158 thinking tokens per answer vs 2,419 (random name) and 1,984 (no name)
- The model never mentioned the expert's name in any of the 75 expert answers, suggesting the gain comes from more internal reasoning rather than flattery
Takeaway: telling the model who you are — especially a recognized domain expert — measurably improves response quality. Full data and reproduction scripts are open-sourced.
More from Models
- Ben's Bites: Anthropic builds office suite inside Claude, Gemini 4 Carbon rolls out — Ben's Bites · 2026-11-12
- Unsloth's new quantization makes local Qwen Flash Next feel almost like Qwen4 — danielhanchen · 2026-11-12
- Benchmark criticized for cherry-picking known failures; same model scores 75% vs 56% across harnesses — tallmetommy · 2026-11-12
- RF-DETR Large: best OSS-licensed object detector at just 34M params — TheZachMueller · 2026-11-12
- Continu Intel launches ci-proto: tops fev-bench with 64x context window, open weights — s_mohinii · 2026-11-12
- Reddit user finds no single LLM wins for agents: speed, accuracy, cost tradeoffs — LeadWithHitesh · 2026-11-12