DEJAN open-sources full data behind its Gemini expert-name experiment: 225 blind-rated answers
dejanseo · x · 2026-11-12
DEJAN released an interactive dashboard and full dataset for its experiment showing Gemini scores higher when the user is named as a recognized expert. 15 topics × 15 asks (expert name / random name / no name), all 225 answers blind-scored 1-10 by a separate Claude session.
The dashboard ranks experts by score boost and shows per-topic blind scores and thinking token counts. The download includes answers.csv (all answers with thinking summaries and rater reasons), tasks.csv, a SQLite database, and reproduction scripts — fully transparent methodology.
More from Research
- Alibaba's Co-Design Bench: top coding agent fully completes just 26.4% of real user requests — krishnan · 2026-11-12
- "Hard to tell a 92% model from a 99% model": the bottleneck is evals — chris_j_paxton · 2026-11-12
- Smartwatch PPG + accelerometer signals reframed to detect hypertension signs — CurieuxExplorer · 2026-11-12
- Error analysis beats generic metrics: a practical framework for AI evals — PawelHuryn · 2026-11-12
- SteerDistill: teacher-guided quantization keeps W4A16 agents within 1 point of BF16 — _akhaliq · 2026-11-12
- MIMIC and BMW train video-action model hitting up to 100% success on industrial tasks — pess_r · 2026-11-12