Renaming PDF to 'final_draft' Boosts LLM Review Scores: Evaluation Weirdness
CShorten30 · x · 2026-08-11
A user observed that when using LLMs to review academic papers, simply renaming the uploaded file from paper.pdf to paperfinaldraftpdfreadyforreview.pdf significantly boosts the model's average scores.
Addressing this weirdness in LLM reviews, a researcher pointed to a paper on 'Reasoning-intensive Regression,' noting that while such biases are unexpected, the problem is solvable through specific approaches.
Related event: LLM Peer Review Fooled by 'Final Draft' Filenames(3 posts)→
More from Fun
- Meme: We develop AGI, and AGI says Yann LeCun was right — BLUECOW009 · 2026-08-11
- The Challenge of 5,000 Employees Building a Usable App — sandylikesfrogs · 2026-08-11
- Humanity vs. Robots Decathlon: Reddit Debates Events Where Humans Win — OxidusRouge · 2026-08-11
- Developer Creates Standup Channel for AI Agents, One Apologizes for Weekend Absence — GabGarrett · 2026-08-11
- Grok Build Remakes Classic Winamp 2 for macOS, Open-Sources C11 Code — Daniel_Farinax · 2026-08-11
- Heartbreak by Cron Job: Woman Dumps Boyfriend for Automating Relationship with AI Agent — tech__unicorn · 2026-08-11