Evaluating Small Models: Multiple-Choice Tests Are Easy to Game
mervenoyann · x · 2026-07-14
AI researcher Merve Noyant points out that when evaluating large models, most multiple-choice question assessments (MCQA) actually make it easy for models to "cheat": even if a model gets stuck in a loop or hallucinates, the structure of the multiple-choice options helps it organize and anchor the correct answer. Consequently, she prefers researching small models and emphasizes the importance of conducting "vibe tests."
Related event: Multiple-Choice Evaluations May Overestimate LLMs(2 posts)→
More from Research
- Greg Brockman announces AI-assisted discoveries in math, CS, and quantum field theory — gdb · 2026-10-27
- AI is simultaneously flooding journals with slop and cleaning up human slop — _akpiper · 2026-10-27
- CMU-led team proposes "epiplexity," a new information measure for computationally bounded learners — davidmanheim · 2026-10-27
- Four Gemini agents beat Minecraft autonomously in 27 hours, writing 2,783 programs — rohanpaul_ai · 2026-10-27
- OpenAI PDF claims refutation of ETH, solving 3SAT in exp(n/(log n)^(1/2+o(1))) — rrwilliams · 2026-10-27
- OpenAI Outlines Plan to Build Scientific Understanding with Academia — Chemical-Agency-3997 · 2026-10-27