Researchers Discuss Better Question Design for Measuring Model Preferences

kromem2dot0 and repligate discuss how asking models open-ended 'A, B, or something else' questions yields more accurate preference measurements, and suggest calibrating by setting option B to something the model is already known to like.

2026-10-03 ~ 2026-10-03 · 2 related posts

• Ask 'A or B or something else?' — single-option prompts bias model preference answers — repligate · 2026-10-03
• Offering a known-liked option as 'B' reveals how much a model actually wants 'A' — repligate · 2026-10-03