Do annotation tasks really need multiple questions? A debate on labeling signal design
dmnsl1 · x · 2026-10-19
A developer questions whether any real annotation use case requires multiple questions: most could be single-choice questions or multi-label outputs with pointwise scoring and proper thresholds, and multi-question serving could avoid state-question cross-attention with a properly forged state.
LightOn researcher Antoine Chaffin replies that difficulty, correctness, and interestingness are distinct signals; guidelines often come with checklists; multiple questions can serve as a data representation method. But he agrees such tasks feel marginal enough to just pay for multiple inferences — or encode question info within the state if batching is needed.
Related event: Ex-OpenAI Researcher Questions Design of Multi-Question Decision Models(3 posts)→