HCI textbook author: paired t-intervals beat CLMMs for small-sample Likert data
IanArawjo · x · 2026-10-19
HCI researcher Ian Arawjo, preparing to release a textbook on HCI methods, tackles how to report confidence intervals for Likert data in a within-subjects study with 12 participants and 3 conditions. Complex approaches like CLMMs 'print garbage' at this sample size.
Based on his own simulation studies, paired t-intervals perform reasonably well (if conservative) even on non-normal LLM evaluation outputs, and may be safer in practice than fancier solutions. He reports that nonparametric CIs via Wilcoxon and CLMMs both fail badly on this data. A Bayesian Normal-Inverse-Gamma method with a weakly informative prior edged out logit-transformed t-intervals but introduces philosophical inconsistency that's hard to defend to reviewers — so for simplicity, the paired t-interval remains a good choice. The real problem, he notes, is the 12-participant sample size itself, which HCI is unlikely to fix.
Related event: Paired t intervals beat CLMM for small-sample Likert data, researcher says(3 posts)→
More from Research
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23
- DRATS: Adaptive Task Sampling Boosts Worst-Task Performance in Multi-Task RL — Cohere · 2026-10-23