Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks
maier_ak · x · 2026-10-16
The author presents evidence that evaluation in affect/prosody modeling is confounded by selection effects:
- With shuffled labels, naive best-of-25 still reaches AUC 0.59 vs 0.50 nested, replicated across 500 permutations and in Italian — many reported signals come from selection, not real capability.
- Counterintuitive findings: a compact 119-feature affect and prosody model, done properly, reaches 0.75 and beats every deep encoder; and a 47-speaker test set cannot rank methods at all.
Takeaway: small test sets plus heavy method-selection freedom render benchmark results meaningless.
Related event: Paper Exposes Inflated Evaluations in Speech-Based Depression Detection(2 posts)→
More from Research
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16
- Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast — weskambale · 2026-10-16
- Sakana AI ships Agentic Scientist, orchestrating coding agents for automated scientific discovery — hardmaru · 2026-10-16
- MAI-Base-2 adopted the 1/(weight_decay·lr) timescale hparam scaling rule — SeunghyunSEO7 · 2026-10-16
- 108M-parameter text-to-image denoiser Surjo-Image-Preview lands on Hugging Face — Consistent-Rule-3720 · 2026-10-16
- Lean proof formalizes optimal packing of 19 unit squares, side ≥ 3 + 4√2/3 — ctjlewis · 2026-10-16