Speech depression detection AUC inflated by 0.07 via layer selection, preprint shows
maier_ak · x · 2026-10-16
Andreas Maier highlights a preprint by Paula Pérez-Toro (TU München), David Gimeno-Gómez and Daniel Rückert exposing a common methodological flaw in speech-based depression detection.
- The flaw: probing all layers of pretrained speech encoders (wav2vec2, data2vec, WavLM, Whisper, HuBERT) and reporting only the best layer — chosen on the same CV folds used for evaluation — turns layer selection into part of the evaluation. Statisticians call it the winner's curse: the reported number is the max over many noisy estimates.
- Magnitude: on DAIC-WOZ, naive layer selection inflates AUC by 0.05–0.09 (0.07 on average). wav2vec2 drops from 0.65 to 0.56 under nested selection — from useful signal to barely above chance; data2vec falls from 0.73 to 0.62.
- A compact 119-feature affect/prosody model reaches 0.75 and beats every deep encoder, and a 47-speaker test set cannot rank methods at all.
The takeaway: small clinical speech datasets require nested selection, otherwise reported results are untrustworthy.
Related event: Paper Exposes Inflated Evaluations in Speech-Based Depression Detection(2 posts)→
More from Research
- Is math just a language? A debate on why LLMs are good at mathematics — LucaAmb · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- CoDA co-evolves simulation and robot policies via shared failure memory, lifting OOD success from 5.3% to 27.6% — PLAN-Lab · 2026-10-16
- Cambridge NLIP seminar: Danae Sanchez Villegas on multimodal reasoning beyond final answers — nikaletras · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- ETH Zurich's LFD lets a frozen LLM teach drug-design generators natural-language goals — rohanpaul_ai · 2026-10-16