Frontier Models Are Bad at Hearing ASMR: 8 LLM Judges vs Human Labels on 480 Clips

MissAppleby · reddit · 2026-10-22

A developer training text2asmr (text-to-ASMR audio generation) benchmarked 8 frontier models as audio labellers on 480 clips; they performed far worse than expected, showing frontier audio models still can't hear fine-grained foley.

Setup: same prompt and 20 sound labels per judge; external clips from FSD50K/ESC-50 measure recall, human-labelled 6-second ASMR clips from their site measure F1.

Key findings:

Gemini 3.1 Pro's trigger F1 vs people was just 0.325, GPT Audio 0.264; spectrogram-image judges Claude Opus 5.5 and Grok 4.7 scored only 0.121 and 0.110
Clean sound effects ≠ ASMR: Gemini 3.1 Pro named brushing on 92% of clean FSD50K clips but F1 was 0.18 on real ASMR
Paper is nearly invisible: named only 4 times out of 40 paper/page-turn clips by both Gemini judges
Voices are fine: whisper F1 hit 0.70–0.79; physical triggers are where models collapse
Each judge has a favourite wrong answer (Gemini 2.5 Pro said "sticky" 59 times where nothing sticky was labelled)
Cost gap: Gemini 3.1 Pro at $0.0088/clip vs MiMo at $0.0008

Method: all model judges plus an AudioSet AST tagger, energy detector, and custom CLAP model go into Dawid-Skene fusion, weighted per class by measured reliability; uncertain clips and classes no judge can hear go to humans first. The student model CLAP-ASMR (fine-tuned laion/clap-htsat) improved test mAP from 0.516 to 0.549 with 2,400 Freesound clips. A crowdsourcing site (asmrboard) has collected 1,056 labels from 34 people; weights and labels will be released open-source.

Original post →

Frontier Models Are Bad at Hearing ASMR: 8 LLM Judges vs Human Labels on 480 Clips

More from Multimodal