Reverse Turing test for speech: humans score 0.71 vs Gemini 0.54 in cocktail-party transcription
cloud_kj · reddit · 2026-10-29
A developer built an audio-CAPTCHA-style benchmark: a synthetic target sentence mixed over staggered background voices (4 dB attenuation), tested against 146 human transcriptions and 8 ASR configs. Humans averaged 0.71 word accuracy vs 0.54 for instructed Gemini 3.8 Flash and 0.55 for Qwen3-ASR 0.6B; pass rates 77% vs 55%. The gap (+0.13, CI +0.03–0.23, AUC 0.62) is too narrow for a reliable CAPTCHA—a single instruction prompt lifted Gemini from 0.34 to 0.54. Code, data and audio corpus are open-sourced as crosstalk.
More from Research
- DOE's Genesis Mission Awards $159M to 12 Superintelligence-for-Science Projects — ingliguori · 2026-10-29
- Kaggle kicks off Bench Summit, arguing today's AI benchmarks fail on relevance, compute and verification — MeganRisdal · 2026-10-29
- MIT team finds a literal map of the world inside Llama-2's activations — burny_tech · 2026-10-29
- SWE-sweep appears in a Google DeepMind frontier model card within a month — OfirPress · 2026-10-29
- Stanford colloquium: choosing insight over overload in the age of AI — jonc101x · 2026-10-29
- A 2-neuron +0.75 tweak makes 'evil Qwen': So8res on what steering really reveals — builderjaydub · 2026-10-29