2026-08-11
An information-theoretic framework measures meaning per channel: for sarcasm, audio carries ~14x the information of text; for emotion ~9x; for questionhood just 2.4x.
Transcribing speech to text strips away more than background noise. Prosody, the pitch, tempo, and loudness of an utterance, vanishes the moment you keep only the words, and prosody is exactly what signals whether a sentence is sarcastic, angry, or a question. The question is how much is lost, and which kinds of meaning go missing.
Prior work used mutual information to measure how much text and prosody overlap, which only shows the two channels are redundant. It says nothing about what that prosodic information is about, or how much of it text cannot recover. This ACL 2026 paper closes that gap: for a specific dimension of meaning (say, whether an utterance is sarcastic), it decomposes how much information the audio and the text each contribute.
The core idea is to split a meaning feature Y (sarcasm, emotion, or questionhood) into a sum of mutual-information terms. Two steps make it concrete:
Why audio A instead of prosody P directly? No pretrained model consumes prosody features alone, while speech encoders (Whisper, wav2vec 2.0) are mature. The authors argue the approximation is sound: the extra term MI(Y;A|T,P) mostly reflects speaker identity and background noise, which are irrelevant to the target meaning.
Text classifiers use the GPT-2 family (tiny through xl); audio classifiers use Whisper and wav2vec 2.0 encoders with a classification head. Sarcasm uses MUStARD (690 sitcom utterances, balanced); emotion uses MSP-Podcast (10 emotion classes); questionhood is built by segmenting and aligning MSP-Podcast. Each setting runs 20 hyperparameter seeds and picks the best.
Mutual information each channel contributes (bits):
| Feature | H(Y) | Audio MI(Y;A) | Text MI(Y;T) | Audio/Text |
| Sarcasm | 0.99 | 0.22 | 0.016 | about 14x |
| Emotion | 2.58 | 0.52 | 0.06 | about 9x |
| Questionhood | 0.98 | 0.65 | 0.27 | 2.4x |
For sarcasm and emotion, the two features that hinge on tone, the audio channel carries an order of magnitude more information than text. As uncertainty coefficients MI(Y;A)/H(Y), both land near 0.2, meaning audio resolves about a fifth of the uncertainty in these features while text barely touches them.
Questionhood flips the pattern. English questions carry strong textual cues (subject-auxiliary inversion, wh-words), so prosody adds less and the audio advantage shrinks to 2.4x, exactly as predicted.
The classifiers are strong in absolute terms: the best sarcasm model beats the Castro et al. (2019) baseline by 10 accuracy points, and the best emotion model matches published 2024 results. Audio models consistently beat text models, and Whisper medium and large beat both wav2vec 2.0 and audio-plus-text models.
For anyone building voice AI, the takeaway is direct: signals like sarcasm and emotion are mostly gone once you transcribe, so the model has to consume audio directly. Customer-service bots and voice assistants that operate on a single utterance with no long context lean hardest on prosody. Prior work suggests the unique information prosody carries is mostly local, so adding longer context does not erase its contribution.
The larger value is a reusable framework. Swap in another meaning dimension (syntactic ambiguity, turn-taking), another channel (gesture, facial expression via vision-language models), or another language, and the same decomposition applies. For linguistic typology, it offers a finer ruler for studying how different languages distribute information across channels.
The one to remember: text models were fine-tuned only up to GPT-2 XL, not the strongest LLMs available, because of compute. The authors state this explicitly. MI(Y;T) is therefore underestimated, and the audio-over-text advantage may be inflated. Treat the order-of-magnitude headline with that discount.
Second, approximating MI(Y;P|T) with MI(Y;A|T) is a workaround. The extra term includes speaker identity, laughter, and filler words, not pure prosody. Isolating prosody cleanly needs a model that consumes prosody features alone, which does not yet exist.
The authors also note: experiments use a single sentence with no discourse context (adding it would raise MI(Y;T)); only English is tested, and tone languages may differ; the sarcasm dataset is artificially balanced though real sarcasm is rare; emotion uses single labels over only 10 classes, though one utterance often carries several.