EmoHarness: scaffolding lifts small models' multimodal emotion recognition by up to 13.4 points

UIUC-CS · hf · 2026-11-11

Researchers from UIUC, UC Davis and Stanford present EmoHarness (arXiv, Nov 9, 2026): small multimodal models misjudge emotion when words, voice and visual cues conflict. Rather than touching the model, they wrap the frozen student inside a harness — the prompts, settings and verification code around its calls.

How it works

A human-written starting harness (scaffold) numbers the student's visual observations, transcript and prosody descriptions as citable evidence, and has the student weigh candidate emotions against them.
Meta-Harness Evolving (MHE): a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide what's kept, and the Builder never sees evaluation items.

Results

On 5,507 MME-Emotion items, task-macro Recognition rises 7.6–13.4 points over a single video-only call across four students and both Builders.
Ablations: transcripts and prosody supply most of the boost (+5.38 to +10.71); the scaffold contributes -0.79 to +2.54 on audio-augmented calls; evolved edits add at most +0.95 and none survive Holm correction.
The emotion context transfers to IntentQA with small gains that also don't survive correction.

Takeaway: EmoHarness helps small multimodal students chiefly by supplying and organizing audio-derived evidence.

Original post →

EmoHarness: scaffolding lifts small models' multimodal emotion recognition by up to 13.4 points

More from Research