OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction
Jiahao Huang, Zheng Lian, Jingyi Zhang, Zhide Chen, Xiaojiang Peng, Shaonan Wang
cs.HC
2026-08-06
OneEmo folds eight emotion tasks (perception, understanding, interaction) into a 4.5B model trained on EmoWorld-130K and an Emo-Chord RL recipe, beating same-scale and even 16B rivals on most benchmarks.
Multimodal emotion models tend to specialize in a single task: recognition here, humor there, with no sharing between them. The authors argue this fractures three things that should reinforce each other: emotion perception (reading expression and tone), emotion understanding (catching sarcasm, humor, intent), and emotion interaction (empathetic replies, emotional support). They share underlying representations, so training them apart wastes data and leaves reasoning potential untapped. OneEmo tries to do all three in one model.
OneEmo unifies eight tasks across three domains: perception (multimodal sentiment analysis, basic emotion recognition, open-vocabulary emotion recognition), understanding (intent recognition, humor understanding, sarcasm understanding), and interaction (empathetic response generation, emotional support conversation).
Training has two parts. The data side, EmoWorld-130K, builds samples with explicit reasoning trajectories through a three-stage pipeline: theory-based distillation using psychological frameworks (Ekman's basic emotions, incongruity-resolution theory, appraisal theory, DSM/ICD-11 criteria), automated reverse verification for factual consistency, then re-annotation by psychology postgraduates with a 10% random audit. Supervised fine-tuning on this corpus already shows multi-task mutual gains.
The second part is the Emo-Chord RL recipe: an off-policy cold start stabilizes a baseline, then GRPO drives hybrid optimization. Crucially it adds a dynamically weighted SFT auxiliary loss to stop capability degradation during exploration, splits reward into format, thought, and answer components under a unified allocation, and uses task-aware linear-decay gating to keep reasoning chains from ballooning.
OneEmo has 4.5B parameters. Representative perception numbers:
| Task (metric) | Result |
| MSA SIMS (F1) | 85.54 |
| Basic emotion recognition MER'24 (hit rate) | 80.21 |
| Open-vocabulary emotion recognition (emotion wheel) | 70.42 |
In understanding, sarcasm hits WAF 74.16, humor 72.48, and intent recognition only 56.46. It beats same-generation open generalists including Cosmos3-Nano (16B), and on basic and open-vocabulary recognition it edges past GPT-5-mini and MiMo-v2.5 (310B) despite a 70× parameter gap. In blind human evaluation, empathetic replies beat GPT-5-mini 72% of the time and Qwen3.5-9B 80%; emotional support wins 57% and 84% respectively. Ablations hold up: dropping the answer reward costs 2.97 points across the eight tasks, the format reward 2.70, the thought reward 1.16, and skipping the cold start to go straight to RL degrades hard tasks like intent recognition sharply. On both the Qwen3-VL-4B and InternVL-3.5-4B backbones, all eight tasks improve in a Pareto sense.
Affective computing has been stuck with many strong single-task specialists but few general models. OneEmo shows a 4.5B model can span perception, understanding, and interaction, and the Emo-Chord recipe (SFT auxiliary loss plus length gating) is reusable for other multi-task RL settings where you cannot afford to lose baseline competence. For teams building companion, customer-service, or mental-health applications, it is a ready open-source baseline with code released.
The authors concede several points: current datasets are largely scripted cinematic material, with little authentic in-the-wild interaction; the task spectrum is narrow, omitting continuous emotion forecasting and long-term companionship; emotion is culturally contingent; and real deployment would need physiological and behavioral signals. Two concerns stand out: interaction is evaluated mostly by LLM judges on a 1-to-5 scale plus a 100-sample human blind test, a small sample for a subjective metric like empathy that LLM judges can bias; and intent recognition at 56.46 shows the understanding domain is far less mature than perception, so this unification is for now mostly a perception-layer win.