Journal Article

A Three-stage multimodal emotion recognition network based on text low-rank fusion

Yang, Youlong, Ning, Tong, Zhao, Linlin

2024

Overview

Multimodal emotion recognition has achieved good results in emotion recognition tasks by fusing multimodal information such as audio, text, and visual. How to use multimodal interaction and fusion to transform sparse unimodal into compact multimodal has become a vital research hotspot in multimodal emotion recognition. However, in multimodality, the extracted unimodal information needs to be representative. The multimodal fusion will cause the loss of feature information, which creates a particular challenge for multimodal emotion recognition. To address these problems, this paper proposes a three-stage multimodal emotion recognition network based on text low-rank fusion by extracting unimodal features, combining bimodal features, and fusing multimodal features. Specifically, we introduce a Residual-based Attention Mechanism for the first feature extraction stage, which can filter out redundant information and extract valuable unimodal information. Then, we use the Cross-modal Transformer to complete the inter-modal interaction. Finally, we introduce a Text-based Low-rank Fusion Module that enhances multimodal fusion by leveraging the complementarity between different modalities, ensuring comprehensive fused features. The accuracy of the proposed model on CMU-MOSEI, CMU-MOSI, and IEMOCAP datasets is 82.1%, 80.8%, and 83.0%, respectively. Meanwhile, many ablation experiments are conducted in this paper to verify the effectiveness and generalization of the model.

• Publisher: Springer Berlin Heidelberg,Springer Nature B.V
• Subject: Ablation / Audio data / Computer Communication Networks / Computer Graphics / Computer Science / Cryptology / Data Storage Representation / Datasets / Deep learning / Emotion recognition / Emotions / Feature extraction / Motion capture / Multimedia Information Systems / Neural networks / Operating Systems / Regular Paper