SKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis

Abstract

Multimodal sentiment analysis is an active research field that aims to recognize the user’s sentiment information from multimodal data. The primary challenge in this field is to develop a high-quality fusion framework that effectively addresses the heterogeneity among different modalities. However, prior research has primarily concentrated on intermodal interactions while neglecting the semantic sentiment information conveyed by words in the text modality. In this paper, we propose the Sentiment Knowledge Enhanced Attention Fusion Network (SKEAFN), a novel end-to-end fusion network that enhances multimodal fusion by incorporating additional sentiment knowledge representations from an external knowledge base. Firstly, we construct an external knowledge enhancement module to acquire additional representations for the text modality. Then, we design a text-guided interaction module that facilitates the interaction between text and the visual/acoustic modality. Finally, we propose a feature-wised attention fusion module that achieves multimodal fusion by dynamically adjusting the weights of the additional and each modality’s representations. We evaluate our method on three challenging multimodal sentiment analysis datasets (CMU-MOSI, CMU-MOSEI, and Twitter2019). The experiment results demonstrate that our model significantly outperforms the state-of-the-art models. The source code is publicly available at https://github.com/doubibobo/SKEAFN.