Factorization-Based Information Reconstruction for Enhancing Missing Modality Robustness

In recent years, Multimodal Sentiment Analysis (MSA) has emerged as a prominent research area, utilizing multiple signals to better understand human sentiment. Previous studies in MSA have primarily concentrated on performing interaction and fusion with complete signals. However, they have overlooked the issue of missing signals, which commonly arise in real-world scenarios due to factors such as occlusion, privacy concerns, and device malfunctions, leading to reduced generalizability. To this end, we propose a Factorization-based Information Reconstruction Framework (FIRF) to mitigate the modality missing problem in the MSA task. Specifically, we propose a fine-grained complementary factorization module that factorizes modality into synergistic, modality-heterogeneous, and noisy representations and design elaborate constraint paradigms for representation learning. Furthermore, we design a distribution calibration self-distillation module that fully recovers the missing semantics by utilizing bidirectional knowledge transfer. Comprehensive experiments on two datasets indicate that FIRF has a significant performance advantage over previous methods with uncertain missing modalities.

I. Introduction

Multimodal Sentiment Analysis (MSA) has garnered significant attention in recent years. Unlike traditional emotion recognition approaches that rely solely on a single signal [1], MSA combines multiple signals to achieve a more comprehensive understanding of human emotions [2]. Research has shown that leveraging complementary semantic information from different signals improves the generation of accurate multimodal representations [3]. While substantial progress has been made in MSA under the assumption that all modalities are fully available during both training and inference [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], this ideal scenario rarely holds true in real-world applications. Modalities are frequently missing due to challenges such as security constraints, environmental noise, or hardware limitations. Such incomplete multimodal data significantly degrades the performance of MSA systems.