FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis

Ziyang Liu Pengjunfei Chu Shuming Dong Chen Zhang Mingcheng Li Jin Wang · 2025

Read FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis by Ziyang Liu Pengjunfei Chu Shuming Dong Chen Zhang Mingcheng Li Jin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16086.txt.

Introduction

Ziyang Liu and five co-authors recover missing senses in multimodal sentiment analysis, where text, voice, and face should agree but often arrive incomplete. Occlusion, privacy limits, and device faults knock out whole modalities in real use, and most earlier methods assume complete input, so their generalisation suffers. Their framework, FSRF, rebuilds the missing semantics by factoring what each modality carries.

A de-redundant factorisation module splits each modality into shared, unique, and noisy parts, with tailored constraints guiding that split. A distribution-aligned self-distillation module then transfers knowledge both ways to restore the semantics of absent modalities. Across two data sets with uncertain missing patterns, FSRF holds a significant advantage over previous methods, the six authors report. The design targets sentiment systems that must keep working when one channel drops out entirely.

Catalog record

CREATORZiyang Liu Pengjunfei Chu Shuming Dong Chen Zhang Mingcheng Li Jin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16086

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis Ziyang Liu Pengjunfei Chu Shuming Dong Chen Zhang Mingcheng Li Jin Wang Abstract In recent years, Multimodal Sentiment Analysis (MSA) has become a research hotspot that aims to utilize multimodal data for human sentiment understanding. Previous MSA studies have mainly focused on performing interaction and fusion on complete multimodal data, ignoring the problem of missing modalities in real-world applications due to occlusion, personal privacy constraints, and device malfunctions, resulting in low generalizability. To this end, we propose a Factorization-guided Semantic Recovery Framework (FSRF) to mitigate the modality missing problem in the MSA task. Specifically, we propose a de-redundant homo-heterogeneous factorization module that factorizes modality into modality-homogeneous, modality-heterogeneous, and noisy representations and design elaborate constraint paradigms for representation learning. Furthermore, we design a distribution-aligned self-distillation module that fully recovers the missing semantics by utilizing bidirectional knowledge transfer. Comprehensive experiments on two datasets indicate that FSRF has a significant performance advantage over previous methods with uncertain missing modalities. 1 1 footnotetext: Corresponding author. I Introduction Multimodal Sentiment Analysis (MSA) has become a prominent focus in recent years. Unlike conventional emotion recognition tasks that rely on a single modality [ 1 ] , MSA integrates various modalities to enhance the understanding of human sentiment [ 2 ] . Prior research has demonstrated that integrating complementary semantics across multiple modalities enhances the creation of more precise multimodal representations [ 3 ] . MSA has been well studied so far under the assumption that all modalities are available in the training and inference phases [ 4 , 5 , 6 , 7 ] . However, in practical applications, modalities are often missing due to factors such as security issues, background noise, or sensor limitations. These incomplete multimodal datasets substantially impair the performance of MSA. In recent years, many studies [ 8 , 9 , 10 , 11 , 12 , 13 , 14 ] have focused on tackling the challenge of missing modalities in MSA, which can be classified into two distinct paradigms: (i) generative methods [ 9 , 8 ] and (ii) joint learning methods [ 10 , 14 , 12 ] . Generative methods focus on reconstructing missing features and semantics within modalities by utilizing the distributions of the accessible modalities. For example, GCNet [ 8 ] proposes two graph neural network-based modules to capture speaker and temporal dependencies in conversations to address the problem of missing modality in MSA. In contrast, joint learning methods aim to develop joint multimodal representations by exploiting correlations between modalities. For instance, CorrKD [ 14 ] introduces a correlation-decoupled knowledge distillation framework that leverages cross-sample, cross-category, and cross-response correlations to create robust joint multimodal representations for mitigating the modality missing problem in MSA. However, these approaches are limited by two factors: (1) Implementing complex inter-modality interactions based on incomplete modalities ignores the extraction of task-relevant semantics and task-independent noise. (2) The recovery paradigm for missing semantics is coarse-grained and static, without sufficient consideration of high-dimensional distributional alignment. To address the above problem, we propose a Factorization-guided Semantic Recovery Framework (FSRF) to solve the modality missing problem in MSA. Our strength comes from the following three core contributions: (i) We propose a de-redundant homo-heterogeneous factorization module that decomposes each modality into modality-homogeneous, modality-heterogeneous, and noise representations, and design elaborate constraint paradigms for representation learning. (ii) We propose a distribution alignment-based self-distillation module that utilizes the Sinkhorn distance and JS divergence to achieve bidirectional constraints between two networks. (iii) Extensive experiments conducted on two datasets demonstrate that FSRF has a significant performance advantage over previous methods with uncertain missing modalities and comparable performance with complete modalities. II Methodology II-A Problem Formulation Given a multimodal video dataset \mathcal{D}=\{\bm{X}_{i},\bm{Y}_{i}\}^{N}_{i=1} , where N is the number of samples. Each sample is containing three distinct modalities, denoted as \bm{S}=[\bm{M}_{L},\bm{M}_{A},\bm{M}_{V}] , where \bm{M}_{L}\in\mathbb{R}^{T_{L}\times d_{L}},\bm{M}_{A}\in\mathbb{R}^{T_{A}\times d_{A}} , and \bm{M}_{V}\in\mathbb{R}^{T_{V}\times d_{V}} denote language, audio, and visual modalities, respectively. \Phi=\{L,A,V\} denotes the set of modality types. T_{m}(\cdot) and d_{m}(\cdot) represent the sequence length and the embedding dimension, where m\in\Phi . To effectively replicate the uncertainty of modality missingness observed in real-world scenarios, we define two types of missing cases: (1) intra-modality missingness , which indicates some frame-level features in the modality sequences are missing. (2) inter-modality missingness , which indicates some modalities are entirely missing. The goal is to perform utterance-level sentiment recognition by utilizing multimodal data that includes missing modalities. Fig. 1 : The overall architecture of FSRF. We propose a De-redundant Homo-Heterogeneous Factorization (DHF) mechanism and a Distributional Alignment-based Self-Distillation (DAS) module to accurately recover missing information. II-B Modality Preprocessing The Modality Random Missing (MRM) strategy performs both intra-modality missing and inter-modality missing, generating two heterogeneous modality missing samples \bm{S}_{p} and \bm{S}_{q} from the input complete modality sample \bm{S} . Our framework receives \bm{S}_{p} and \bm{S}_{q} as input in parallel and symmetrically. For brevity, here we denote a particular sample of the input to the network with \bm{S}=[\bm{M}_{L},\bm{M}_{A},\bm{M}_{V}] . The language modality is fed into the pre-trained BERT [ 15 ] to obtain the average feature for each word, denoted as \bm{C}_{L}\in\mathbb{R}^{d_{L}} . The audio and visual modalities plus the positional encoding are then fed into the Transformer encoder [ 16 ] , whose last elements of the output are expressed as \bm{C}_{A}\in\mathbb{R}^{d_{A}} and \bm{C}_{V}\in\mathbb{R}^{d_{V}} . Each \bm{C}_{m} is passed through a fully-connected layer to unify the dimensions of the features to obtain \bm{H}_{m} . II-C De-redundant Homo-Heterogeneous Factorization Modality missing destroys the contextual semantics inherent in multimodal data, significantly impacting the effectiveness of multimodal fusion and semantic extraction. Previous studies of MSA with missing modalities [ 17 , 10 , 11 ] focus on designing complex cross-modal interaction paradigms to mine valuable information in incomplete modalities. These approaches treat incomplete modalities as the minimal interaction unit and ignore the shared, specific, and noisy information contained in the modalities, thus introducing significant cumulative errors. Therefore, we propose a De-redundant Homo-heterogeneous Factorization (DHF) module, to fully exploit the multilevel representations in modality. Specifically, the DHF factorizes each modality into three categories of representations: (1) Modality-homogeneous representation, denoting the representation reflecting the sentiment homogeneity shared among all modalities. (2) Modality-heterogeneous representation, which represents different modality-specific information for characterizing sentiment in different modalities. (3) Noise representation, which indicates the noise information contained in each modality. For each modality m\in\Phi , we obtain the modality-homogeneous representation by a modality-shared encoder, denoted as \bm{R}_{m}^{ho}=\mathcal{E}^{ho}(\bm{{H}_{m}}) , a modality-heterogeneous representation by a modality-specific encoder, denoted as \bm{R}_{m}^{he}=\mathcal{E}_{m}^{he}(\bm{{H}}_{m},\bm{R}_{m}^{ho}) , and a noise representation by a modality-specific encoder, denoted as \bm{R}_{m}^{n}=\mathcal{E}_{m}^{n}(\bm{{H}}_{m}) . All encoders are composed of multi-layer perceptrons with the ReLU activation. Modality-homogeneous representations should contain sentiment information shared among all modalities in the sample, and modality-heterogeneous representations of different modalities should be inconsistent. To achieve this objective, we propose a contrastive learning-based representation constraint mechanism, which employs NT-Xent loss [ 18 ] to constrain the distribution of different representations in the latent space. It has stronger constraints and training stability than traditional contrastive learning loss through normalization and temperature parameters. The loss is denoted as: \mathcal{L}_{DHF}^{ho-he}=-\frac{1}{6}\sum_{a\in\Phi}\,\sum_{b\in\Phi,a\neq b}log\frac{\mathcal{G}(\bm{R}^{ho}_{a},\bm{R}^{ho}_{b})}{\mathcal{G}{(\bm{R}^{ho}_{a},\bm{R}^{ho}_{b})}+\mathcal{G}(\bm{R}^{ho}_{a},\bm{R}^{he}_{a})}, (1) where \mathcal{G}(\bm{X},\bm{Y})=exp(-\mathcal{D}(\bm{X},\bm{Y})/\tau) , \mathcal{D}(\bm{X},\bm{Y}) is the cosine distance of \bm{X} and \bm{Y} , and \tau is the temperature hyper-parameter. Additionally, each modality has its unique

Read the complete text