Robust Multi-modal Task-oriented Communications with Redundancy-aware Representations

Jingwen Fu, Ming Xiao, Zhonghao Lyu, Mikael Skoglund, Celimuge Wu · 2025

Read Robust Multi-modal Task-oriented Communications with Redundancy-aware Representations by Jingwen Fu, Ming Xiao, Zhonghao Lyu, Mikael Skoglund, Celimuge Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08642.txt.

Introduction

Jingwen Fu and four co-authors design a communication system that sends only what a task needs from text, audio, and video traveling together. Bandwidth is tight and channels are noisy, so the system must squeeze out overlap between the streams without losing the details the receiving task depends on.

A two-stage variational information bottleneck does the squeezing. Each stream is first compressed alone, keeping its task-specific features, while an adversarial module trims cross-stream dependence so the streams complement rather than repeat each other. A second stage compresses the fused result and hardens it against channel distortion. On multi-modal emotion recognition tests the framework beats existing baselines on accuracy and reliability. The gains grow largest when the signal-to-noise ratio is low.

Catalog record

CREATORJingwen Fu, Ming Xiao, Zhonghao Lyu, Mikael Skoglund, Celimuge Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Multi-modal Task-oriented Communications with Redundancy-aware Representations Jingwen Fu, Ming Xiao, Zhonghao Lyu, Mikael Skoglund, Celimuge Wu Abstract Semantic communications for multi-modal data can transmit task-relevant information efficiently over noisy and bandwidth-limited channels. However, a key challenge is to simultaneously compress inter-modal redundancy and improve semantic reliability under channel distortion. To address the challenge, we propose a robust and efficient multi-modal task-oriented communication framework that integrates a two-stage variational information bottleneck (VIB) with mutual information (MI) redundancy minimization. In the first stage, we apply uni-modal VIB to compress each modality separately, i.e., text, audio, and video, while preserving task-specific features. To enhance efficiency, an MI minimization module with adversarial training is then used to suppress cross-modal dependencies and to promote complementarity rather than redundancy. In the second stage, a multi-modal VIB is further used to compress the fused representation and to enhance robustness against channel distortion. Experimental results on multi-modal emotion recognition tasks demonstrate that the proposed framework significantly outperforms existing baselines in accuracy and reliability, particularly under low signal-to-noise ratio regimes. Our work provides a principled framework that jointly optimizes modality-specific compression, inter-modal redundancy, and communication reliability. I Introduction The rapid advance of deep learning has precipitated a paradigm shift in the design of communication systems [ 17 ] . Conventional communication systems have been primarily concerned with reliable symbol transmission, i.e., a problem identified as the first level of communication by Shannon and Weaver [ 12 ] . However, the increasing prevalence of intelligent applications is driving a shift from this classical paradigm towards high-level communication objectives [ 13 ] . The first of these, semantic communication (SemCom), is used to accurately convey the intended meaning of the source data, rather than to ensure mere bit-level fidelity [ 17 ] . A further evolution is task-oriented communication (TOC), where the objective is to transmit the minimal information required for a recipient to successfully execute a specific task. For many emerging applications, such as autonomous driving and remote healthcare, information from a single modality is often insufficient to characterize complex real-world events. Consequently, multi-modal TOC or SemCom has emerged as a critical and rapidly advancing area of research [ 6 ] . To achieve efficient multi-modal TOC/SemCom, a few frameworks have been proposed. For instance, [ 18 ] has integrated visual and textual data for visual question-answering (VQA) communication systems. Similarly, unified frameworks to transmit shared modalities like images, text, and audio have been developed in [ 23 ] . In these systems, the information bottleneck (IB) is shown to be a powerful tool for balancing the trade-off between information compression and task performance [ 13 ] . Moreover, methods based on variational information bottleneck (VIB) [ 13 ] and robust information bottleneck (RIB) [ 19 ] have further provided a tractable upper bound for high-dimensional data compression, and coded redundancy reduction to improve transmission efficiency, respectively. These pioneering works have laid a solid foundation for extracting and transmitting task-relevant semantic information. Despite significant progress, existing multi-modal TOC frameworks face two critical challenges. First, although current models effectively compress intra-modality information, they often neglect the redundancy across different modalities. For example, in a video segment, visual modality and audio modality may convey the same positive sentiment. Transmitting both may waste channel bandwidth. Second, the fused multi-modal representations are typically vulnerable to channel impairments, such as noise and fading, which can significantly compromise semantic reliability and degrade task performance. These issues motivate a key research question: How can a multi-modal TOC framework be designed to effectively mitigate inter-modal redundancy and simultaneously enhance the robustness of the fused semantic features against adverse channel conditions such as noise and fading? To tackle these challenges, we propose a novel framework that integrates a two-stage VIB with mutual information (MI) redundancy minimization. The proposed method follows a hierarchical processing architecture. First, we apply a dedicated uni-modal VIB for each modality, i.e., text, audio, and video, as the first-stage bottleneck to extract modality-specific and task-relevant features. Then, after feature fusion, we introduce a novel cross-modal MI discriminator module, which adversarially suppresses inter-modal dependencies through discriminator training. The module encourages complementary, rather than redundant, information across modalities. Finally, the fused multi-modal representation is further compressed by the second-stage multi-modal VIB to enhance robustness against channel distortion. This end-to-end design, which explicitly incorporates the wireless channel into training, enhances robustness and preserves semantic integrity under channel distortion. The main contributions are summarized as follows. • We propose a novel two-stage VIB architecture for a multi-modal TOC system, which explicitly considers both modality-specific and fused multi-modal representations via VIB rate-distortion regularization. The proposed hierarchical architecture enables more efficient compression of task-relevant semantics and simultaneously improves robustness against channel distortion. • Our work introduces a cross-modal redundancy reduction module that minimizes MI among modality pairs. Specifically, we formulate pairwise MI among modality representations and further derive and prove a bounded variational lower bound on MI, ensuring both theoretical soundness and training stability. To stabilize the adversarial learning process, we incorporate a gradient reversal layer (GRL) that enables efficient and unified end-to-end training, where encoders are optimized to suppress redundant information while discriminators tighten the MI estimates. This mechanism effectively reduces redundancy among modalities, yielding complementary and compact multi-modal representations. • Extensive experiments on multi-modal tasks show that the proposed framework effectively reduces redundancy, improves transmission robustness under varying channel conditions, and consistently outperforms benchmark in both accuracy and robustness. The remainder of this paper is organized as follows. Section II reviews related work. Section III presents our system model and problem formulation. Sections IV and V detail the proposed scheme. Section VI presents the experimental setups and results. Finally, Section VII concludes the paper. Notations: In what follows, upper-case letters (e.g. X ) stand for random variables, and lower-case letters (e.g. x ) are their realizations. The Kullback-Leibler (KL) divergence between two probability distributions p(x) and q(x) is denoted as D_{\text{KL}}(p\|q) . The Jensen-Shannon (JS) divergence is denoted as D_{\text{JS}}(p\|q) . The statistical expectation of X is denoted as \mathbb{E}(X) . II Related Works II-A Semantic and Task-oriented Communications The classic communication theory of Shannon defines three levels of transmission targets, i.e., technical, semantic, and effectiveness [ 12 ] . Traditional systems focused on the technical level (bit-accuracy). Recent advances in deep learning have prompted a paradigm shift towards semantic transmission [ 17 ] . SemCom aims to transmit the meaning of data rather than the raw data, as exemplified by the DeepSC system that uses neural encoders to extract textual semantics with improved efficiency [ 17 ] . Building on the principle, TOC further refines the goal to complete a specific task, pruning task-irrelevant information as redundancy [ 4 , 10 ] . As intelligent applications increasingly integrate heterogeneous data sources, multi-modal SemCom/TOC has become an important research area. Researchers have developed multi-modal SemCom systems for diverse scenarios. For example, the authors in [ 18 ] have combined images and text in a VQA SemCom system for multi-user settings. Reference [ 23 ] proposed a unified end-to-end framework to jointly handle image, text, and audio. Moreover, the authors in [ 6 ] have designed distributed architectures for multi-modal semantic relays at network edges. These efforts underscore the growing

Read the complete text