Countering Multi-modal Representation Collapse through Rank-targeted Fusion
Seulgi Kim, Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib · 2025
Read Countering Multi-modal Representation Collapse through Rank-targeted Fusion by Seulgi Kim, Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06450.txt.
Introduction
Seulgi Kim and co-authors stop sensor fusion from collapsing into a single dull signal. When data streams merge, feature collapse drains the variety from individual dimensions, and modality collapse lets one loud sensor drown the rest. Human action anticipation, which must fuse many sensors, suffers from both failures at once.
Effective rank, a measure of how richly a representation spreads across dimensions, becomes the shared yardstick for both problems. Their Rank-enhancing Token Fuser blends weak features from one modality with complementary features from another, provably raising the fused rank. Depth paired with ordinary video kept balance instead of collapsing, and tests on three action datasets beat prior methods.
Catalog record
| CREATOR | Seulgi Kim, Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06450
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Countering Multi-modal Representation Collapse through Rank-targeted Fusion Seulgi Kim, Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib Abstract Multi-modal fusion methods often suffer from two types of representation collapse: feature collapse where individual dimensions lose their discriminative power (as measured by eigenspectra), and modality collapse where one dominant modality overwhelms the other. Applications like human action anticipation that require fusing multifarious sensor data are hindered by both feature and modality collapse. However, existing methods attempt to counter feature collapse and modality collapse separately. This is because there is no unifying framework that efficiently addresses feature and modality collapse in conjunction. In this paper, we posit the utility of effective rank as an informative measure that can be utilized to quantify and counter both the representation collapses. We propose Rank-enhancing Token Fuser , a theoretically grounded fusion framework that selectively blends less informative features from one modality with complementary features from another modality. We show that our method increases the effective rank of the fused representation. To address modality collapse, we evaluate modality combinations that mutually increase each others’ effective rank. We show that depth maintains representational balance when fused with RGB, avoiding modality collapse. We validate our method on action anticipation, where we present R3D , a depth-informed fusion framework. Extensive experiments on NTURGBD, UTKinect, and DARai demonstrate that our approach significantly outperforms prior state-of-the-art methods by up to 3.74%. Our code is available at: https://github.com/olivesgatech/R3D . Citation S. Kim, K. Kokilepersaud, M. Prabhushankar, G. AlRegib, “Countering Multi-modal Representation Collapse through Rank-targeted Fusion,” in 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, Arizona, USA, 2026. Review Date of Acceptance: Nov. 9th 2025 Codes https://github.com/olivesgatech/R3D Bib @inproceedings{kim2025countering, title={Countering Multi-modal Representation Collapse through Rank-targeted Fusion}, author={Kim, Seulgi and Kokilepersaud, Kiran and Prabhushankar, Mohit and AlRegib, Ghassan}, booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision}, year={2026}} Contact {seulgi.kim, kpk6, mohit.p, alregib}@gatech.edu https://alregib.ece.gatech.edu Figure 1 : This is a toy figure describing feature and modality collapse using spectral decomposition. (e): The ideal fused representation preserves complementary eigenvectors from both data modalities. (f): In contrast, feature collapse occurs when the fused representation varies along a subset of eigenvectors. (g): Modality collapse occurs when one modality dominates and suppresses the contribution of the eigenvectors of the other modality. 1 Introduction Modern data collection paradigms [ 33 ] utilize multifarious sensor modalities to represent an environment, as each modality captures unique aspects of a scene. For example, RGB excels at conveying visual cues in objects such as color or texture, while depth captures geometric structures and directional relationships between objects (Figure 1 (a-b)). Hence, multi-modal learning [ 70 ] is indispensable for human-centric tasks such as action anticipation [ 36 , 76 , 7 , 25 , 11 , 41 ] , video understanding [ 28 , 32 ] , and action recognition [ 67 ] , where single modalities fall short in expressing subtle semantic variations within or across scenes. Recent multi-modal fusion methods have achieved empirical success through contrastive losses [ 54 , 21 ] , attention blocks [ 69 ] , or various specialized objectives [ 18 ] . However, they often rely on learning representations via indirect pre-text tasks or alignment losses [ 48 ] , rather than directly targeting the informative content in each modality. This lack of targeted information fusion makes multi-modal fusion models susceptible to the common pitfalls: feature collapse [ 44 ] and modality collapse [ 12 ] . A central technical challenge we tackle in this paper is the lack of a unifying information framework that targets both feature and modality collapse. We illustrate targeted information fusion in Figure 1 . In (c) and (d), RGB and Depth are projected into their respective representation spaces using their corresponding encoders. Each space is visualized with a bar plot of eigenvalues associated with eigenvectors [u_{1},...,u_{6}] and a toy 3D spherical projection of the data points. Intuitively, the eigenvectors of each space represent the principal directions of variation in modality-specific representations. For the purposes of discussion, we define each eigenvector as a combination of projections from individual features (channels). Under this view, a channel is considered informative if it contributes significantly to key eigenvector directions, indicating strong alignment with the principal variations in the space. Thus, we interpret eigenvectors in Figure 1 as information components , that is, the feature directions encoding useful content from each modality. An ideal representation space after fusion should reflect Figure 1 (e), where each information component is equally represented, resulting in a space that integrates feature directions encoding useful content from both modalities. However, in reality, different types of collapse can occur in the fused representation space. In Figure 1 (f), we visualize feature collapse where certain information components become limited in their contribution to the overall space after fusion. This limits the overall feature diversity and the generalizability of the model [ 40 , 10 ] . In Figure 1 (g), we observe modality collapse [ 12 ] where information components are expressed for one modality over the other, limiting the sharing of complementary features between the modalities. To address these challenges, we propose an ideal fusion strategy for action anticipation that addresses both representation collapse. First, to counter feature collapse, we aim to increase the effective rank [ 57 ] of the fused representation matrix, as it serves as a proxy for the diversity of information. This is motivated by prior work defining the effective rank as matrix entropy [ 62 , 55 , 74 ] . Based on this, we develop Rank-enhancing Token fuser , a theoretically grounded fusion framework that selectively blends less informative channels (i.e., channels that contribute little to the principal eigenvector directions) with complementary features from another modality. Second, we posit that compatible modalities mutually increase each other’s effective rank. Hence, to address modality collapse, we perform an analysis of different modalities to pair with RGB for action anticipation. We demonstrate through a relative rank analysis that the depth modality results in the most balanced feature space between both modalities. We use this intuition to develop R3D (Rank-enhancing fusion in 3D), the first depth-informed action anticipation architecture. We demonstrate performance improvements across a wide variety of benchmarks. Our main contributions are summarized as follows: 1. Rank-enhancing Token Fuser : We are the first to formulate multi-modal fusion as a problem of rank-targeted fusion to simultaneously address feature collapse and modality collapse. We provide theoretical conditions under which selective channel blending provably increases effective rank and correspondingly prevents representation collapse. 2. Depth-aware 3D Action Anticipation : We present R3D , the first depth-informed framework for 3D action anticipation and show that depth is the most complementary modality to pair with RGB for preserving modality-specific features. 3. State-of-the-art performance : R3D achieves up to 3.74% performance improvement on NTURGBD, UTKinect, and DARai, setting a new benchmark for multimodal action anticipation. 2 Related Works 2.1 Action Anticipation The availability of large-scale video datasets [ 14 , 24 , 42 , 46 , 39 ] has spurred significant progress in addressing the challenge of action anticipation. These methods can be broadly categorized into two types. First, short-term action anticipation focuses on predicting a single future action that will occur within a