A Closer Look at Multimodal Representation Collapse

Abhra Chaudhuri, Anjan Dutta, Tu Bui, Serban Georgescu · 2025

Read A Closer Look at Multimodal Representation Collapse by Abhra Chaudhuri, Anjan Dutta, Tu Bui, Serban Georgescu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22483.txt.

Introduction

Abhra Chaudhuri, Anjan Dutta, Tu Bui, and Serban Georgescu find why mixed tools drop a sense in a 2025 paper. A tool fed sight and sound should use both, yet oft it leans on one and lets the next waste.

The drop has a gear: loud junk in one sense twines, in the shared head, with the true marks of the next. The twine masks those marks, and the sense falls out of the call. A school where a full tool teaches a lean one can free the knot, the team proves. Their own fix deals the shared head's seats anew by rule. On mixed test sets the fix held both senses in play, and stood in when a sense went lost.

Catalog record

CREATORAbhra Chaudhuri, Anjan Dutta, Tu Bui, Serban Georgescu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22483

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Closer Look at Multimodal Representation Collapse Abhra Chaudhuri, Anjan Dutta, Tu Bui, Serban Georgescu Abstract We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/ . 1 Introduction Figure 1: When noisy features of one modality exist in entanglement with predictive features of another in the fusion head (the probability of which increases with the number of modalities), it results in a sub-optimal solution wherein the predictive value of the former modality is diminished by the inevitable existence of noisy features. Freeing up rank bottlenecks allows for the denoising of such features along independent dimensions without affecting the latter modality, while simultaneously allowing the predictive features of the former modality to contribute to loss reduction. Figure 2: Polysemanticity with and without feature interference between noisy and predictive features. All horizontal axes correspond to the value of the weight of the polysemantic neuron. For (b) and (c), the vertical axes correspond to the title of the columns. When the predictive features of modality 1 (M1) and the noisy features of modality 2 (M2) activate along the same region in the interpolation regime of the same neuron (a - top), it prevents the predictive features of M2 from contributing to loss minimization (c - top) because of the unintended inclusion of and the unwanted interference from the noisy features of M2 (b - top), leading to its collapse. However, when they are disentangled (here, by being mapped to disjoint sub-regimes in the weights-space around a switching threshold: a - bottom), they result in non-interfering activation patterns (b - bottom), and effectively, a feature-wise separable effect on the marginal loss (c - bottom). A number of recent works in the multimodal learning literature have observed that models that aim to learn a fusion of several modalities often end up relying only on a subset of them ( Javaloy et al., 2022 ; Wu et al., 2024 ) . This phenomenon, termed as modality collapse , has been empirically observed across a diverse range of fusion strategies ( Javaloy et al., 2022 ; Ma et al., 2022 ; Zhang et al., 2022 ; Zhou et al., 2023 ; Wu et al., 2024 ) , and has serious implications for the scenario when certain modalities can go missing at test time ( You et al., 2020 ; Ma et al., 2022 ; Wu et al., 2024 ) . If a model is reliant only on a subset of modalities and if it is those that specifically go missing at test time, the model could end up completely non-functional. Although there have been several attempts at mitigating modality collapse based on a priori conjectures about what might be causing them, such as conflicting gradients ( Javaloy et al., 2022 ) or interactions between data distributions and the fusion strategy ( Ma et al., 2022 ) , to the best of our knowledge, there have been no prior efforts towards developing a bottom-up understanding of the underlying learning-theoretic phenomena at play. We aim to bridge this gap by developing a mechanistic theory of multimodal feature encoding that is agnostic of the specific choice of fusion strategy (see Task Setup in Section 3 ). We start by showing that modality collapse arises as a result of an unintended entanglement among the noisy and the predictive features of different modalities through a shared set of polysemantic neurons ( Elhage et al., 2022 ) , which we observe, via Lemma 1 , to increase quadratically with the number of modalities. It implies that predictive features of some modalities cannot be learned without also including noisy features from the other modalities. The noisy features then effectively suppress the predictive value of the modality that they come from, leading to their observed collapse in the fused representation, a process we formalize in Theorem 1 . As depicted in Figure 1 , in the optimization landscape, collapse corresponds to a suboptimal solution such that any step around it along the dimension of entanglement (which is the only available dimension for optimization in the given state), would lead to a simultaneous denoising of one modality and the forgetting of predictive features from the other. If the latent factors underlying the modalities are sufficiently complementary, we show that cross-modal predictive–predictive feature entanglements are less likely to occur than predictive–noisy entanglements ( Lemma 3 ) – so, we mainly focus on the latter in this work. We find that this cross-modal entanglement of features happens due to faulty neural capacity allocation ( Scherlis et al., 2022 ) among modalities during the optimization of the fusion head (illustrated in Figure 2 ), which we observe, in Lemma 2 , to be a result of the well-known low-rank simplicity bias of neural networks ( Huh et al., 2023 ) limiting the rank of the gradient updates received at any given layer. Consequently, through Theorem 2 , we arrive at the result that this gradient-rank bottleneck forces SGD to parameterize the fusion head neurons in a polysemantic manner. Interestingly, we observe in Theorem 3 , that knowledge distillation into the modalities that collapse, from the ones that survive, implicitly averts cross-modal polysemantic entanglements. It does so by freeing up rank bottlenecks at the level of the student encoders. As a result, as again shown in Figure 1 , the noisy features of the modalities that would otherwise collapse, are allocated dedicated dimensions in the latent space, which the fusion operator can then leverage to denoise the output representations. This allows for the complete incorporation of predictive features from all the modalities without any noisy interference, thereby preventing collapse. Under the condition of identifiability ( Gulrajani & Hashimoto, 2022 ) of modality-specific causal factors up to equivariances of the underlying mechanisms ( Ahuja et al., 2022 ) , we propose an algorithm called Explicit Basis Reallocation (EBR), which automatically identifies cross-modal feature entanglements and learns independent denoising directions in the latent space to counteract their hindrance on empirical risk minimization. Consequently, the concrete feature-to-basis mapping across modalities obtained from EBR can be used to identify suitable substitution candidates for dealing with missing modalities at test-time. To summarize, we (i) provide a theoretical understanding of modality collapse based on polysemantic neurons in the fusion head leading to unwanted cross-modal entanglements, and the low-rank simplicity bias of neural networks; (ii) show that cross-modal knowledge distillation into the modalities undergoing collapse

Read the complete text