Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process

Tsai Hor Chan, Feng Wu, Yihang Chen, Guosheng Yin, Lequan Yu · 2025

Read Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process by Tsai Hor Chan, Feng Wu, Yihang Chen, Guosheng Yin, Lequan Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20736.txt.

Introduction

Tsai Hor Chan, Feng Wu, Yihang Chen, Guosheng Yin, and Lequan Yu tackle fusion in multimodal learning. A system must join inputs such as scans, text, and numbers without flattening what makes each input distinct. Past methods stress aligning the inputs and can over smooth the strong features inside each one.

Their framework borrows from the Dirichlet process, a Bayesian method whose rich get richer rule gives growing weight to the most prominent parts. Each input is modeled as a blend of bell shaped groups whose weights the process sets on the fly. The design seeks an automatic balance between lifting clear in mode features and aligning across modes, tested in health and finance style settings.

Catalog record

CREATORTsai Hor Chan, Feng Wu, Yihang Chen, Guosheng Yin, Lequan Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20736

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process Tsai Hor Chan, Feng Wu, Yihang Chen, Guosheng Yin, Lequan Yu Abstract Developing effective multimodal fusion approaches has become increasingly essential in many real-world scenarios, such as health care and finance. The key challenge is how to preserve the feature expressiveness in each modality while learning cross-modal interactions. Previous approaches primarily focus on the cross-modal alignment, while over-emphasis on the alignment of marginal distributions of modalities may impose excess regularization and obstruct meaningful representations within each modality. The Dirichlet process (DP) mixture model is a powerful Bayesian non-parametric method that can amplify the most prominent features by its richer-gets-richer property, which allocates increasing weights to them. Inspired by this unique characteristic of DP, we propose a new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment. Specifically, we assume that each modality follows a mixture of multivariate Gaussian distributions and further adopt DP to calculate the mixture weights for all the components. This paradigm allows DP to dynamically allocate the contributions of features and select the most prominent ones, leveraging its richer-gets-richer property, thus facilitating multimodal feature fusion. Extensive experiments on several multimodal datasets demonstrate the superior performance of our model over other competitors. Ablation analysis further validates the effectiveness of DP in aligning modality distributions and its robustness to changes in key hyperparameters. Code is anonymously available at https://github.com/HKU-MedAI/DPMM.git 1 Introduction Multimodal learning aims to aggregate information from multiple modalities to generate meaningful representations for downstream tasks. It has been widely explored in the context of vision-language models ( fu2023multimodal ; el2023visionlanguage ) , audio-visual applications ( chen2023iquery ; mo2023audiovisual ; huang2023egocentric ) , image-video models ( girdhar2023omnimae ; gan2023cnvid ) and healthcare applications ( wu2024MUSE ; hayat2022medfuse ) . For example, multimodal learning has been applied to various healthcare tasks such as clinical outcome prediction ( zhang2023improving ; wu2024MUSE ) , report generation ( song2022cross ; cao2023mmtn ) , and clinical trial site selection ( theodorou2024framm ) . One of the main tasks in multimodal learning is to fuse information from different modalities, which can be divided into early, joint, or late fusion strategies ( huang2020fusionSurvey ) . The joint fusion strategy is the most popular multimodal fusion paradigm ( huang2020fusionSurvey ) due to its strong ability to capture structural inter-modality interactions. It aims to align the distributions of different modality representations, creating a unified representation that effectively captures these interactions and preserves meaningful cross-modal information for downstream prediction tasks ( hayat2022medfuse ) . Figure 1 : Dirichlet process based modality alignment (top right) vs. conventional modality alignment (bottom right). The Dirichlet process allocates more weights to capture the prominent features (e.g., \mathbin{\scalebox{0.72}{$\bigstar$}},\triangle ) while conventional methods inevitably shrink the intra-modal features during alignment. However, due to the inherent heterogeneity of different modalities (e.g., medical images, medical reports, electronic health records), properly aligning the distributions of various modalities for fusion remains a challenging problem. Despite the success of existing alignment strategies, such as contrastive learning wang2022multi and adversarial alignment linsley2023adversarialAlign , most of them primarily focus on aligning features while neglecting the important intra-modality representations within each modality. As a result, this leads to weaker fused representations, ultimately limiting the performance of downstream tasks and reducing the generalizability and robustness of multimodality models. Consequently, there is a strong need for a novel approach that can not only effectively align the feature distributions but also preserve the important intra-modal representations. Bayesian nonparametric methods have shown great success in representation learning due to their flexibility and capability in density estimation. Specifically, the Dirichlet process (DP) has been frequently adopted since it can highlight the most significant features in the covariates with its richer-gets-richer property zhang2022DPMRMST . Figure 1 presents an illustrative example of how DP can amplify the feature signals for each modality. Despite its capability, the potential of DP in tackling multimodal representation learning is still under-explored. Moreover, the issue of unpaired observations (i.e., missing observations on some modalities) presents a significant challenge in multimodal learning. Most existing methods operate under the assumption that all modalities are available for every observation. However, in real-world scenarios, it is common for certain modalities, such as medical images or reports, to be absent for some patients due to clinical or administrative reasons in health care yao2024drfuse . The existing solutions either discard these observations or impute simple values hayat2022medfuse (e.g., zeros) to address the missing modality problem. However, the feature signals in each modality after learning the alignment objectives are shrunk, leading to weaker feature representations and worse imputation performance. Therefore, it is necessary to amplify the intra-modal representations to generate representative features for the observations with unpaired observations. In light of the above challenges, we propose a novel Dirichlet process-driven multimodal learning framework, namely DPMM ( D irichlet P rocess M ixture M ultimodal Learning), to tackle the multimodal fusion paradigm from a probabilistic perspective. We formulate the multimodal feature distribution as a Gaussian mixture model whose weights are allocated by DP, such that prominent features can receive increasing attention. Our contributions can be summarized as: (1) We introduce a novel multimodal learning framework via the Dirichlet process to amplify the feature signals while properly aligning them across modalities. We treat the joint distribution as a DP mixture model to allow DP to select the most useful features using its richer-gets-richer property. (2) We adopt stochastic variational inference to efficiently optimize the proposed Dirichlet process model, which overcomes the scalability limitations of traditional optimization algorithms based on Markov chain Monte Carlo (MCMC) and enables the scalability of our model to large-scale datasets. (3) We utilize the learned marginal distributions as the representation generator to accurately impute the missing observations with the prominent features highlighted by DP. This enhances the features supplied to training and downstream tasks and hence leads to improved performance. (4) Empirical results on four multimodal datasets demonstrate the superior performance of our framework and ablation analysis further corroborates the effectiveness of DP in modality alignment and feature amplification. 2 Related Works Cross-modal Alignment. Cross-modal alignment aims to make representations of different modalities comparable and aligned in a shared latent space hubert2017learning ; xian2016latent , and it has been widely applied in various multimodal tasks, such as image-text retrieval song2022cross , image captioning zhao2020cross , and multimodal classification fang2022learning . Some existing works focus on learning a shared representation using fusion strategies, such as concatenation hayat2022medfuse ; trong2020late or transformer-based mechanisms zhang2023improving ; theodorou2024framm without alignment. Despite being effective for certain tasks, these methods may yield suboptimal results when dealing with highly heterogeneous modalities yao2024drfuse . To address this issue, some works use contrastive learning to align representations by maximizing the similarity between different modalities wang2022multi ; wang2023cross . Other studies use variational inference methods theodoridis2020cross ; zhao2024conditional ; fang2022learning , such as variational autoencoders (VAEs), to align representations by minimizing the Kullback–Leibler (KL) divergence between latent distributions of different modalities. However, these methods often emphasize on the shared information between modalities, which may compromise the preservation of crucial modality-specific features that are necessary for downstream tasks. Figure 2 : Overview of our proposed DPMM framework. We first input the

Read the complete text