Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders

Rogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer · 2025

Read Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders by Rogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01134.txt.

Introduction

Rogelio Mancisidor and co-authors improve how a learning model joins facts from several kinds of input, such as image and text, into one shared picture. Common methods merge the parts as if they did not affect each other, which the authors call too hopeful a guess.

Their method lets the expert view from each part depend on the others, and it learns how much each subset of inputs should count. The new model strikes a better balance between outputs that hang together and outputs of high quality. The gap to single input models shrinks as more kinds of input are added, and test scores for sorting stay strong.

Catalog record

CREATORRogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders Rogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer Abstract Multimodal learning with variational autoencoders (VAEs) requires estimating joint distributions to evaluate the evidence lower bound (ELBO). Current methods, the product and mixture of experts, aggregate single-modality distributions assuming independence for simplicity, which is an overoptimistic assumption. This research introduces a novel methodology for aggregating single-modality distributions by exploiting the principle of consensus of dependent experts (CoDE), which circumvents the aforementioned assumption. Utilizing the CoDE method, we propose a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities. The resulting CoDE-VAE model demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estimations. CoDE-VAE further minimizes the generative quality gap as the number of modalities increases. In certain cases, it reaches a generative quality similar to that of unimodal VAEs, which is a desirable property that is lacking in most current methods. Finally, the classification accuracy achieved by CoDE-VAE is comparable to that of state-of-the-art multimodal VAE models. CoDE-VAE is available at: https://github.com/rogelioamancisidor/codevae . 1 Introduction Motivated by the observation that learned representations from multimodal data tend to be more generalizable ( Wu & Goodman 2018 ; Wu & Goodman 2019 ) , variational autoencoders (VAEs) ( Kingma & Welling 2014 ; Rezende et al. 2014 ) have been used to learn representations from multiple data modalities as they are capable of simultaneously generating new observations and inferring joint representations. However, during model inference, it is not guaranteed that we will have access to all modalities, as multimodal data is expensive to obtain ( Wu & Goodman 2018 ; Wu & Goodman 2019 ; Sutter et al. 2021 ) . Therefore, multimodal VAEs should be able to generate representations even when some modalities are missing. This fact has led to a line of research on scalable multimodal generative models with VAEs ( Wu & Goodman 2018 ; Shi et al. 2019 ; Sutter et al. 2021 ; Hwang et al. 2021 ; Palumbo et al. 2023 ) in which the number of single-modality distributions, called experts , is linear to the number of modalities M . The product of experts (PoE) ( Hinton 2002 ) and mixture of experts (MoE) are currently the predominant methods to estimate joint variational posterior distributions in multimodal VAEs. However, both assume independence between the experts for simplicity. This is an overoptimistic assumption, as the data modalities are simply different sources of information on the same object. Besides, both have flaws in the estimation of the joint posterior. PoE can produce posterior distributions with lower density if any of the experts also has low density, and if the precision of the experts is miscalibrated, the results can be biased ( Shi et al. 2019 ) . The weights in MoE, on the other hand, can be seen as probabilities that the distribution is correct. However, the notion of a “correct model” is not reasonable in this context ( Winkler 1981 ) , as the objective is to aggregate estimates from expert distributions. Furthermore, MoE is inefficient in high-dimensional spaces, as the joint posterior cannot be sharper than any of the experts ( Hinton 2002 ) . In addition to the aforementioned limitations in estimating the joint variational posterior distributions, the optimization of multimodal VAEs requires a valid evidence lower bound (ELBO). In the past, this has been obtained by summing the ELBOs corresponding to each of the k -th subsets {\mathbb{X}}_{k}\in{\mathcal{P}(\mathbb{X})} ( Sutter et al. 2021 ) , where {\mathbb{X}} is the multimodal data, {\mathcal{P}(\mathbb{X})} is its powerset, and where each k -th ELBO contributes equally to optimization. We argue that not all subsets provide the same information and, therefore, should not contribute equally to optimization. Methods for aggregation of expert distributions, such as PoE or MoE, are called consensus of experts ( Winkler 1981 ) . Although the information provided by expert distributions can exhibit dependence, limited attention has been paid to it. Therefore, this research advances the method introduced in ( Winkler 1981 ) by extending it to multivariate data and applying it to the approximation of joint variational distributions in multimodal VAEs, which is a non-trivial task. Our proposed consensus of dependent experts (CoDE) method expresses the dependence between experts through the experts’ error of estimation and estimates the joint variational distributions with a principled Bayesian method, which circumvents the notion of a correct model and the approximation of vaguer posterior distributions than any of the experts. Furthermore, when the expert distributions are miscalibrated, or uncertain, the joint distribution estimated by CoDE leans towards expert distributions that are more certain. Based on CoDE, we introduce a novel multimodal VAE model (CoDE-VAE) that not only takes into account the dependence between expert distributions when estimating joint posterior variational distributions, but also learns the contribution of each k -th ELBO term in optimizing the overall ELBO. An important distinction in estimating joint posterior distributions and optimizing the CoDE-VAE model is that it does not rely on sub-sampling 1 1 1 Sub-sampling in this research refers only to the definition of consensus distributions as a mixture model and the training of some, but not all, \mathcal{L}_{k} terms (defined in Section 3 ). ( Daunhawer et al. 2022 ) , which has been shown to harm the approximation of the joint distribution of the data and yet is used by some multimodal VAEs. The empirical analysis in this research shows that, when dependence between experts is considered, CoDE-VAE exhibits better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estimations. Furthermore, CoDE-VAE minimizes the generative quality gap as the number of modalities increases, achieving unconditional FID scores similar to unimodal VAEs, which is a desirable property that is lacking in most current models. Finally, CoDE-VAE achieves a classification accuracy that is comparable to that of current state-of-the-art multimodal VAEs. The contributions of this research are: (1) the development of a novel consensus of experts method that takes into account the stochastic dependence between experts; (2) the use of this novel consensus of experts method to derive a new ELBO that learns the contribution of each subset of modalities to optimization ; (3) to show that the generative coherence and generative quality of generated modalities, as well as likelihood approximations in multimodal VAEs, are higher when the dependence between experts is considered and the contribution of ELBO terms to optimization is learned. 2 Related Work The initial research on multimodal learning with VAEs focused on bimodal data, e.g., ( Suzuki et al. 2017 ; Vedantam et al. 2018 ) where the consensus distribution is approximated by concatenating the two modalities or ( Wang et al. 2017 ) where the authors simply approximate the two expert distributions, assuming that one modality is available during training and test time. Since then, there have been different lines of research to improve performance in multimodal VAEs. For example, the information on the class labels has been used to improve the performance in discriminative tasks ( Tsai et al.

Read the complete text