Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision

Jiali Cui [email protected], Zhiqiang Lao [email protected], Heather Yu [email protected] · 2026

Read Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision by Jiali Cui [email protected], Zhiqiang Lao [email protected], Heather Yu [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00644.txt.

Description

Jiali Cui, Zhiqiang Lao, and Heather Yu teach energy-based models to generate coherent multimodal data, such as matched images and text, by enlisting a variational auto-encoder as a partner. Energy-based models are flexible but hard to train: learning needs sampling by Markov chains in the joint data space, and chains started from noise mix poorly and miss the links between modalities. Multimodal auto-encoders capture those links through a shared latent generator, yet their simple Gaussian shapes limit realism.

The framework interweaves likelihood training of all three models with Markov chain refinements in data and latent spaces. The generator produces coherent samples that seed the energy model's chains, while the inference model supplies smart latent starting points for the generator's sampling. Each model thus rescues the other's sampling. Experiments show better synthesis quality and cross-modal coherence than various baselines.

Catalog record

CREATORJiali Cui [email protected], Zhiqiang Lao [email protected], Heather Yu [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00644

Browse the full library · About the Tower of Babel