Harmonizing Visual Representations for Unified Multimodal Understanding and Generation
Size Wu Wenwei Zhang Lumin Xu Sheng Jin Zhonghua Wu, Qingyi Tao Wentao Liu Wei Li Chen Change Loy · 2025
Read Harmonizing Visual Representations for Unified Multimodal Understanding and Generation by Size Wu Wenwei Zhang Lumin Xu Sheng Jin Zhonghua Wu, Qingyi Tao Wentao Liu Wei Li Chen Change Loy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21979.txt.
Introduction
One model that both reads images and makes them must hold two kinds of knowledge at once. Size Wu and co-authors build such a model, called Harmon, in a 2025 paper. Older unified models lean on codes that keep pixel detail but lose meaning, which hurts reading tasks.
Harmon starts from an encoder trained by hiding parts of images and asking for them back. That encoder proved sharp at naming the concepts in a picture. A three-stage training plan then grows both skills in one shared frame. The model hit top scores on image-making tests while matching rivals at image reading. Code and models are set for public release.
Catalog record
| CREATOR | Size Wu Wenwei Zhang Lumin Xu Sheng Jin Zhonghua Wu, Qingyi Tao Wentao Liu Wei Li Chen Change Loy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21979
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Harmonizing Visual Representations for Unified Multimodal Understanding and Generation Size Wu Wenwei Zhang Lumin Xu Sheng Jin Zhonghua Wu, Qingyi Tao Wentao Liu Wei Li Chen Change Loy Abstract Unifying visual understanding and generation within a single multimodal framework remains a significant challenge, as the two inherently heterogeneous tasks require representations at different levels of granularity. Current approaches that utilize vector quantization (VQ) or variational autoencoders (VAE) for unified visual representation prioritize intrinsic imagery features over semantics, compromising understanding performance. In this work, we take inspiration from masked image modelling (MIM) that learns rich semantics via a mask-and-reconstruct pre-training and its successful extension to masked autoregressive (MAR) image generation. A preliminary study on the MAR encoder’s representation reveals exceptional linear probing accuracy and precise feature response to visual concepts, which indicates MAR’s potential for visual understanding tasks beyond its original generation role. Based on these insights, we present Harmon , a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder. Through a three-stage training procedure that progressively optimizes understanding and generation capabilities, Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks while matching the performance of methods with dedicated semantic encoders (e.g., Janus) on image understanding benchmarks. Our code and models will be available at https://github.com/wusize/Harmon . Figure 1 : An overview of image generation and understanding examples. All results are obtained by the proposed Harmon-1.5B, which uses a shared visual encoder for both tasks. 1 Introduction “ What I cannot create, I do not understand. ” — Richard Feynman Generating images from textual descriptions and understanding images through targeted questions represent the forefront of AI research. With significant progress in both tasks, a plethora of text-to-image generation models [ 71 , 9 , 64 , 100 , 11 , 69 , 70 , 3 ] and multimodal understanding LLMs [ 99 , 53 , 52 , 44 , 95 , 21 , 92 , 17 , 16 , 88 ] have been developed, driven by advancements in model architectures and computation scaling. To push the envelope even further, unifying these two tasks into a single and cohesive framework that handles both understanding and generation is highly desirable for next-generation multimodal intelligence. Early attempts [ 77 , 78 , 81 , 84 ] have combined state-of-the-art diffusion models and multimodal LLMs by prompting the diffusion generators with LLM embeddings. These approaches are constrained by weak integration between image generation and text sequence modeling, resulting in suboptimal performance in instruction-based generation [ 32 ] . More recently, a tighter coupling of generation and understanding has been achieved through either next-token prediction approaches [ 54 , 79 , 89 , 86 ] or denoising diffusion frameworks [ 98 , 91 , 97 , 90 ] , where visual representations for both tasks are jointly modeled. Figure 2 : Linear probing and feature visualization (using GradCAM++ [ 8 ] ) provide crucial insights into the suitability of representations learned by VQGAN, VAE, and MAR models for serving as the visual encoder in a unified multimodal framework. Visual understanding and generation are inherently heterogeneous tasks that require representations at different levels of granularity. In particular, visual understanding demands coarse-grained, high-level representations, whilst generation prefers fine-grained, intrinsic imagery features. Consequently, following standard practices in visual generation [ 71 , 76 ] , most approaches employ either a vector quantization (VQ) model or a variational autoencoder (VAE) to compress visual inputs into discrete tokens or continuous latents. These encoders are primarily pre-trained to preserve intrinsic image features for pixel-level reconstruction rather than capturing visual semantics, resulting in limited image understanding capabilities. Therefore, to obtain a harmonious representation with shared visual encoding for unified multimodal understanding and generation, a more integrated and moderate approach is warranted. We take inspiration from masked image modeling (MIM) [ 34 , 2 ] that can effectively encourage the model to develop a richer semantic representation of visual data via a simple mask-and-reconstruct pretext task. This learning by generating philosophy can learn balanced representations capable of both generation and understanding, making it a promising approach to reconcile these heterogeneous tasks in shared encoding. Moreover, the recent success of masked autoregressive (MAR) [ 46 , 28 ] modeling that extends MIM to autoregressive image generation further substantiates the potential of MIM-based paradigms in this realm. It aligns with the de facto autoregressive prediction paradigm in MLLMs, making it more suitable for integrated synergy between generation and understanding. Driven by this revelation, we conduct preliminary studies involving linear probing and feature visualization to measure the representation power of widely used VQGAN and VAE encoders in generative models and the MAR encoder, which is shown in Figure 2 . Our findings reveal that VQGAN and VAE encoders struggle to capture high-level semantics, such as object concepts and attributes, which are crucial for effective visual understanding. In contrast, the MIM-trained MAR encoder achieves competitive linear probing accuracy and produces precise feature map activations. This verifies our hypothesis that a MIM-based approach can yield more robust semantic representations, which are vital for unifying the tasks of generation and understanding. Figure 3 : We propose Harmon, a unified framework with a shared MAR encoder for image generation and understanding tasks. Building on these insights, in this paper, we present Harmon , a unified autoregressive model for multimodal generation and understanding as shown in Figure 3 . To harmonize visual representations for two tasks in shared encoding, we propose a unified visual encoder (MAR-based) that captures both coarse semantics and fine-grained features and promotes consistent semantic grounding for both tasks. In Harmon , images are generated in a masked autoregressive manner while texts are regressed by next-token prediction. Furthermore, to align this visual encoder with the LLM language space, we design a three-stage training pipeline to gradually develop a unified multimodal generation and understanding model. Thanks to the versatility of the MIM paradigm, the proposed MAR-based encoder can be thoroughly optimized for both generation and understanding in each training stage. We comprehensively assess Harmon on multimodal generation and understanding benchmarks. For text-to-image generation, it surpasses all unified methods with similar model scales on the GenEval benchmark [ 32 ] , which measures the alignment between generated images and user instructions. Harmon also beats all unified models on the WISE benchmark [ 61 ] that necessitates world knowledge comprehension, Harmon produces images with higher visual quality, achieving state-of-the-art performance on the MJHQ30K benchmark [ 45 ] . For image understanding benchmarks, Harmon performs competitively with methods adopting a separate semantic encoder for visual understanding ( i.e . , Janus [ 87 ] and Janus-Pro [ 15 ] ) while surpassing methods that use VQ or VAE encoders by a large margin. Moreover, the synergy between the two tasks via a shared representation is also observed in our experiment, where image generation is improved by co-training with image understanding. 2 Related Work Multimodal Understanding. By injecting visual signals from a semantic encoder to an LLM, multimodal LLMs (MLLMs) [ 99 , 21 , 53 , 52 , 44 , 57 , 16 , 85 ] demonstrate exceptional general and fine-grained image understanding capabilities. Such semantic encoders, including CLIP [ 68 ] and