UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation
Bin Lin 1,3 , Zongjian Li 1,3 , Xinhua Cheng 1,3 , Yuwei Niu 1,3 , Yang Ye 1,3 , Xianyi He 1,3 , Shenghai Yuan 1,3 , Wangbo Yu 1,3 , Shaodong Wang 1,3 , Yunyang Ge 1,3 , Yatian Pang 1 , Li Yuan 1,2, † · 2025
Read UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation by Bin Lin 1,3 , Zongjian Li 1,3 , Xinhua Cheng 1,3 , Yuwei Niu 1,3 , Yang Ye 1,3 , Xianyi He 1,3 , Shenghai Yuan 1,3 , Wangbo Yu 1,3 , Shaodong Wang 1,3 , Yunyang Ge 1,3 , Yatian Pang 1 , Li Yuan 1,2, † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03147.txt.
Introduction
Bin Lin and co-authors present UniWorld-V1 in a 2025 paper on a single model for seeing and making images. Most unified systems handle image understanding and text-to-image generation, but real applications also demand careful perception and editing of existing pictures.
Inspired by clues about how a leading commercial model works, the authors build on semantic features from multimodal language models rather than the usual image compressors. Trained on a modest few million examples, the system performs strongly across understanding, generation, editing, and perception tasks. The team released the model, code, and data openly to support further work.
Catalog record
| CREATOR | Bin Lin 1,3 , Zongjian Li 1,3 , Xinhua Cheng 1,3 , Yuwei Niu 1,3 , Yang Ye 1,3 , Xianyi He 1,3 , Shenghai Yuan 1,3 , Wangbo Yu 1,3 , Shaodong Wang 1,3 , Yunyang Ge 1,3 , Yatian Pang 1 , Li Yuan 1,2, † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03147
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation Bin Lin 1,3 , Zongjian Li 1,3 , Xinhua Cheng 1,3 , Yuwei Niu 1,3 , Yang Ye 1,3 , Xianyi He 1,3 , Shenghai Yuan 1,3 , Wangbo Yu 1,3 , Shaodong Wang 1,3 , Yunyang Ge 1,3 , Yatian Pang 1 , Li Yuan 1,2, † Abstract Although existing unified models achieve strong performance in vision-language understanding and text-to-image generation, they remain limited in addressing image perception and manipulation—capabilities increasingly demanded in practical applications. Recently, OpenAI introduced the powerful GPT-4o-Image model, which showcases advanced capabilities in comprehensive image perception and manipulation, sparking widespread interest. Through carefully designed experiments, we observe that GPT-4o-Image likely relies on semantic encoders rather than VAEs for feature extraction, despite VAEs being commonly regarded as crucial for image manipulation tasks. Inspired by this insight, we propose UniWorld-V1 , a unified generative framework built upon semantic features extracted from powerful multimodal large language models and contrastive semantic encoders. Using only 2.7M training data, UniWorld-V1 achieves impressive performance across diverse tasks, including image understanding, generation, manipulation, and perception. We fully open-source the UniWorld-V1 framework, including model weights, training and evaluation scripts, and datasets to promote reproducibility and further research. Code: https://github.com/PKU-YuanGroup/UniWorld-V1 Models: https://huggingface.co/LanguageBind/UniWorld-V1 Data: https://huggingface.co/datasets/LanguageBind/UniWorld-V1 Figure 1 : Showcase of UniWorld-V1’s versatile capabilities. The left two panels illustrate image perception and manipulation examples, and the right panel presents comparisons with state-of-the-art models and training data resources. 1 Introduction Unifying image understanding and generation has demonstrated remarkable capabilities on multi-modal models. Recent studies [ 40 , 55 , 44 ] demonstrate that carefully designed architectures can be jointly optimized to perform well on both understanding and generation benchmarks. The remarkable visual understanding and generation capabilities exhibited by the recent GPT-4o-Image model have further energized the open-source community, leading to a surge of new unified models [ 4 , 9 ] aiming to replicate its performance. GPT-4o-Image achieves superior generation performance on various image-to-image tasks across different domains, which can be divided into two categories, including image perception tasks [ 36 , 8 ] ( e.g. , detection) and image manipulation [ 48 ] ( e.g. , editing) tasks. Nevertheless, most unified models are limited to image-to-language understanding tasks and language-to-image generation tasks, with very few addressing image-to-image perception and manipulation tasks. However, attempting to tackle image perception and manipulation tasks in one model is difficult because such an all-in-one model requires multiple superior capabilities, including (1) the textual and visual unified understanding ability for correctly addressing user intention, (2) the pixel-level information maintaining ability for image reconstruction and region editing, and (3) the semantic extraction ability for cross-domain perception and visual conception composition. Therefore, the requirements of various capabilities impose specific designs for the unified generation model, especially for visual feature injection. Recent attempts including Step1X-Edit [ 26 ] and FLUX-Kontext [ 15 ] introduce variational autoencoders (VAEs) for extracting visual features and perform well on individual image editing tasks. However, their methods encounter challenges when extended to multiple perception and manipulation tasks simultaneously due to the visual features encoded by VAEs, which imply a wealth of low-frequency information, thereby limiting their performance when facing semantic-level tasks. Inspired by the success of GPT-4o-Image, we investigate the integration of visual features into unified generative models for image manipulation tasks, a direction that remains insufficiently explored in current research. Therefore, we carefully construct experiments on GPT-4o-Image to infer the visual feature extraction manner that GPT-4o-Image likely adopts, and we infer that GPT-4o-Image employs visual features extracted by semantic encoders rather than VAEs by observing from the experimental results. Based on our essential observation shown in Section 2 , we propose a unified generative model named UniWorld-V1 for both image perception and manipulation tasks, which consists of pre-trained multi-modal large models for providing auto-regressive understanding tokens and pre-trained high-resolution contrastive semantic encoders for extracting visual features with both pixel-level local information and semantic-level global conceptions. As a result, UniWorld-V1, trained on only 2.7M samples, consistently outperforms BAGEL [ 9 ] (trained on 2665M samples) on the ImgEdit-Bench [ 48 ] for image manipulation. It also surpasses the specialized image editing model Step1X-Edit across multiple dimensions, including add, adjust, and extract on ImgEdit-Bench. Additionally, for text-to-image generation, UniWorld-V1 outperforms BAGEL on WISE [ 29 ] and achieves performance comparable to GPT-4o-Image on GenEval [ 14 ] . Furthermore, beyond its strong capabilities in the aforementioned tasks, UniWorld-V1 also excels in Image Perception tasks such as detection, segmentation, and depth prediction. In summary, we are the first open-source model to achieve such comprehensive and powerful capabilities across the multimodal domain. We summarize our primary contributions as follows: • Provide insights into unified architecture design by hypothesizing, through extensive observation of GPT-4o-Image, that it likely does not adopt a VAE-based structure. • Propose UniWorld-V1, a unified architecture that leverages a high-resolution semantic encoder to deliver reference-image control signals. UniWorld-V1 achieves performance comparable to BAGEL using only 2.7M training samples. • Collect, curate, and open-source high-quality training data. We fully release the code, model weights, datasets, and training & evaluation scripts to support and advance future research. Figure 2 : Empirical Observations of GPT-4o-Image. We verify local consistency of edits in (a). We explore the relationship between model comprehension and generation in (b)–(e), conducting observations within the GPT-4o architecture and across architectures using Qwen2.5VL-32B. 2 Observation GPT-4o-Image [ 1 ] achieves impressive performance in various image tasks in a generative way, and we divide supported image tasks into two categories: Image Perception (detection, segmentation, depth prediction, etc. ) and Image Manipulation (editing [ 26 , 58 , 6 ] , reference style transfer [ 38 ] , subject consistency generation [ 52 , 51 ] , etc. ). It is generally believed within the community that GPT-4o-Image demonstrates the necessity of unifying understanding and generation by integrating an auto-regressive understanding module [ 60 , 19 , 21 ] with a diffusion-based generation module [ 47 ] for addressing various image tasks with complex requirements. However, we consider that additional visual features beyond auto-regressive tokens from understanding models should be injected to maintain image information, since we fail to train an effective generation model from solely VLM outputs. Recent notable unified image manipulation approaches, exemplified by Step1X-Edit [ 26 ] and FLUX-Kontext [ 15 ] , simultaneously introduce variational autoencoders (VAEs) to extract image featuresa as reference image control. Although their models perform well on individual editing tasks, our experiments show that they fail to converge when extended to multiple image perception and manipulation tasks, which indicates that the manner of additional visual feature injection remains underexplored. To explore the visual feature injection method that GPT-4o-Image utilizes, we construct two groups of experiments and obtain several key observations (see Figure 2 ), from which we infer that GPT-4o-Image likely employs visual features extracted by semantic encoders rather than VAEs. Editing Experiment. We first require GPT-4o-Image to execute a local image editing task with the instruction: “Turn the advertisement on the back of the bus blue”, as shown in Figure 2 (a). Before the editing, both yellow-labeled and green-labeled texts are in the top-right corner of the bus. However, the yellow-labeled text is on the right,