RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark

Yang Shi Yuhao Dong Yue Ding Yuran Wang Xuanyu Zhu Sheng Zhou Wenting Liu, Haochen Tian Rundong Wang Huanqian Wang Zuyan Liu Bohan Zeng Ruizhe Chen Qixun Wang, Zhuoran Zhang Xinlong Chen Chengzhuo Tong Bozhou Li Chaoyou Fu Qiang Liu, Haotian Wang Wenjing Yang Yuanxing Zhang Pengfei Wan Yi-Fan Zhang · 2025

Read RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark by Yang Shi Yuhao Dong Yue Ding Yuran Wang Xuanyu Zhu Sheng Zhou Wenting Liu, Haochen Tian Rundong Wang Huanqian Wang Zuyan Liu Bohan Zeng Ruizhe Chen Qixun Wang, Zhuoran Zhang Xinlong Chen Chengzhuo Tong Bozhou Li Chaoyou Fu Qiang Liu, Haotian Wang Wenjing Yang Yuanxing Zhang Pengfei Wan Yi-Fan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24897.txt.

Introduction

Yang Shi and co-authors build RealUnify, a benchmark that tests whether unified vision models truly gain from joining seeing and making. Many new models both read images and generate them in one body. Old tests grade each skill alone, so they cannot tell if the two skills help each other.

RealUnify holds one thousand hand-checked cases across ten classes and thirty-two subtasks. One axis asks if understanding guides generation, such as using logic to draw the right scene. The other asks if generation aids understanding, by rebuilding a scrambled input in the mind. A dual scoring plan grades both the final answer and the direct skill path. Early runs expose how far true synergy still has to go. The set gives builders a sharper target for general-purpose vision models.

Catalog record

CREATORYang Shi Yuhao Dong Yue Ding Yuran Wang Xuanyu Zhu Sheng Zhou Wenting Liu, Haochen Tian Rundong Wang Huanqian Wang Zuyan Liu Bohan Zeng Ruizhe Chen Qixun Wang, Zhuoran Zhang Xinlong Chen Chengzhuo Tong Bozhou Li Chaoyou Fu Qiang Liu, Haotian Wang Wenjing Yang Yuanxing Zhang Pengfei Wan Yi-Fan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark Yang Shi Yuhao Dong Yue Ding Yuran Wang Xuanyu Zhu Sheng Zhou Wenting Liu, Haochen Tian Rundong Wang Huanqian Wang Zuyan Liu Bohan Zeng Ruizhe Chen Qixun Wang, Zhuoran Zhang Xinlong Chen Chengzhuo Tong Bozhou Li Chaoyou Fu Qiang Liu, Haotian Wang Wenjing Yang Yuanxing Zhang Pengfei Wan Yi-Fan Zhang Ziwei Liu Abstract The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify , a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation , which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding , which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol , which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling. 1 Introduction The field of multimodal artificial intelligence has undergone a paradigm shift with the rise of unified models that integrate both visual understanding (e.g., visual question answering) and generation (e.g., text-to-image synthesis) within a single neural architecture ( Deng et al. 2025 ; Lin et al. 2025 ; OpenAI 2025 ) . While such unification offers architectural elegance, its most compelling promise lies in the potential for synergetic effects between capabilities: leveraging knowledge and reasoning from understanding to guide more accurate generation , and employing internal generative simulation (e.g., “thinking with images”) to facilitate more complex understanding . A fundamental open question remains: do the two core capabilities, i.e., understanding and generation, mutually enhance each other? This question further motivates a reconsideration of architectural design: should we pursue a unified model that integrates both, or co-located models without functional synergy? A primary obstacle in answering this question is the lack of a suitable benchmark . As illustrated in Figure 1 , current evaluation frameworks predominantly assess understanding and generation in isolation (Stage 1), and some benchmarks ( Xie et al. 2025b ) combine tasks from both domains to evaluate capabilities simultaneously (Stage 1.5). Recent efforts like T2I-CoReBench ( Li et al. 2025c ) and WISE ( Niu et al. 2025 ) have begun exploring whether understanding enhances generation quality, but do not explicitly test whether success on a task depends on the interaction of both capabilities. Thus, there remains a pronounced lack of rigorous benchmarks with systematic design to probe the very essence of unification: bidirectional capability synergy. For example, while text-to-image benchmarks excel at measuring output fidelity and aesthetic quality, they cannot determine whether a model’s reasoning abilities meaningfully enhance its generative process, nor whether generative simulation improves visual understanding. Figure 1: Illustration of RealUnify . Unlike benchmarks focused on either understanding or generation (Stage 1), those that merely integrate both capabilities (Stage 1.5), or even those that preliminarily explore the mutual enhancement between understanding and generation (Stage 2), RealUnify stands as the first benchmark to comprehensively evaluate and fully harness the synergy between these capabilities, making it a pioneering effort in assessing ability synergy for unified models. To address this gap, we introduce RealUnify, the first benchmark aimed at answering the fundamental question: Can unified models effectively leverage their synergy between understanding and generation abilities to solve complex tasks? The core innovation of RealUnify lies in its meticulously designed task suite, where each instance requires an intricate interplay between understanding and generation. RealUnify structures its evaluation around two core tracks: (1) Understanding Enhances Generation (UEG), which tests whether knowledge and reasoning improve generation accuracy, and (2) Generation Enhances Understanding (GEU), which examines whether generative reconstruction and visualization can support more effective visual reasoning. As shown in Table 1 , its 1,000 instances span 10 categories and 32 manually crafted and validated subtasks, which in particular requires synergy between understanding and generation, such as tasks that require mathematical computation prior (understanding) to generate images and visual tracking of multi-step transformations (generation) to answer questions (understanding). Moreover, a cornerstone of RealUnify is its dual-evaluation protocol including the direct evaluation and stepwise evaluation, enabling precise diagnosis of whether unified models achieve genuine capability synergy or merely functional coexistence. Specifically, direct evaluation tests whether models can achieve end-to-end synergy in a realistic setting (closer to the intrinsic capability of models during the realistic deployment), whereas stepwise evaluation decomposes tasks into understanding and generation, revealing whether performance limits arise from weak individual capabilities or from the lack of genuine synergy. Through extensive evaluations of 12 leading unified models and 6 state-of-the-art specialized baselines using our dual-evaluation protocol, we uncover a striking conclusion: despite their unified architecture, current models still struggle to synergize understanding and generation capabilities effectively. This finding is robustly supported by three key empirical patterns. First, under direct evaluation , models perform poorly on both UEG (average 37.5\% for best open-source) and GEU tasks, indicating their inability to spontaneously integrate capabilities in end-to-end scenarios. Second, and more diagnostically, the stepwise evaluation reveals a revealing dissociation: when UEG tasks are decomposed into “understanding-then-generation” stages, performance improves significantly (e.g., BAGEL ( Deng et al. 2025 ) improves from 32.7\% to 47.7\% ), demonstrating that models possess the required knowledge but cannot seamlessly integrate it. Conversely, decomposing GEU tasks into “generation-then-understanding” stages causes performance to degrade, suggesting that models default to relying on understanding shortcuts rather than effectively leveraging generation. Third, when we construct an “oracle” model by combining the best specialist models (Gemini-2.5-Pro ( Comanici et al. 2025 ) for understanding and GPT-Image-1 ( OpenAI 2025 ) for generation) in a stepwise manner, it achieves 72.7\% on UEG tasks, establishing a high-performance upper bound that current unified models fall far short of. Collectively, these results indicate that architectural unification alone is insufficient. To fully realize the potential of capability synergy, unified models require more advanced training schemes and stronger inductive biases. 2 Related Work Unified Multimodal Models. Recently, unified models ( Wang et al. 2024a ; Chen et al. 2025b ; Team 2024 ; Deng et al. 2025 ; Wu et al. 2025b ; Wang et al. 2025a ; Lin et al. 2025 ) have emerged as a central research direction in multimodal intelligence. These frameworks integrate both visual understanding and generation within a single architecture and have

Read the complete text