UmniBench: Unified Understanding and Generation Model Oriented Omni-dimensional Benchmark

Kai Liu, Leyang Chen, Wenbo Li, Zhikai Chen, Zhixin Wang, Renjing Pei, Linghe Kong, Yulun Zhang · 2025

Read UmniBench: Unified Understanding and Generation Model Oriented Omni-dimensional Benchmark by Kai Liu, Leyang Chen, Wenbo Li, Zhikai Chen, Zhixin Wang, Renjing Pei, Linghe Kong, Yulun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17196.txt.

Introduction

Kai Liu and coauthors present UmniBench, a benchmark that grades unified multimodal models on understanding, generation, and editing in one pass. Past grading split those skills across separate data sets, missing how a single model blends them.

UmniBench uses human-checked prompts and question pairs, then lets the model under test judge its own generations and edits with its understanding skill. The bench spans 13 domains and over 200 concepts, and it can also split scores by skill for fine reading. Twenty-four models were graded, covering unified systems and single-skill large models. The team offers the bench as a fuller, fairer yardstick for unified models.

Catalog record

CREATORKai Liu, Leyang Chen, Wenbo Li, Zhikai Chen, Zhixin Wang, Renjing Pei, Linghe Kong, Yulun Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UmniBench: Unified Understanding and Generation Model Oriented Omni-dimensional Benchmark Kai Liu, Leyang Chen, Wenbo Li, Zhikai Chen, Zhixin Wang, Renjing Pei, Linghe Kong, Yulun Zhang Abstract Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model. 1 Introduction A unified multimodal model (UMM) refers to a single model that seamlessly integrates understanding, generation, and editing within a single architecture, with the aim that these capabilities mutually reinforce one another 8 ; 31 ; 26 . Recent advances in UMMs have shown impressive capabilities in cutting-edge models, which mainly focus on architecture design, post-training paradigms, and dataset construction. However, current evaluation protocols for these models are largely decoupled 9 ; 32 . They assess understanding, generation, and editing separately rather than in an integrated manner. To be specific, understanding benchmarks typically take text and images as input and use multiple-choice answers to probe comprehension and reasoning 7 . By contrast, generation and editing benchmarks take short textual prompts or text–image pairs as input to assess quality and consistency of the output image 11 . Despite all these benchmarks allowing accurate evaluation in their corresponding sub-tasks, they are inherently unsuitable for evaluating UMMs. This is because these evaluations capture each facet in isolation, and they do not reflect the holistic competence of unification, namely, the seamless integration of understanding, generation, and editing. Figure 1 : Advantages of our proposed UmniBench compared with previous UMMs isolated evaluation protocols. Figure 2 : The overview of UmniBench. All 13 domains involved in UmniBench are enumerated in the table above, with left-hand and right-hand panels presenting representative images generated under each concept under the specific domain. Consequently, there is an urgent need for a benchmark specifically designed to evaluate the holistic capabilities of unified generation–understanding models. To address this gap, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. While existing pure generation and editing models have achieved remarkable image quality, they remain deficient in complex instruction following 2 . In particular, these models often struggle to accurately interpret complex user intents, especially when the input requires nontrivial reasoning. Researchers have therefore begun integrating understanding capabilities into generative models, aiming to improve unified models’ handling of complex intents in generation and editing tasks. Accordingly, our proposed UmniBench focuses on image generation and editing scenarios that demand substantive understanding and reasoning. Conventional benchmarks typically rely on external models, hand-crafted rules, or human evaluators to score system outputs 23 . Because UMMs intrinsically possess understanding, generation, and editing capabilities, they can, in principle, serve as their own evaluators. A natural property of image generation and editing tasks is that the prompt itself constitutes a relatively accurate description of the target image. Consequently, we can derive question–answer pairs about image content directly from the prompt and skip the generated image. Once a generated or edited image is obtained, these QA pairs can be used to assess: (1) whether the model’s generation or editing adheres to the instruction, and (2) whether the model can comprehend the image and the question to produce the correct answer. The entire evaluation pipeline can thus be executed by the UMMs alone, enabling self-assessment of its understanding, generation, and editing abilities. Table 1 : Summary of Multimodal Benchmarks. We compare existing works from the following aspects: multi-modal evaluation, number of single abilities, unified ability, dependency on third-party review, and diversity of evaluation. Benchmark Publication Reasoning # Single Unified No third-party Diversity Abilities Ability review req. Tags Sub-tags ReasonPix2Pix 13 arXiv’24 ✓ Edit ✗ ✓ - - ReasonEdit 12 arXiv’24 ✓ Edit ✗ ✗ - - GenEval 9 NeurIPS’23 ✗ Gen. ✗ ✗ 6 6 DPG-Bench 10 arXiv’24 ✗ Gen. ✗ ✗ 5 13 T2I-CompBench++ 11 NeurIPS’23 ✗ Gen. ✗ ✗ 4 8 ConceptMix 32 NeurIPS’24 ✗ Gen. ✗ ✗ 8 8 KRIS-Bench 33 arXiv’24 ✓ Edit ✗ ✗ 3 22 EditWorld 38 MM’25 ✓ Edit ✗ ✗ 7 7 RISEBench 41 NeurIPS’25 ✓ Edit ✗ ✗ 4 4 UmniBench Ours ✓ Und., Gen., Edit ✓ ✓ 13 195 Moreover, prior benchmarks typically cover only a single vertical domain of knowledge, such as mathematics or coding. In contrast, researchers envision UMMs that span the vast majority of human knowledge. Accordingly, UmniBench comprises 13 domains that encompass common categories, including the spatial relations, logical reasoning, and everyday scenarios, as depicted in Fig. 2 . Within each domain, there are 15 concepts, and each concept contains 3 cases. Collectively, this design ensures a comprehensive, multi-scenario evaluation of model robustness and stability. Moreover, the self-generation-self-evaluate paradigm is inherently immune to the risk of data leakage. Our core contributions can be summarized as threefold: • We propose UmniBench , a benchmark target for UMMs with omni-dimensional evaluation. To the best of our knowledge, UmniBench is the first benchmark to assess models not only their understanding and generation abilities individually, but also jointly. • UmniBench provides omni-dimensional evaluation. 13 domains and 15 concepts within each domain are included to conduct a nuanced evaluation. Moreover, the unique evaluation method of UmniBench is inherently immune to the risk of data leakage. • State-of-the-art open-sourced methods and proprietary models with unified ability or single ability are evaluated based on our proposed benchmark. 2 Related Works 2.1 Unified Multimodal Model UMMs refer to architectures that possess both understanding and generation capabilities, with the aim that the two abilities complement and reinforce each other 16 ; 31 . In terms of design philosophy, current designs mainly embed the two capabilities in distinct parameter sets 17 ; 16 . Accordingly, we can categorize models by their degree of coupling into tightly coupled and loosely coupled variants. In tightly coupled models , information flows bidirectionally and interleaves across the two parameter spaces 8 . Taking Bagel as an example, a Mixture-of-Experts architecture processes tokens for understanding and generation separately, while allowing interaction through self-attention. By contrast, loosely coupled models permit primarily unidirectional information flow—from the parameters associated with one ability to those of the other. Concretely, in UniPic2, tokens produced by the MLLM for understanding are injected into the diffusion model via a lightweight linear layer. However, tokens from the diffusion process do not influence the tokens output by the MLLM. 2.2 Benchmarks Evaluation of UMMs is typically conducted by employing multiple benchmarks that assess each capability in isolation.

Read the complete text