CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions

Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao · 2026

Read CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions by Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26174.txt.

Introduction

Wang and co-authors present CREval, an automatic way to grade creative image editing under complex instructions. Scoring by large models alone is opaque and often incomplete. CREval instead grades edits through a question-answer pipeline that can be inspected. Its bench, CREval-Bench, holds over 800 editing samples and 13,000 test queries. Tasks span three categories and nine creative dimensions. Closed models beat open ones on these tasks, the authors find. All models still struggle to complete such edits well. User studies show the scores track human judgment closely.

Catalog record

CREATORChonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49498
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao Abstract Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question–answer (QA)–based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval’s automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research. Figure 1 : Evaluation of state-of-the-art image generation and editing models using CREval, with GPT-4o serving as the evaluator. Each edited image is evaluated across three metrics: Instruction Following (IF), Visual Consistency (VC), and Visual Quality (VQ). The results indicate that the complex and creative instructions in CREval-Bench pose substantial challenges for current image manipulation models. † † footnotetext: 🖂Corresponding Author, † Project Leader Figure 2 : Comparison with previous benchmark. The CREval-Bench dataset extends existing instruction-based editing benchmarks by incorporating more complex, creative, and semantically rich instructions. Such design facilitates a comprehensive evaluation of model performance in handling imaginative and complex instruction editing tasks. In (b), the edited image examples on the right correspond one-to-one with the image-instruction pairs on the left. 1 Introduction In the context of image manipulation tasks guided by instructions, can models maintain robust performance when these instructions exhibit high complexity and innovative characteristics? Currently, multimodal generative models 4 ; 52 ; 63 ; 33 ; 22 have demonstrated remarkable capabilities in instruction-based image editing tasks. Notably, models like GPT-Image-1 30 and Gemini 2.5 Flash Image 10 have significantly improved their comprehension of complex instructions compared to earlier models 3 ; 35 ; 40 ; 39 ; 51 . However, current generative image generation and editing models still face significant challenges when handling complex instruction-based tasks, particularly in “free-style creative image editing” scenarios as illustrated in Figure 1 . These challenges include: 1 Insufficient instruction following: Models struggle to accurately interpret and execute complex user instructions, leading to incomplete or incorrect edits; 2 Visual feature inconsistency: Models fail to preserve key visual characteristics of the subject’s identity, resulting in a loss of core information; 3 Poor visual quality: Generated images often contain artifacts and distortions, diminishing reality and fidelity. More critically, current evaluation benchmarks 1 ; 14 ; 29 ; 55 ; 28 ; 6 ; 50 ; 42 primarily focus on common tasks like object addition, replacement, deletion, color adjustment, and logical reasoning as shown in Figure 2 (a). These works fail to effectively evaluate the performance of free-style creative image editing tasks, as illustrated by the example case in Figure 2 (b). Table 1 : Comparison to other existing benchmark. Our benchmark provides a comprehensive evaluation of creative image manipulation by leveraging VQA-based scoring. Dataset Size Scoring Creative Fully-Automatic ImgEdit-Bench 55 791 MLLMs scoring ✗ ✓ KRIS-Bench 50 1267 MLLMs scoring ✗ ✓ RISE-Bench 62 360 MLLMs scoring ✗ ✓ GEdit-Bench 27 606 MLLMs scoring ✗ ✓ CREval-Bench 874 VQA scoring ✓ ✓ To address this gap, we propose CREval, a fully automated, multidimensional evaluation pipeline, along with a benchmark named CREval-Bench. This framework is specifically designed to provide objective, fully automated evaluation for complex instruction–based creative image manipulation. CREval-Bench covers three major categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. As summarized in Table 1 , in contrast to prior approaches that rely solely on Multimodal Large Language Models (MLLMs) as holistic evaluators, we decompose the evaluation process into three complementary metrics: Instruction Following (IF), Visual Consistency (VC), and Visual Quality (VQ). For each sample, we derive targeted Question–Answer (QA) pairs grounded in the original image and the associated instruction. Instead of directly asking MLLMs to assign scores, which often leads to incomplete coverage of evaluation dimensions and limited interpretability due to the lack of transparent scoring rationales, we prompt MLLMs to respond to these structured queries. This enables transparent scoring because the responses explicitly indicate where points should be awarded or deducted, leading to more comprehensive and interpretable evaluation. The evaluation process uses MLLMs ( e.g . , GPT-4o) to compute quantitative scores from triplets {[I_{i},I_{o},Q]} , which represent the input image, output image, and evaluation query. These triplets are fed into the MLLMs to generate objective evaluation results. We conduct a comprehensive evaluation of mainstream open- and closed-source editing models using CREval. The results show that Seedream 4.0 38 achieves the highest overall performance, exhibiting a strong balance across IF, VC, and VQ. Qwen-Image-Edit-2509 48 ranks second, with both models outperforming GPT-Image-1 30 in aggregate performance. While closed-source models currently lead in overall performance, open-source models have shown promising progress. With continued technological advancement and community development, the competitiveness of open-source models is expected to further improve in the near future. In summary, the main contributions of this paper are as follows: • We propose CREval , a fully automated and QA-based evaluation framework for CR eative image manipulation under complex instructions, addressing the limitations of MLLMs scoring. • We build CREval-Bench , a comprehensive benchmark covering 3 categories and 9 dimensions to systematically and fairly evaluate diverse creative editing scenarios. • We conduct extensive experiments on state-of-the-art image generation and editing models, revealing their strengths and limitations in handling complex and flexible editing tasks, and providing insights for future research. • User studies demonstrate strong consistency between CREval scores and human preference judgments, confirming the reliability and robustness of the proposed evaluation framework. 2 Related Work 2.1 Instruction-based Image Editing Models Instruction-based image editing models aim to achieve semantic-level modifications and creations of image content by precisely understanding natural language instructions. The core challenge lies in balancing the accuracy of instruction adherence, the structural fidelity of the original image, and the generalization capability of the editing task. Early works such as 16 ; 3 ; 57 ; 14 ; 60 ; 20 ; 34 have achieved promising results in image editing guided by human instructions. 43 ; 61 ; 41 ; 47 further extends this line of research by learning visual features to enable more precise and controllable image modifications. 58 ; 13 ; 17 integrates MLLMs to enhance semantic understanding and reasoning capabilities, thereby addressing the limitations of CLIP-based text encoders, which often fail to produce satisfactory representations in complex scenarios. Subsequent developments in image editing have shifted from single diffusion models toward multimodal Transformer 45 and Flow Matching 26 architectures. Stable Diffusion 3 8 replaces the conventional U-Net

Read the complete text