ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services

Fengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen · 2026

Read ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services by Fengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24023.txt.

Introduction

Ji and seven colleagues introduce ServImage, a benchmark that ties image generation and editing to real commercial value. It gathers paid design tasks and designer deliverables, scores outputs on meeting briefs, visual quality, and commercial need, and adds a model that predicts whether a human would pay. That payment model reaches 82 percent accuracy on human-annotated candidates, grounding image evaluation in buying decisions.

Catalog record

CREATORFengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58453
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services Fengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen Abstract Recent image generation and editing models demonstrate robust adherence to instructions and high visual quality on academic benchmarks. However, their performance on paid, real-world design projects remains uncertain. We introduce ServImage , a benchmark that explicitly correlates model outputs with economic value in commercial design projects. ServImage consists of (i) ServImageBench : a dataset of 1.07k paid commercial design tasks and 2.05k designer deliverables totaling over $295k, covering portrait, product, and digital content, along with 33k candidate images and 33k human annotations. (ii) ServImageScore : an integrated scoring system that combines three quality dimensions: baseline requirements fulfilment, visual execution quality, and commercial necessity satisfaction. These three dimensions are designed to characterize the factors that drive human payment decisions and indicate whether an image is commercially acceptable. (iii) ServImageModel : under this scoring system, we propose a payment prediction model trained on the human-annotated candidate images, achieving 82.00% accuracy in predicting human payment decisions and producing calibrated payment probabilities. ServImage provides a comprehensive foundation for assessing the commercial viability of image generation models and offers a scalable resource for future research on economically grounded vision systems Github. rmTeXGyreTermesX [*devanagari]rmLohit Devanagari [*arabic]rmNoto Sans Arabic 1 Introduction Figure 1: Overview of the ServImage benchmark and evaluation framework. (a) We collect 1,070 paid design tasks from online crowdsourcing platforms and group them into Portrait, Product, and Digital categories. (b) Through the dataset pipeline with rule extraction/point mining, we obtain BRF, VEQ, and CNS scores to form ServImageScore. (c) Sixteen image generation and editing models produce about 33k candidate images. (d) Optionally, built on these scores, ServImageModel predicts human payment decisions. (e) We present representative tasks and model outputs as ServImage case studies. (f) Under the standard settlement scenario, we evaluate each model independently on the full task set and compare their commercial capability using the economic metric of total revenue. Recent years have witnessed remarkable progress in image generation and editing. In image generation, diffusion models such as Stable Diffusion Rombach et al. 2022 and GLIDE Nichol et al. 2021 have advanced high-quality text-to-image generation. In image editing, models like InstructPix2Pix Brooks et al. 2023 and MagicBrush Zhang et al. 2023 ; Zhang et al. 2025a have improved editing precision and controllability. These advances have broadened the use of image generation from conventional tasks such as object removal and style transfer to more complex design tasks, including poster design, logo creation, and IP illustration Chen et al. 2024 ; Wang et al. 2025b , which originate from real-world commercial scenarios with well-defined requirements and economic returns. On the evaluation side, recent benchmarks emphasized instruction following and semantic controllability Ma et al. 2024 , optical realism and reflection consistency Zeng et al. 2024 , and physical commonsense understanding, ensuring that generated images obey plausible geometry, gravity, and material properties Farshad et al. 2023 ; Zhao et al. 2025 ; Ryu et al. 2025 ; Gu et al. 2025 . However, to the best of our knowledge, no existing benchmark evaluates how well image models satisfy real-world commercial design requirements, particularly in terms of practical usability and economic acceptability. In real design services, commercial acceptability depends on a coupled set of factors beyond semantic and perceptual correctness. A deliverable must first satisfy baseline business requirements, such as size, resolution, format, copyright safety, and policy compliance, before it can even be considered for delivery. It must then achieve sufficient visual execution quality to be publishable, and, crucially, align with the client’s specific commercial intent, including brand style, marketing message, and platform constraints. Whether a client approves and pays for an image ultimately reflects these business-rule, visual, and commercial considerations jointly, which are not explicitly captured by existing benchmarks. The fundamental question in a business context is not merely “Is it good?” but rather “Is it worth paying for?” Patwardhan et al. 2025 ; Mazeika et al. 2025 . To close this gap, we introduce ServImage , a benchmark explicitly grounded in real monetary outcomes for commercial design services. Concretely, ServImage contains three components. ServImageBench collects 1.07k paid commercial design tasks from crowdsourcing platforms, with over $295k of contract value, together with 33k candidate images and 33k human payment decisions from 16 mainstream models, spanning portrait services, e-commerce products, and digital content, and covering both text-to-image generation and image-editing workflows. ServImageScore provides an integrated evaluation scheme that decomposes image quality into Baseline Requirements Fulfilment, Visual Execution Quality, and Commercial Necessity Satisfaction, with a unified scoring protocol that jointly captures business-rule adherence, perceptual quality, and task-specific commercial fit. Finally, ServImageModel is a settlement model that, guided by ServImageScore and task prices, produces calibrated payment probabilities, serving as a reference method for estimating the expected revenue of different models under realistic payment rules. ServImage demonstrates from the perspective of commercial payment that models which appear strong under technical metrics capture only a fraction of the attainable commercial value. In summary, our contributions include: advancing generative model evaluation from traditional technical metrics to a market-grounded perspective based on real paid tasks, human payment behavior, and price structures; uncovering the mechanisms by which commercial value is formed, showing that user preferences, task prices jointly determine economic outcomes beyond image quality alone; and providing an economic inference framework that estimates attainable revenue under realistic payment rules, revealing a substantial gap between technical performance and actual commercial value capture. 2 Related Work Image Generation and Editing Benchmarks. Evaluation of generative models has progressed from traditional single-number perceptual metrics, CLIP Score, LPIPS, and PSNR/SSIM Radford et al. 2021 ; Zhang et al. 2018 ; Wang et al. 2004 to multi-dimensional assessment frameworks. Generation-oriented benchmarks such as GenAI-Bench and OmniGenBench foreground compositional reasoning and text rendering Li et al. 2024 ; Peng et al. 2024 ; Zhou et al. 2025 , while editing-oriented ones like EDITVAL, VIEScore, and IE-Bench emphasise controllability and region consistency Basu et al. 2023 ; Sun et al. 2025 ; Wang et al. 2025a . Unified efforts, including I2EBench, ICE-Bench, and RISEBench, attempt to bridge this divide by jointly evaluating instruction following, visual quality, and reasoning Ma et al. 2024 ; Pan et al. 2025 ; Xu et al. 2025b . However, a fundamental limitation persists: existing tasks lack market-validated value distributions, and current evaluations rely on conventional technical metrics that provide no insight into commercial viability. VLM-based evaluation for image generation and edition. To automate assessment, the VLM-as-a-Judge paradigm has become prevalent, using task-specific prompts to achieve high correlations with human preference Wu et al. 2025 ; Ye et al. 2025 ; Pu et al. 2025 ; Wang et al. 2025b ; Zhang et al. 2025b ; Wang et al. ; Yang et al. 2026 . Systems like LMM4Edit Xu et al. 2025a ; Ji et al. 2025 use few-shot learning for robust editing evaluation. While these methods align well with human perception, they don’t capture the economic value of outputs. This gap shows a decoupling where high technical metrics don’t necessarily lead to willingness to pay. Although similar mappings from model performance to economic outcomes have been explored in other domains,

Read the complete text