GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models

Muhammad Atif Butt, Alexandra Gomez-Villa, Tao Wu, Javier Vazquez-Corral, Joost Van De Weijer, & Kai Wang · 2025

Read GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models by Muhammad Atif Butt, Alexandra Gomez-Villa, Tao Wu, Javier Vazquez-Corral, Joost Van De Weijer, & Kai Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20586.txt.

Introduction

Muhammad Atif Butt and five co-authors present GenColorBench, the first comprehensive benchmark for color precision in text-to-image generation. Modern image models produce high-quality pictures but still fail at fine-grained color control, often missing the exact color a prompt names. Existing benchmarks test composition and prompt adherence while neglecting color or judging it only coarsely, including no treatment of numerical color values.

The benchmark grounds its tests in established color systems and contains forty-four thousand color-focused prompts covering more than four hundred colors. Perceptual and automated assessments evaluate popular models and reveal wide performance variation across color conventions. The authors identify which conventions models understand best and where they fail, offering the benchmark as a guide for improving precise color generation.

Catalog record

CREATORMuhammad Atif Butt, Alexandra Gomez-Villa, Tao Wu, Javier Vazquez-Corral, Joost Van De Weijer, & Kai Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models Muhammad Atif Butt, Alexandra Gomez-Villa, Tao Wu, Javier Vazquez-Corral, Joost Van De Weijer, & Kai Wang Abstract Recent years have seen impressive advances in text-to-image generation, with image generative or unified models, generating high-quality images from text. Yet these models still struggle with fine-grained color controllability, often failing to accurately match colors specified in text prompts. While existing benchmarks evaluate compositional reasoning and prompt adherence, none systematically assess the color precision. Color is fundamental to human visual perception and communication, critical for applications from art to design workflows requiring brand consistency. However, current benchmarks either neglect color or rely on coarse assessments, missing key capabilities like interpreting RGB values or aligning with human expectations. To this end, we propose GenColorBench, the first comprehensive benchmark for T2I color generation, grounded in color systems like ISCC-NBS and CSS3/X11, including numerical colors which are absent elsewhere. With 44K color-focused prompts covering 400+ colors, it reveals models’ true capabilities via perceptual and automated assessments. Evaluations of popular T2I models using GenColorBench show performance variations, highlighting which color conventions models understand best and identifying failure modes. Our GenColorBench assessments will allow to guide improvements in precise color generation. The benchmark will be made public upon acceptance. 1 Introduction Text-to-image (T2I) generation has witnessed remarkable progress in recent years, with state-of-the-art models like Stable Diffusion ( Rombach et al. 2022 ) and FLUX ( Labs 2024 ) demonstrating unprecedented capabilities in generating high-quality, photorealistic images from text prompts. These advances have enabled diverse applications ranging from creative content generation to automated design workflows. However, despite their impressive overall performance, T2I models still struggle with fine-grained controllability, particularly in generating images that precisely match specific visual attributes described in text prompts ( Chefer et al. 2023 ; Ge et al. 2023a ) . While numerous benchmarks, discussed in Table 1 , have been proposed to evaluate various aspects of T2I model performance—including compositional reasoning ( Huang et al. 2025 ; Ghosh et al. 2023 ) , prompt adherence ( Hu et al. 2024 ) , and faithfulness ( Hu et al. 2023 ) —none systematically evaluates the critical ability to generate precise colors as specified in text prompts. Color represents a fundamental dimension of human visual perception and serves as a primary channel for human communication about objects and scenes, with color categories forming a universal basis for describing and distinguishing visual phenomena across cultures ( Berlin & Kay 1991 ; Witzel & Gegenfurtner 2018 ) . This perceptual importance translates directly into practical applications where accurate color generation is essential—from multimedia applications and artistic creation to design workflows requiring brand consistency, aesthetic control and faithful reproduction of real-world scenes. However, existing T2I evaluation benchmarks critically underestimate this importance by either neglecting color evaluation entirely or reducing it to coarse categorical assessments that fail to capture their real color capabilities. Current benchmarks do not assess whether models generate colors that maintain color consistency across different contexts, or produce colors that align with human memory and expectations for familiar objects. To address this, we propose GenColorBench, the first comprehensive benchmark designed to systematically evaluate the color generation capabilities of T2I models. Unlike existing benchmarks that rely on coarse categorical assessments, our benchmark is grounded in established color naming systems, including the ISCC-NBS, and CSS3/X11, and uniquely incorporates evaluation of numerical color specifications (RGB values and hex codes) that are completely absent from existing benchmarks. With over 44K+ prompts specifically designed for color evaluation covering over 400+ colors, GenColorBench provides both the scale and specificity necessary to reveal models’ true color generation capabilities through both perceptual color evaluation and automated assessment methods. We conduct extensive evaluations of several popular image generation models and unified models using GenColorBench, revealing significant variations in color generation capabilities across different models and color specification methods. Our analysis provides insights into which color naming conventions and numerical representations are most effectively understood by current models, and identifies common failure modes in color generation tasks. The main contributions of this work are threefold: (i) We introduce GenColorBench, a large-scale benchmark containing over 44,464 prompts covering 400+ colors specifically designed to evaluate the capabilities of T2I models across five distinct color generation tasks; (ii) We provide comprehensive evaluations of state-of-the-art T2I models, analyzing their performance on precise color generation and identifying key limitations; (iii) We establish baseline performance metrics and evaluation protocols that can guide future research in improving color controllability in generative models. MAJOR FINDINGS OF OUR EVALUATION • Current models exhibit notable shortcomings in adhering to precise color specifications, underscoring the urgent need for enhanced color controllability (Table 4 ). • Model performance is tightly linked to category semantics. Categories with strong color associations (e.g. Fruits and Vegetables—yellow bananas, green grass) pose greater challenges (Fig. 2 , Fig. 3 ). • Models are better at understanding basic colors (yellow, pink, blue), while they struggle more with intermediate colors (Fig. 5 (Left)). Similarly, models favor “light” and “dark” modifiers over more nuanced ones like “-ish”, suggesting a limited grasp of subtle color variations (Fig. 5 (Right)). • Vision-language models fall short as reliable tools for color evaluation (Table 3 ). Color Evaluation Tasks Benchmark Scale Focus CNA MCC COA NCU ICA Color Evaluation Methods GenEval ( Ghosh et al. 2023 ) 553 Compositionality \checkmark \checkmark \approx \times \times Mask2Former + CLIP ViT-L/14 T2I-CompBench++ ( Huang et al. 2025 ) 6000 Compositionality \checkmark \approx \times \times \times BLIP-VQA DPG-Bench ( Hu et al. 2024 ) 1065 Prompt Adherence \checkmark \checkmark \times \times \times mPLUG-large VQA TIFA ( Hu et al. 2023 ) 1000 Faithfulness \checkmark \checkmark \times \times \times mPLUG-large VQA Commonsense-T2I ( Fu et al. 2024 ) 1000+ Reasoning \approx \approx \times \times \times self-proposed (accuracy) Winoground-T2I ( Zhu et al. 2023 ) 11,000 Compositionality \checkmark \checkmark \times \times \times Human Rating + DSG-VQA Wise ( Niu et al. 2025 ) 1000 Reasoning \approx \approx \times \times \times WiScore, Aesthetic Quality MMMG ( Luo et al. 2025 ) 4456 Disciplinary Knowledge \checkmark \checkmark \checkmark \times \times GPT/Gemini/QWEN VQA Partiprompt ( Yu et al. 2022 ) 1600 Compositionality \checkmark \checkmark \times \times \times FID OneIG-Bench ( Chang et al. 2025 ) 2440 Compositionality \times \times \times \times \times FID DrawBench ( Saharia et al. 2022 ) 200 Compositionality \checkmark \checkmark \times \times \times Human Rating EvalAlign ( Tan et al. 2024 ) 3000 Compositionality \checkmark \checkmark \times \times \times MLLM-VQA Evalmuse ( Han et al. 2024 ) 4000 Compositionality \checkmark \checkmark \times \times \times FGA-BLIP2, PN-VQA GenColorBench (Ours) 44,464 Color Understanding \checkmark \checkmark \checkmark \checkmark \checkmark VQA + Color Metrics GenColorBench-Mini (Ours) < 10K Color Understanding \checkmark \checkmark \checkmark \checkmark \checkmark VQA + Color Metrics Table 1: Overview of existing T2I evaluation benchmarks. Abbreviations for color evaluation tasks: CN = Color Name Understanding, MC = Multi-Color Composition, CO = Color–Object Association, NCU = Numeric Color Understanding, ICA = Implicit Color Association. While these benchmarks are widely adopted for assessing various aspects of T2I generation—such as compositionality, prompt adherence, and reasoning—they lack comprehensive coverage of key color understanding and evaluation tasks. GenColorBench is specifically designed to fill this gap by supporting a broad spectrum of color-related tasks. (

Read the complete text