NumColor: Precise Numeric Color Control in Text-to-Image Generation

Muhammad Atif Butt, Diego Hernández, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer · 2026

Read NumColor: Precise Numeric Color Control in Text-to-Image Generation by Muhammad Atif Butt, Diego Hernández, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13547.txt.

Introduction

Muhammad Atif Butt and co-authors present NumColor for precise numeric color control in text-to-image generation. Diffusion models fail to read hex codes and RGB values, since subword tokenization splits them into meaningless tokens. NumColor adds a Color Token Aggregator that detects color specifications regardless of tokenization. A ColorBook of 6,707 learnable embeddings maps colors into text encoder space in perceptually uniform CIE Lab terms. Trained only on FLUX, it transfers zero-shot to SD3, SD3.5, and PixArt models. Color accuracy rose 4 to 9 times across five models.

Catalog record

CREATORMuhammad Atif Butt, Diego Hernández, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71291
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NumColor: Precise Numeric Color Control in Text-to-Image Generation Muhammad Atif Butt, Diego Hernández, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer Abstract Text-to-image diffusion models excel at generating images from natural language descriptions, yet fail to interpret numerical colors such as hex codes ( #FF5733 ) and RGB values ( rgb(255,87,51) ). This limitation stems from subword tokenization, which fragments color codes into semantically meaningless tokens that text encoders cannot map to coherent color representations. We present NumColor, that enables precise numerical color control across multiple diffusion architectures. NumColor comprises two components: a Color Token Aggregator that detects color specifications regardless of tokenization, and a ColorBook containing 6,707 learnable embeddings that map colors to embedding space of text encoder in perceptually uniform CIE Lab space. We introduce two auxiliary losses, directional alignment and interpolation consistency, to enforce geometric correspondence between Lab and embedding spaces, enabling smooth color interpolation. To train the ColorBook, we construct NumColor-Data, a synthetic dataset of 500K rendered images with unambiguous color-to-pixel correspondence, eliminating the annotation ambiguity inherent in photographic datasets. Although trained solely on FLUX, NumColor transfers zero-shot to SD3, SD3.5, PixArt- α \alpha , and PixArt- Σ \Sigma without model-specific adaptation. NumColor improves numerical color accuracy by 4–9 × \times across five models, while simultaneously improving color harmony scores by 10–30 × \times on GenColorBench benchmark. 1 Introduction Large-scale diffusion models have transformed text-to-image (T2I) generation, achieving remarkable visual fidelity. However, achieving precise color control remains a challenge. Although models reliably respond to coarse color descriptions such as basic color names ( e.g . , red), their performance degrades for precise color description. The degradation is especially severe when conditioned on numeric color codes ( e.g . , \blacksquare #E34234 in hexadecimal) [ 4 ] . Because such codes are the standard for communicating color in professional design software (such as Adobe Suite, Canvas, and Blender), this limitation restricts the applicability of these models in advertising, brand-driven content creation, and product visualization, where strict color fidelity is required. In Fig. 1 (bottom-left), we provide representative failure cases in generation conditioned on numerical color codes for several models, including FLUX and SD3. A recent benchmark of T2I models [ 4 ] confirms these findings, reporting accuracy below 15% on tasks involving numerical color specifications. This stems from how standard text encoders such as CLIP [ 46 ] and T5 [ 47 ] tokenize these inputs [ 41 ] : rather than treating them as coherent semantic units representing color, they are fragmented into separate tokens ( e.g . , \blacksquare #FF5733 becomes # , FF , 57 , 33 ), which not only lack any meaningful color representation in the embedding space, but also disrupt cross-attention. As shown in Fig. 1 (top-left), each sub-token attends to arbitrary image regions rather than the target object, breaking color-object binding during generation. This is further supported by our embedding geometry analysis in Fig. 1 (right) across five text encoders. While color names preserve strong consistency with the ground-truth color space, both RGB and hexadecimal codes exhibit an approximate 60% degradation. Beyond tokenization, the problem is further exacerbated by limited exposure to numerical color descriptions in the training data of T2I models. Image captions in large-scale text–image datasets predominantly describe colors using color names (e.g., “red,” “solferino,” “navy blue”), while explicit numerical color codes are rarely used. To address the lack of training data for numerical color supervision, we introduce NumColor-Data , a large-scale synthetic dataset. It contains 500,000 rendered images of diverse objects spanning a wide range of precisely specified colors. Each image provides unambiguous color-to-pixel correspondence, eliminating lighting and material ambiguities inherent in photographic datasets. In this work, we propose NumColor , a framework that enables precise numerical color generation in T2I diffusion models. While prior work such as ColorPeel [ 5 ] has explored color prompt learning, it is limited to a small fixed set of color names and requires retraining for each new color. NumColor instead addresses the problem in a rigorous fashion by proposing a new color encoder and a codebook that allow the models to understand precise color specifications. More in detail, we address the tokenizer fragmentation by introducing a tokenizer-agnostic Color Token Aggregator that identifies numerical color spans independently of the underlying text encoder. The detected colors are then mapped to the text embedding space via a Color-based codebook, that we term ColorBook : a compact set of learnable embeddings grounded in the perceptually uniform CIE Lab color space, injected directly into the text encoder to enable proper color-object binding during generation. In summary, our main contributions are: • NumColor. A framework for precise numerical color conditioning in T2I models. It introduces a tokenizer-agnostic Color Token Aggregator that operates independently of the tokenization scheme (e.g., CLIP or T5), with ColorBook , a learnable codebook that maps colors in the sRGB gamut to the text embedding space. • NumColor-Data. The first large-scale dataset of objects with precisely numerical color descriptions. It contains over 500K images and covers the sRGB gamut, enabling clean supervision for numerical color conditioning. • Results. We evaluate NumColor on 9.8K prompts spanning 400+ colors from GenColorBench [ 4 ] . Our method achieves 4–9 \times improvement in color accuracy and 10–30 \times improvement in color harmony across FLUX, SD3, SD3.5, PixArt- \alpha , and PixArt- \Sigma . The ColorBook, trained solely on FLUX, transfers zero-shot to SD3, SD3.5, PixArt- \alpha , and PixArt- \Sigma without any model-specific adaptation. Figure 1 : Lack of Numeric Color Understanding. Left: Text encoders fragment hex codes into arbitrary subwords that attend to unrelated image regions, leading to incorrect color generation. Right: CKA [ 31 ] and neighborhood consistency (k=8) between text embeddings and Lab space show that color names preserve perceptual structure, while hex codes and RGB values degrade by \sim 60\% . NumColor recovers the structure comparable to color names. 2 Related Work 2.1 Text-based Image Generation and Editing Alongside the rapid evolution of generative AI, remarkable advancements have been achieved in T2I generation and editing over the past several years. In particular, T2I diffusion models [ 26 , 23 ] have emerged as more effective paradigms, outperforming GANs [ 22 ] , VAEs [ 30 ] , autoregressive [ 18 ] , and normalizing flows [ 14 , 15 ] in T2I generation tasks. Diffusion models are probabilistic generative frameworks that learn underlying data distributions through iterative denoising processes, starting from Gaussian noise distribution. To enhance controllability, these models can be conditioned on various signals, including class labels [ 54 ] , images [ 39 ] , or text prompts [ 42 ] . With recent scaling of diffusion models, state-of-the-art T2I models such as SD3 [ 17 ] and FLUX [ 33 ] have significantly outperformed their predecessors [ 48 , 7 , 9 ] . However, T2I diffusion models lack inherent support for text-guided image manipulation. Prompt-to-Prompt [ 25 ] pioneered controllable editing by aligning spatial layouts with text prompts via cross-attention, while InstructPix2Pix [ 3 ] extended this through instruction tuning to enable text-guided manipulation. To circumvent the need for explicit image inversion, Imagic [ 28 ] optimized text embeddings to align original image appearance with target

Read the complete text