Automatic Image Colorization with Convolutional Neural Networks and Generative Adversarial Networks

Changyuan Qiu Hangrui Cao Qihan Ren Ruiyu Li Yuqing Qiu University of Michigan {peterqiu,hangrui,qihanren,ruiyuli,qyuqing}@umich.edu · 2025

Read Automatic Image Colorization with Convolutional Neural Networks and Generative Adversarial Networks by Changyuan Qiu Hangrui Cao Qihan Ren Ruiyu Li Yuqing Qiu University of Michigan {peterqiu,hangrui,qihanren,ruiyuli,qyuqing}@umich.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05068.txt.

Introduction

Changyuan Qiu and co-authors explore automatic colorization of gray images with neural networks. Their twenty twenty-five project joins convolutional networks with generative adversarial networks. Color is hard to restore since two of three image channels are lost in a gray photo.

Scene meaning gives strong hints: sky tends to blue, clouds to white, and grass to green. Any color photo can serve as training data, so examples abound. Early work framed color as a plain regression, which misses that many colorings can be right. The adversarial setup lets the model pick vivid, plausible colors. Uses run from photo restoration to animation coloring done by machine.

Catalog record

CREATORChangyuan Qiu Hangrui Cao Qihan Ren Ruiyu Li Yuqing Qiu University of Michigan {peterqiu,hangrui,qihanren,ruiyuli,qyuqing}@umich.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Image Colorization with Convolutional Neural Networks and Generative Adversarial Networks Changyuan Qiu Hangrui Cao Qihan Ren Ruiyu Li Yuqing Qiu University of Michigan {peterqiu,hangrui,qihanren,ruiyuli,qyuqing}@umich.edu Abstract Figure 1: Colorization results on CIFAR10 1 Introduction Image colorization, the task of adding colors to grayscale images, has been the focus of significant research efforts in computer vision in recent years for its various application areas such as color restoration and automatic animation colorization [ 15 , 1 ] . The colorization problem is challenging as it is highly ill-posed with two out of three image dimensions lost, resulting in large degrees of freedom. However, semantics of the scene as well as the surface texture could provide important cues for colors: the sky is typically blue, the clouds are typically white and the grass is typically green, and there are huge amounts of training data available for learning such priors since any colored image could serve as a training data point [ 20 ] . Colorization is initially formulated as a regression task [ 5 ] , which ignores the multi-modal nature of color prediction. In this project, we explore automatic image colorization via classification and adversarial learning. We will build our models on prior works, apply modifications for our specific scenario and make comparisons. 2 Related Works Recently, deep learning techniques progressed notably for image colorization, and the fully-automatic colorization task (which does not take interactive input compared with the Scribble-based user-guided one) is commonly tackled with 3 approaches: regression, classification and adversarial learning [ 1 , 5 , 20 , 10 , 15 ] . [ 5 ] takes the lead in investigating fully-automatic colorization with deep neural networks (DNNs) and formulates image colorization as a regression problem: the DNN takes the extracted feature descriptors at each pixel as input and outputs the continuous values of the colored channels at the corresponding pixel, with the loss function to be the Mean Square Error (MSE) between the predicted colored channel values and the ground truth. [ 20 ] first formulates the task as a classification problem by quantizing the ab channels of the CIE Lab color space into 313 discrete ab pairs and using the multinomial cross entropy loss to train the model. [ 10 , 15 ] uses conditional Generative Adversarial Networks (cGANs) for automatic colorization of grayscale images. 3 Approach We summarize the overall objective of colorization as follows: given a single lightness channel of an image X\in\mathbb{R}^{H\times W\times 1} , predict the two corresponding color channels \hat{Y}=\mathcal{F}(X)\in\mathbb{R}^{H\times W\times 2} , where \mathcal{F} is the mapping function to be learned. Figure 2: Network architecture for classification-based colorization. Each Conv block is composed of 2 or 3 repeated Conv and ReLU layers. All spatial downsamplings in the network are achieved by Conv layer with stride greater than 1. We have two variants for this architecture w.r.t the output layer. Variant 1: we upsample the ab probability distribution to the original size, and calculate loss w.r.t the quantized ground truth. The upsampling methods include bilinear interpolation and ConvTranspose. Variant 2: we keep the ab probability distribution unchanged ( Id in the figure) , but downsample the quantized ground truth to make the size match. Figure 3: Network architecture for generator (U-Net) of GAN. The symmetric architecture consists of left contracting path and right expansive path. For the green contracting path (encoder), each block is a 4\times 4 convolutional layer with stride 2 for down-sampling, followed by batch normalization and Leaky-ReLU activation function with slope 0.2. For the right expansive path (decoder), each block consists of a 4\times 4 transposed convolutional layer with stride 2 for up-sampling and concatenation with the mirroring layer from the contracting path. Then, the concatenated block goes through a 3\times 3 convolution layer with stride 1 for halving channels, followed by batch normalization and ReLU activation function. The output layer is a 1\times 1 convolution layer followed by a Tanh activation function. 3.1 Classification-based Colorization In this approach, we treat the colorization task as a classification problem instead of a regression problem due to the ambiguity of colorization. We employ an architecture similar to U-Net [ 18 ] without skip-connection and with dilated convolutions following [ 20 , 4 , 19 ] . Details of our architecture are shown in Figure 2 with two variants. To formalize, the ab color space is quantized to bins of size 10, and yields a total of 313 possible ab pairs. Our network maps the input X to a probability distribution \hat{Z}\in[0,1]^{H\times W\times Q} , where Q=313 is the number of quantized ab pairs. Our loss function is formulated as follows. L_{cl}(\hat{Z},Z)=-\sum_{h,w}\sum_{q}Z_{h,w,q}\log(\hat{Z}_{h,w,q}) where \hat{Z} is the predicted probability distribution and Z is the quantized ground truth, using the soft-encoding scheme in [ 20 ] . Therefore, the network is not strictly end-to-end learned. To obtain the final colorized image, we first map the predicted probability distribution \hat{Z} to color channels \hat{Y} by function \hat{Y}=\mathcal{H}(\hat{Z}) , and then concatenate \hat{Y} to the input lightness channel X . For function \mathcal{H} , we use the annealed-mean operation in [ 20 ] with temperature T=0.38 . Unlike [ 20 ] , we did not apply class rebalancing. The technique is originally used to correct the bias towards lower ab values, but we empirically observed that it disrupted the training process in our case. 3.2 GAN-based Colorization Generative Adversarial Networks (GAN) [ 7 ] are composed of two competing neural network models. For this colorization problem setting, the generator takes grayscale images and generates colorized versions; the discriminator gets colored images either from the generator or the labels, concatenated with the grayscale images, and tries to identify which pair contains the real colored image [ 17 ] . Similar to the approach in [ 17 ] , we utilize deep convolutional neural networks as generative models for our adversarial framework. Since for the colorization problem setting, the input is grayscale images instead of random noises, we employ a conditional GAN instead of the traditional one. The cost functions are formulated as follows. \displaystyle\min_{\theta_{G}}J^{(G)}\left(\theta_{D},\theta_{G}\right)=\min_{\theta_{G}}-\mathbb{E}_{z}\left[\log\left(D\left(G\left(\mathbf{0}_{z}|x\right)\right)\right)\right]+\lambda\left\|G\left(\mathbf{0}_{z}|x\right)-y\right\|_{1} \displaystyle\max_{\theta_{D}}J^{(D)}\left(\theta_{D},\theta_{G}\right)=\max_{\theta_{D}}\left(\mathbb{E}_{y}[\log(D(y|x))]+\mathbb{E}_{z}[\log\left(1-D\left(G\left(\mathbf{0}_{z}|x\right)|x\right)\right)]\right) where G\left(\mathbf{0}_{z}|x\right) is the colorized image produced by the generator, with input as zero noise \mathbf{0}_{z} with the grayscale image x as a prior and y is the ground truth. We build up and train a Conditional Deep Convolutional Generative Adversarial Network (C-DCGAN) following [ 15 ] . We employ a modified U-Net [ 18 ] for our basic architecture. For the generator G , it is constructed as the modified U-Net model as shown in Figure 3 . For the discriminator D , it only utilizes the contracting part (encoder) in the U-Net model, with the number of channels being doubled after each down-sampling. The output layer is a 4\times 4 convolutional layer with stride 1, which generates a 1 dimensional output. Finally, the sigmoid activation function is used to map the output to a probability of the input image being real. To better control the color space, we separate the brightness channel and color channels using the CIE Lab color space, where we only need to predict two color channels ab in the generator. 4 Experiment Dataset We evaluate our models on the canonical dataset CIFAR-10 [ 13 ] .

Read the complete text