Unsupervised content and style learning for multimodal cross-domain image translation Authors: Zhijie Lin, Jingjing Chen, Xiaolong Ma, Chao Li, Huiming Zhang, Lei Zhao Published: 2024 ABSTRACT Recently, impressive progress has been made in cross-domain image translation using image generation models pre-trained on massive amounts of data since these pre-trained generative models have strong generative capabilities.However, due to the fact that these pre-trained generative models are usually generated based on the principle of diffusion, there are often the following issues, such as color pattern distortion and the inability to maintain content structure during the translation process. Existing methods fail to meet the requirements of cross-domain image translation, where the shape (content structure) of the image should remain unchanged or undergo minimal change. To address these challenges, we propose an unsupervised content and style learning method for cross-domain image translation. Our approach utilizes self-structure attention loss to preserve content structure and color constraint loss to map the color space of the reference image to the translated image. This effectively separates the image’s content structure from its color style, enabling high-quality multi-modal cross-domain image translation. Experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches in metrics such as LPIPS, NDB, JS, IS, and excels in maintaining content structure and color patterns. Introduction The primary objective of cross-domain image translation is to develop a function that can effectively map images between the source and target domains. The translated image should maintain similar content to the original source domain image while adopting the style characteristics of the target domain. This area has attracted significant interest from researchers in the realms of deep learning and computer vision, as it finds extensive applications in style transfer1,2, image editing3–5, super resolution6, and colorization7. Previous cross-domain image translation approaches utilized paired samples to train the conditional deep neural network model8–10 or a simple regression model11,12 in a supervised fashion. However, these methods are not feasible in numerous application scenarios due to their reliance on paired data. In scenarios where paired samples are unavailable, several approaches13–19 have successfully translated images in an unsupervised manner by leveraging latent code and cycle consistency constraints. While the aforementioned methods have achieved visually realistic translation outcomes, they are only capable of generating unimodal translation results, which may not align with real-world scenarios. Given a source domain image, there exist numerous corresponding target domain images that satisfy the requirements of cross-domain image translation. To produce multimodal translation results, various methods such as MUNIT20 and DRIT21 have been proposed. These approaches typically involve designing distinct network frameworks and loss constraints to disentangle the image’s content and style, then combining the image content (from the source domain) with the style of a reference image (from the target domain) to generate diverse translation results. Different image translation tasks exhibit a wide spectrum of variations in shape and texture between the source and target domains. Image translation tasks such as photo2vangogh and photo2portrait involve minor shape alterations, whereas tasks like selfie2anime, apple2orange, and cat2dog entail more significant shape changes. While current multimodal cross-domain image translation methods have demonstrated remarkable efficacy across various image translation datasets, they often struggle to address both types of image translation tasks simultaneously. These approaches exhibit varying performance levels based on the extent of shape and texture modifications between the source and target domains. In this study, our objective is not to enhance the resilience of multimodal cross-domain image translation methods to optimize their performance across diverse image translation tasks. Instead, we aim to present a viable technical framework tailored for specific fields requiring cross-domain image translation, such as film post-production and image style editing. These specialized applications necessitate translation results that faithfully reflect the style of a reference image while maintaining a high degree of structural consistency with the source domain image. Particularly, these scenarios demand minimal shape alterations during the image translation process. For cross-domain image translation tasks of this nature, where there is little to no change in shape, current multimodal cross-domain translation approaches still exhibit the following issues: There exists a notable disparity in the content structure between the source domain image and the translated result image, which hinders meeting the application requirements.The stylistic variance between the translated result image and the reference image (from the target domain) causes the breakdown of color patterns (only certain prominent color patterns are learned), resulting in insufficient color richness.To address the above issues, drawing inspiration from the self-attention mechanism22, we present a novel approach called SSAL-GAN, with “SSAL” representing self-structure attention loss. Our self-structure attention loss ensures a high level of content consistency between the translated images and the source image. Additionally, we have devised a color loss function based on statistics to enhance the color richness of the translated images. The experimental results across multiple datasets demonstrate that our method effectively preserves the content structure of the source image and accurately maps the color space of the reference image to the translated image. In summary, our method makes the following contributions: We introduce a self-structure loss function to enhance the preservation of content structure in cross-domain image translation. This loss function effectively leverages long-range dependencies among local structures to uphold the relative relationships between microstructures, ensuring consistency in content structure between the source domain image and its corresponding translation result.Our method proposes a color loss function based on statistical analysis, primarily focusing on capturing the color information from the reference image. By enforcing constraints through this color loss function, the color distribution of the translated image aligns with that of the reference image, significantly reducing instances of color mode collapse during image translation.Our framework achieves disentangled learning of image content and style. In contrast to prevailing methods like MUNIT20, DRIT21, and DSMGAN23, our approach circumvents the need for cycle consistency loss functions and other intricate constraints. The elimination of these complex loss functions eliminates the necessity for meticulous hyperparameter tuning during training. There exists a notable disparity in the content structure between the source domain image and the translated result image, which hinders meeting the application requirements. There exists a notable disparity in the content structure between the source domain image and the translated result image, which hinders meeting the application requirements. The stylistic variance between the translated result image and the reference image (from the target domain) causes the breakdown of color patterns (only certain prominent color patterns are learned), resulting in insufficient color richness. The stylistic variance between the translated result image and the reference image (from the target domain) causes the breakdown of color patterns (only certain prominent color patterns are learned), resulting in insufficient color richness. We introduce a self-structure loss function to enhance the preservation of content structure in cross-domain image translation. This loss function effectively leverages long-range dependencies among local structures to uphold the relative relationships between microstructures, ensuring consistency in content structure between the source domain image and its corresponding translation result. We introduce a self-structure loss function to enhance the preservation of content structure in cross-domain image translation. This loss function effectively leverages long-range dependencies among local structures to uphold the relative relationships between microstructures, ensuring consistency in content structure between the source domain image and its corresponding translation result. Our method proposes a color loss function based on statistical analysis, primarily focusing on capturing the color information from the reference image. By enforcing constraints through this color loss function, the color distribution of the translated image aligns with that of the reference image, significantly reducing instances of color mode collapse during image translation. Our method proposes a color loss function based on statistical analysis, primarily focusing on capturing the color information from the reference image. By enforcing constraints through this color loss function, the color distribution of the translated image aligns with that of the reference image, significantly reducing instances of color mode collapse during image translation. Our framework achieves disentangled learning of image content and style. In contrast to prevailing methods like MUNIT20, DRIT21, and DSMGAN23, our approach circumvents the need for cycle consistency loss functions and other intricate constraints. The elimination of these complex loss functions eliminates the necessity for meticulous hyperparameter tuning during training. Our framework achieves disentangled learning of image content and style. In contrast to prevailing methods like MUNIT20, DRIT21, and DSMGAN23, our approach circumvents the need for cycle consistency loss functions and other intricate constraints. The elimination of these complex loss functions eliminates the necessity for meticulous hyperparameter tuning during training. Related work Image cross-domain translation Many image applications, such as image style transfer1,2, image editing3, image super-resolution6, and image colorization7, could be seen as translating an image into another image. Those two images usually belong to two different image domains, called source domain and target domain. A vital issue of cross-domain image translation is determining which properties of the source domain image to preserve in the translated target domain and how to hold them. Some approaches have been presented to preserve certain properties of the source domain, such as pixel gradients24, pixel values25, semantic features19, class labels24, and pairwise sample distances26. Pix2Pix8 is the first work to support different cross-domain image translation tasks using a conditional generative adversarial network. HDPix2Pix9 translates semantic label maps into high-resolution photorealistic images using conditional generative adversarial networks. Both Pix2Pix and HDPix2Pix require paired training images. However, for most applications, paired data is not available or difficult to collect. Several concurrent works, CycleGAN13, DualGAN27, DiscoGAN17, and UNIT18, realize unpaired cross-domain image translation utilizing cycle-consistency loss constraint. CouncilGAN28 utilizes duplicate generators and discriminators and takes the council loss as a constraint to replace cycle-consistency loss. However, these approaches only generate unique-modal translated results. Different network architectures and loss functions are designed to generate multimodal translation results. BicycleGAN29 combines the variational autoencoder (VAE) with conditional generative adversarial networks to generate multiple results from a single source domain image. MUNIT20 and DRIT21 disentangle the images of the source domain and the target domain into content and style using cycle consistency. DSMGAN23 maps latent features in the shared content space to domain-specific content spaces. Zheng et al.30 utilize multi-branch discriminator to learn information of objects to be translated. TransGaGa31 deals with the translation tasks with significant shape variations between domains. MSGAN32 uses a regularization term to increase the diversity of translated results. Zhao et al.33 propose adversarial consistency loss for cross-domain translation, which overcomes the drawbacks of cycle-consistency loss. U-GAT-IT34 incorporates a attention module and a learnable normalization function for unsupervised cross-domain translation. Recently, several works35,36 focus on the few-shot or one-shot unsupervised cross-domain image translation. At present, the most typical unsupervised cross-domain image translation methods, such as MUNIT20, DRIT21, and DSMGAN23, disentangle the input images into content component and style component, combine the style of the reference image (from the target domain) and the content of the source domain image to synthesize the diversity results. The cycle consistency loss is used to maintain the association between the source domain images and the translated images in certain attributes (shape). Although the methods mentioned above have achieved better translation results, for unsupervised cross-domain translation tasks that require the shape between the source domain images and the translated images to be as constant as possible in the process of translation, there are still problems of color pattern collapse and content structure preservation failure described in the section. Unlike the above methods, our method uses a specially designed self-structured loss function and a color statistical loss function to constrain the network’s learning. Our method keeps the content structure of the source image and maps the color space of the reference image to the translated image. Image style transfer There are two research lines in image style transfer, one is based on optimization, the other is based on learning. Among the optimization-based methods, the most typical one is the Gatys et al.’s method1, which mainly uses the covariance matrix of image semantic features to express image style. Other methods include MGANs37, ROTSS38, WCT39, DFP40, etc. Multiple works use a feed-forward network to learn image style transfer to reduce the optimization’s computation cost41–47. Huang et al.2 propose an adaptive instance normalization (AdaIN) layer for better aligning the means and variances of the semantic features between the content and style images for general style transfer.Fig. 1Exemplar results of our method on images of dataset ukiyo2photo13 and photo2vangogh13. The results generated by our method can keep the content of the source domain image well and have rich color information of the reference image. The style images are from target domain as the reference images, and the content images are from the source domain. Exemplar results of our method on images of dataset ukiyo2photo13 and photo2vangogh13. The results generated by our method can keep the content of the source domain image well and have rich color information of the reference image. The style images are from target domain as the reference images, and the content images are from the source domain. Arbitrary style transfer methods use unified models to handle arbitrary inputs by building feed-forward architectures. Liu et al.48 learn the spatial attention score from both shallow and deep features using an adaptive attention normalization module (AdaAttN). An et al.49 alleviate the content leak using reversible neural flows and an unbiased feature transfer module (ArtFlow). Chen et al. citechen2021artistic apply an internal-external scheme to learn feature statistics (mean and standard deviation) as style priors (IEST). Zhang et al.50 learn the style representation directly from image features via contrastive learning to achieve domain enhanced arbitrary style transfer (CAST). In addition to CNN, visual transformer has also been used for style transfer tasks. Wu et al.51 perform content-guided global style composition by a transformer-driven style composition module (StyleFormer). Deng et al.52 propose a transformer-based method (StyTr2) to avoid the biased content representation in style transfer by considering the long-range dependencies of input images. Image style transfer methods mainly focus on learning and transferring colors and brushstrokes and have difficulty on other artistic factors such as object shape and decorative elements. Neural style transfer iteratively optimizes the content image to match the style image, which is time-consuming. Alternatively, adaptive instance normalization (AdaIN) transfers the style of a source image to a target image by matching their feature statistics. In contrast, pix2pix, CycleGAN, and CUT53 use different mechanisms for content preservation. CycleGAN’s cycle consistency for content preservation is often too restrictive, while CUT maximizes mutual information between content and stylized images in a patch-based feature space. This maintains structural information while changing appearance. CLIP model54 has been shown to have semantic representative power resulting from a large-scale dataset of 400 million image and text pairs, which allows for text-driven image manipulation. StyleCLIP55 uses CLIP and pretrained StyleGAN56 to optimize the latent vector of the content input given a text prompt, but its image modification is limited to the domain of the pretrained generator. StyleGAN-NADA57 proposes an out-of-domain image manipulation method that shifts the generative model to new domains. VQGAN-CLIP58 demonstrates that VQGAN59 can also be used as apretrained generative modelto generate or edit high-quality images without training. CLIPstyler60 proposes a CNN encoder-decoder model that learns both content and style properties through patchwise CLIP loss, allowing for image generation and manipulation beyond the domains of pretrained generators. Image style transfer61 can be traced back to the non-realistic rendering algorithm, which originated in the mid-1990s. Non-realistic rendering is one of the two main issues that first emerged in computer graphics. Unlike realistic rendering, the purpose of non-realistic rendering is mainly to simulate artistic rendering styles and create a hand-painted effect. Common non-realistic rendering techniques include cartoon rendering, oil painting rendering, pixel rendering, pencil drawing, sketch drawing, crayon painting, and ink painting. This type of algorithm focuses on simulating a special brush texture and does not extend to the transfer of the overall painting style. The parameterized method of stroke simulation can better produce images with the desired stroke style based on the line draft drawn by the user. However, such methods are greatly influenced by a single style and are difficult to extend to other styles or higher-level semantic styles. In contrast, style transfer involves transforming the color matching and artistic style of the style image to the content image. He et al.62 proposed a review on makeup transfer (MT), which aims to transfer makeup styles from reference images to source images while preserving identity and background information. The generated image includes both the style of the style image and the object of the content image. Recently, image-to-image translation63–65 methods have achieved significant progresses. These methods learn to generate stylized images from the style domain. However, the Per-Style Per-Model method cannot meet the needs of arbitrary style transfer. GAN can learn the statistical information of the style domain, and it has achieved great success in eliminating the artifact of the generated image66. proposed an iterative architecture for Restorable Arbitrary Style Transfer (RAST) to address the content leak issue in arbitrary style transfer. RAST achieves the transmission of both content and style information through multi-restorations, controlling the content-style balance in stylized images by the accuracy of image restoration. To ensure the effectiveness of the proposed architecture, two novel loss functions, multi-restoration loss and style difference loss, are designed. A new quantitative evaluation method is also proposed to measure content preservation and style embedding performance. Garg et al.67 proposed a steganographic technique that uses the Neural Style Transfer (NST) algorithm to generate a stego image with maximum capacity payload while maintaining perceptual quality. The proposed method also involves destylization using conditional Generative Adversarial Networks (cGANs) to recover the secret content from the stego image with minimal distortion. The destyling GAN is forced to learn the embedded secret information using a loss function that learns the same representation as in the embedding NST algorithm. The proposed framework was evaluated on the Imagenet dataset and tested on the PASCAL VOC12 dataset, and it outperformed in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Visual Information Fidelity (VIF). We used a network architecture similar to the AdaIN2 for cross-domain image translation in this paper. Several cross-domain image translation approaches23,43 are also used for image style transfer.Fig. 2Overview of our network architecture. The whole network consists of five modules: content encoder module, style encoder module, AdaIN layer module, decoder module and discriminator module. Overview of our network architecture. The whole network consists of five modules: content encoder module, style encoder module, AdaIN layer module, decoder module and discriminator module. Our model Our goal is to learn a mapping function that translates images from the source domain to the target domain and generates diverse and multimodal outputs. Figure 1 shows that our method generates results that both preserve the content of the source domain image well and have rich color information of the reference image. The shape attributes of the input source images should be changed as little as possible during the cross-domain image translation. To better understand the rest of this article, we define the following symbols. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_S$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_T$$\end{document} be the source and target domains, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s \in D_S$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_t \in D_T$$\end{document} be images of the source domain and the target domain, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_r \in D_T$$\end{document} be reference images from the target domain, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_g \in D_T$$\end{document} be the translated images. Similar to MUNIT20, DRIT21,GP-UNIT68 and DSMGAN23, our network model takes the source domain image and the reference image from the target domain as the input and outputs the translated image. The source domain image mainly provides content information, and the reference image provides style information. Figure 2 depicts the basic framework of our approach. The whole network consists of five modules: content encoder module, style encoder module, AdaIN layer module, decoder module and discriminator module. The decoder could be pre-trained diffusion module or network module being trained using our training strategy. We mainly focus on meeting the requirements of the image cross-domain translation tasks requiring that the shape (content structure) of the image should be unchanged or changed as little as possible in the process of translation. To preserve the content structure of the source domain image during the image cross-domain translation, we design a novel self-structure attention loss function, which can fully consider the long-range dependence of the source domain image in the semantic feature space and transfer this dependence to the translated image. The semantic feature reconstruction loss is used to further enhance the structure preservation of the source domain image content. We design a color statistical loss function to project the color space of the reference image to the translated image. The details of our approach are described below. Adversarial loss Like MUNIT20, we employ the adversarial loss of LSGAN69 to minimize the discrepancy between the distributions of the real target domain images and the translated images.1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{adv} =&\mathop {\min }_U \mathop {\max }_D ({\mathbb {E}}_{x_t \sim p_{t}} {\log D(x_t)} \\ &+ {\mathbb {E}}_{x_s \sim p_{s},x_r \sim p_{t}} \log (1-D(U(x_s,x_r))) ) \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{t}$$\end{document} are the distribution of the source domain images and the target domain images, respectively, D is the discriminator, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U(\cdot )$$\end{document} is the function expressed by our network (SSAL-GAN). Semantic feature reconstruction loss Our network translates the source domain images into the target domain images in an unsupervised way. However, the source domain image is different from the corresponding translated image in pixel level. It is desired that the source domain image is the same as the corresponding translated image in semantic feature space, , which indicates that they are highly consistent in content structure. So the semantic feature reconstruction loss is defined as2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{sfr} = {\mathbb {E}}_{x_s \sim P_{s},x_r \sim P_{t}} \Vert VGG(x_s) - VGG(G(E_1(x_s),E_2(x_r)))\Vert _2 \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_r$$\end{document} is the reference image from the target domain, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} is the source domain image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{t}$$\end{document} are the distribution of the source domain images and the target domain images, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_1$$\end{document} is the content encoder module, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_2$$\end{document} is the style encoder module, G is the decoder module. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$VGG(\cdot )$$\end{document} denotes the function expressed by pre-trained VGG16 network. We take \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Relu4\_1$$\end{document} of VGG-1670 to extract feature of the image. ltion*Color statistic Loss In order to make the style of the translated image consistent with the reference image, we define the color statistical loss function from semantic features and pixel space. We hope to calculate the Relaxed Earth Mover Distance (REMD)71 between the semantic features of the reference image and that of the translated image. When this distance is smallest, these two images have the same style. The color statistical loss used in our method is calculated in the feature space of the image, not in the RGB space of the image. To this end, we first use VGG16 to extract image features. In order to better integrate multiple feature scales, our method uses several features such as RELU3, RELU4, and RELU5 to scale the features of different scales to a unified scale, and then calculate the color statistical loss.3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{cs}^f =&\mathop {max} \left( \frac{1}{n} \sum _{i}\mathop {min}_j \left( 1-\frac{A_i \cdot B_j}{\parallel A_i \parallel \parallel B_j \parallel }\right) ,\right. \\&\left. \frac{1}{n} \sum _{j}\mathop {min}_i\left( 1-\frac{A_i \cdot B_j}{\parallel A_i \parallel \parallel B_j \parallel }\right) \right) \end{aligned} \end{aligned}$$\end{document}where n is the number of feature vectors extracted from source domain image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_i$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B_j$$\end{document} denote feature vector from features set of source domain image and translated image, respectively. The cosine distance used in Equation 3 ignores the magnitude of the feature vectors, which leads to visual artifacts in the translated output. To overcome this we add a statistic matching loss:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{cs}^s =&\frac{1}{d} \parallel \mu _A - \mu _B \parallel _1 + \frac{1}{d^2} \parallel \Sigma _A - \Sigma _B \parallel _1 \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _A$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _B$$\end{document} are the mean of the feature vectors extracted from source domain image and translated image, respectively, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _A$$\end{document},\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _B$$\end{document} are the covarianceare. We define a loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathscr {L}}_{cs}^p$$\end{document} in the pixel space, which is similar to loss \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathscr {L}}_{cs}^f$$\end{document}. The total color statistic loss is following:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathscr {L}}_{cs} = {\mathscr {L}}_{cs}^f + {\mathscr {L}}_{cs}^s + {\mathscr {L}}_{cs}^p \end{aligned}$$\end{document}Fig. 3Comparisons on monet2photo with MUNIT20, DRIT21.Fig. 4Comparisons on summer2winter with MUNIT20, DRIT21.Fig. 5Comparisons on weather dataset with DSMGAN23, DRIT21. Comparisons on monet2photo with MUNIT20, DRIT21. Comparisons on summer2winter with MUNIT20, DRIT21. Comparisons on weather dataset with DSMGAN23, DRIT21. Table 1Quantitative comparison with the state-of-the-art approaches on summer2winter dataset.Method \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document} DRIT71.820.122532.720.0932.76MUNIT79.060.090831.790.0922.79StyTr269.240.101130.120.0882.65GP-UNIT79.720.130833.210.0922.92Oig79.970.143131.320.0912.84SuperStarGAN80.720.150833.790.0932.89SSAL-GAN78.910.144733.130.0902.77SSAL-Diffusion 80.88 0.1622 34.68 0.108 3.08 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 2Quantitative comparison with the state-of-the-art approaches on monet2photo dataset.Method \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document} DRIT115.010.169115.660.123.25DSMGAN127.670.161317.460.163.68StyTr2136.010.169114.060.183.08GP-UNIT107.070.161319.960.113.96SuperStarGAN105.720.180822.010.113.92Oig110.970.193121.320.103.84SSAL-GAN116.110.215619.340.133.88SSAL-Diffusion 96.98 0.2156 22.28 0.09 4.12 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 1Quantitative comparison with the state-of-the-art approaches on summer2winter dataset.Method \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document} DRIT71.820.122532.720.0932.76MUNIT79.060.090831.790.0922.79StyTr269.240.101130.120.0882.65GP-UNIT79.720.130833.210.0922.92Oig79.970.143131.320.0912.84SuperStarGAN80.720.150833.790.0932.89SSAL-GAN78.910.144733.130.0902.77SSAL-Diffusion 80.88 0.1622 34.68 0.108 3.08 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Quantitative comparison with the state-of-the-art approaches on summer2winter dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 2Quantitative comparison with the state-of-the-art approaches on monet2photo dataset.Method \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document} DRIT115.010.169115.660.123.25DSMGAN127.670.161317.460.163.68StyTr2136.010.169114.060.183.08GP-UNIT107.070.161319.960.113.96SuperStarGAN105.720.180822.010.113.92Oig110.970.193121.320.103.84SSAL-GAN116.110.215619.340.133.88SSAL-Diffusion 96.98 0.2156 22.28 0.09 4.12 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Quantitative comparison with the state-of-the-art approaches on monet2photo dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Self-structure attention loss The distance relationship between the local structure (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3\times 3$$\end{document} image patch) and all other local structures in an image constitutes a self-structure attention map. This map can reflect the long-range dependence between the local structures of the image in the pixel space. This relationship can keep the shape and microstructure of the image unchanged. This feature is very important in cross-domain image translation tasks, especially for tasks that require the image shape to remain unchanged or change as little as possible. We first extract patches (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3\times 3$$\end{document}) in source domain image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} and translated image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_g$$\end{document}, then calculate self-structure attention maps of the two images, respectively. We take the source domain image as an example to illustrate how to calculate the self-structure attention map. The calculation of the self-structure attention map of the translated image is also similar. The feature extraction method used to calculate self structural similarity is the same as the color statistical loss feature. We measure the distance of two image patches with normalized inner product (cosine similarity):6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{x,y,x^{'},y^{'}} = \langle \frac{p_{x,y}}{\parallel p_{x,y} \parallel }, \frac{ p_{x^{'},y^{'}}}{\parallel p_{x^{'},y^{'}} \parallel } \rangle \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{x,y,x^{'},y^{'}}$$\end{document} denotes the similarity of the patch \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{(x^{'},y^{'})}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{x,y}$$\end{document} centered in position \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(x^{'},y^{'})$$\end{document} and positon (x, y) in the image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document}. The number of patches extracted from the iamge \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} is N, where N is equal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H\times W$$\end{document} when extracting patches using stride 1. We do convolution operation on the image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} with N convolution filters (extracted from the image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document}) to generate a new image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ms}$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ms}$$\end{document} is a four-dimensional tensor, i.e. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ms}\in R^{H\times W\times H \times W}$$\end{document}, and reshape it to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^{N\times N}$$\end{document}. Then apply a softmax layer to calculate (the self-structure attention map) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A\in R^{N\times N}$$\end{document}:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{ji} = \frac{\langle {p_{i}\cdot p_{j}} \rangle }{ \sum _{i=1}^{N}{\langle p_{i}\cdot p_{j}\rangle } } \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{ji}$$\end{document} measures the similarity of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i^{th}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j^{th}$$\end{document} patch. We define our self-structure attention loss as:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{sa}(x_s,x_t,x_g) = {\mathbb {E}}_{x_s \sim P_{s},x_t \sim P_{t}}\parallel A_s-A_g \parallel _1 \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_r$$\end{document} is the reference image from the target domain, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} is the source domain image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_g$$\end{document} is the translated image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{t}$$\end{document} are the distribution of the source domain images and the target domain images, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_s$$\end{document},\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_g$$\end{document} denote the self-structure attention map of the source domain image and the translated image. This constrains the content structure of the output image, without enforcing any loss directly connected to pixels of the content image. This causes the semantics and spatial layout to be broadly preserved, while allowing pixel values in source domain image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_s$$\end{document} to drastically differ from those in the translated image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_g$$\end{document} . Total loss The total loss function of our net (SSAL-GAN) is defined as:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} {\mathscr {L}}_{total} = \lambda _{adv} {\mathscr {L}}_{adv} + \lambda _{sfr}{\mathscr {L}}_{sfr} + \lambda _{cs}{\mathscr {L}}_{cs} + \lambda _{sa} {\mathscr {L}}_{sa} \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{adv}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{sfr}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{cs}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{sa}$$\end{document} are used to balance the effects among adversarial training, content structure, style consistency. Training details For all the experiments, we used Adam optimiser with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _1$$\end{document} = 0.9 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _2$$\end{document} = 0.999. The batch size was set to 1. All models were trained with a learning rate of 0.0001 and the learning rate dropped by a factor of 0.00005 after every one iterations. We set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{adv}=0.125$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{sfr}=1$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{cs}=0.0125$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{sa}=0.01$$\end{document}. We use images of resolution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$256 \times 256$$\end{document} for all dataset. Experiments Experimental settings We evaluated SSAL-GAN on six different datasets, Yosemite13, monet2photo13, monet2photo13, ukiyoe2photo13, photo2vangogh13, and weather data72. The Monet2Photo dataset consists of 1193 Monet paintings and 7038 natural photos, each with a resolution of 1024x1024 pixels The photo2vangogh dataset contains 7838 JPEG format images, each with a size of 512x512 pixels The Summer2InterYosemite dataset consists of 1540 summer photos and 1200 winter photos, with each photo divided into training and testing subsets, and a photo resolution of 1024x1024. The Ukiyoe2Photo dataset has 5209 images. This dataset contains 5209 facial images from ukiyo-e photographs, with image pixels of 1024x1024 in JPEG format. The weather dataset contains four categories of weather images, totaling approximately 20000 images. In addition, there is a dataset containing 60000 images of six weather conditions, classified as sunny, cloudy, rainy, snowy, foggy, and thunderstorm, with 10000 images for each weather condition. For better comparison, the images in the above dataset were uniformly preprocessed to 512x512 resolution before training. To evaluate the quality of the translated images, we use FID73 to measure the distance between the translated distribution and the real one through features extracted by Inception Network. To measure how faithful the translated image match corresponding source domain image in the content structure, LPIPS29 with ImageNet pre-trained VGG-16 is used. LPIPS is popularly used for assessing the similarity between two images by considering the perceptual similarity. To measure the similarity between the distribution between real images and translated one, we adopt two new metrics, NDB and JSD, presented in74. We compare with the following baselines: MUNIT20, DRIT21, GP-UNIT68, StyTr252, Oig75,SuperStarGAN76 and DSMGAN23. Qualitative comparison We compare our model with existing approaches on six different datasets. As shown in Figs. 3, 4, 5. From these figures, we can see that the image translated by our model is highly consistent with the input source domain image in content. In addition, the generated image is richer in color, while other methods, such as MUNIT20, DRIT21, GP-UNIT68, StyTr252, Oig75,SuperStarGAN76 and DSMGAN23, only learn the salient color. Although the results produced by them look brighter or more prominent, the translated images suffer from color mode collaps, that is to say, the color in the source domain image disappears in the translated image.Table 3Quantitative comparison with the state-of-the-art approaches on photo2vangogh dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}StyTr2250.140.116617.240.8875.10GP-UNIT160.140.146621.240.10275.35DRIT206.140.126618.240.0975.24MUNIT196.010.134320.750.0955.13SSAL-GAN-1175.330.150316.620.0925.04SSAL-GAN-3186.010.144317.520.0945.02SSAL-GAN without SSAL172.010.104322.750.0954.13SSAL-GAN175.330.150316.620.0925.04SSAL-Diffuision120.080.170325.620.1225.64\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold.Table 4Quantitative comparison with the state-of-the-art approaches on weather dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}StyTr2125.760.069439.780.1325.48GP-UNIT103.790.080143.240.1285.99DRIT105.760.089441.350.1325.64MUNIT113.790.070142.910.1285.87SSAL-GAN-1106.910.101739.450.1255.67SSAL-GAN-3110.010.114320.750.0955.13SSAL-GAN without SSAL196.010.084343.750.1424.13SSAL-GAN106.910.101739.450.1255.67SSAL-Diffusion96.910.133746.880.1056.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold.Table 5Ablation study on weather dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}SSAL-GAN without CS116.230.111739.450.1415.67SSAL-GAN withtout SA112.010.112338.240.1355.13SSAL-GAN106.010.121440.720.1295.93SSAL-Diffusion without CS104.010.114343.710.1125.98SSAL-Diffusion withtout SA107.240.112339.450.1185.96SSAL-Diffusion96.910.133746.880.1056.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 3Quantitative comparison with the state-of-the-art approaches on photo2vangogh dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}StyTr2250.140.116617.240.8875.10GP-UNIT160.140.146621.240.10275.35DRIT206.140.126618.240.0975.24MUNIT196.010.134320.750.0955.13SSAL-GAN-1175.330.150316.620.0925.04SSAL-GAN-3186.010.144317.520.0945.02SSAL-GAN without SSAL172.010.104322.750.0954.13SSAL-GAN175.330.150316.620.0925.04SSAL-Diffuision120.080.170325.620.1225.64\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Quantitative comparison with the state-of-the-art approaches on photo2vangogh dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 4Quantitative comparison with the state-of-the-art approaches on weather dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}StyTr2125.760.069439.780.1325.48GP-UNIT103.790.080143.240.1285.99DRIT105.760.089441.350.1325.64MUNIT113.790.070142.910.1285.87SSAL-GAN-1106.910.101739.450.1255.67SSAL-GAN-3110.010.114320.750.0955.13SSAL-GAN without SSAL196.010.084343.750.1424.13SSAL-GAN106.910.101739.450.1255.67SSAL-Diffusion96.910.133746.880.1056.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Quantitative comparison with the state-of-the-art approaches on weather dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Table 5Ablation study on weather dataset.Method\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {FID}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {LPIPS}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {NDB}^\uplus$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {JSD}^\dagger$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {IS}^\uplus$$\end{document}SSAL-GAN without CS116.230.111739.450.1415.67SSAL-GAN withtout SA112.010.112338.240.1355.13SSAL-GAN106.010.121440.720.1295.93SSAL-Diffusion without CS104.010.114343.710.1125.98SSAL-Diffusion withtout SA107.240.112339.450.1185.96SSAL-Diffusion96.910.133746.880.1056.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Ablation study on weather dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dagger$$\end{document} Lower is better. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uplus$$\end{document} Higher is better. Significant values are in bold. Quantitative comparison One hundred content images and fifty reference images were randomly selected to generate 5000 images. We calculate the FID with the training set, repeat ten times, and take the average value. To calculate LPIPS value, one hundred content images and fifty reference images are randomly selected in the test set, and fifty images are generated for each content image. The distance between the two images is calculated and the average value is taken. To evaluate NDB and JSD74 value, we take the whole training set as the clustering reference, produce five thousand images for testing, calculate NDB and JSD ten times, and take the average value. For calculating the value of IS, we randomly select one hundred content images and fifty reference images to generate five thousand images. The value of IS is calculated ten times, and the average value was taken. As can be seen from the Tables 1, 2, 3, and 4, our model performs well on this dataset photo2vangogh and weather. In the other two data set (summer2winter and monet2photo), however, the metrics are comparable to the existing state-of-art methods. Ablation studies Different stride size in image packet extraction We compare the two cases of stride size 1 and 3. When the stride size is 1, we call the model SSAL-GAN-1; when the step size is 3, we call the model SSAL-GAN-3. Tables 3, 4 indicates that the size of step size has little effect on the model, but the training time of network model will be relatively shortened when the step size is large. With and without self-structure attention loss We train the model without self-structure attention loss (called SSAL-GAN without SSAL) on photo2vangogh dataset and weather dataset. Tables 3, 4 indicates that self-structure attention loss significantly improves the quality of translated images. With and without color stastic attention loss We train the model without color stastic attention loss, as shown in Table 5. We found that the model can not successfully generate a visual and realistic result image without color stastic attention loss. With and without pre-trained diffusion model We train the model without pre-trained diffusion model. We found that our model with pre-trained diffusion can generate high-quality images.To verify the effectiveness of the proposed module, we conducted ablation experiments on color statistical loss and self attention similarity loss. The specific results are shown in the table. From the table, it can be seen that the decoder using GAN structure is not as effective as the encoder using diffusion structure. At the same time, the effectiveness of color statistics loss and self attention loss can also be seen, because when these losses are removed, the performance of the method will significantly decrease. Combining two losses together will yield better results. We believe that color statistical loss mainly constrains the color space of two domains to have a certain correspondence, reducing the problem of color pattern collapse. Without the constraint of this loss function, it may result in fewer generated color patterns. In addition, self attention loss mainly considers the relationship between image contents, which is actually the relative relationship of content structure. Once the color of the image changes, the details of the content may also be changed. However, if the relative relationship between contents does not change, at least from a visual perspective, the content structure has not changed. Limitations and discussion For unsupervised cross-domain translation tasks that require the shape between the source domain images and the translated images to be as constant as possible in the process of translation, we propose a novel framework which includes a specially designed self-structured loss function and a color statistical loss function to constrain the network’s learning. The results generated by our method can keep the content of the source domain image well and have rich color information of the reference image. However, due to the strict pixel-level constraint of self-structure attention loss, our method cannot perform shape changes, remove large objects, or ignore irrelevant texture.