NijiGAN: Transform What You See into Anime with Contrastive Semi-Supervised Learning and Neural Ordinary Differential Equations

Kevin Putra Santoso, Anny Yuniarti, Dwiyasa Nakula, Dimas Prihady Setyawan, Adam Haidar Azizi, Jeany Aurellia P. Dewati, Farah Dhia Fadhila, Maria T. Elvara Bumbungan · 2024

Read NijiGAN: Transform What You See into Anime with Contrastive Semi-Supervised Learning and Neural Ordinary Differential Equations by Kevin Putra Santoso, Anny Yuniarti, Dwiyasa Nakula, Dimas Prihady Setyawan, Adam Haidar Azizi, Jeany Aurellia P. Dewati, Farah Dhia Fadhila, Maria T. Elvara Bumbungan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19455.txt.

Introduction

Santoso and seven co-authors present a paper from twenty twenty-four on turning real scenes into anime with a leaner model. A prior leader needs heavy paired data from a fine-tuned generator. That pipeline is bulky and its pairs are low quality. Their model swaps residual blocks for neural differential equations. Continuous change modeling suits the style transfer task. Pseudo-paired data from the prior model supervises training. The new model uses half the parameters of its rival.

Viewer scores edge past a classic anime converter. A standard distance score beats the prior leader too. Ablation studies isolate each design choice. High-fidelity anime scenes are the output goal.

Catalog record

CREATORKevin Putra Santoso, Anny Yuniarti, Dwiyasa Nakula, Dimas Prihady Setyawan, Adam Haidar Azizi, Jeany Aurellia P. Dewati, Farah Dhia Fadhila, Maria T. Elvara Bumbungan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NijiGAN: Transform What You See into Anime with Contrastive Semi-Supervised Learning and Neural Ordinary Differential Equations Kevin Putra Santoso, Anny Yuniarti, Dwiyasa Nakula, Dimas Prihady Setyawan, Adam Haidar Azizi, Jeany Aurellia P. Dewati, Farah Dhia Fadhila, Maria T. Elvara Bumbungan Abstract Generative AI has transformed the animation industry. Several models have been developed for image-to-image translation, particularly focusing on converting real-world images into anime through unpaired translation. Scenimefy, a notable approach utilizing contrastive learning, achieves high fidelity anime scene translation by addressing limited paired data through semi-supervised training. However, it faces limitations due to its reliance on paired data from a fine-tuned StyleGAN in the anime domain, often producing low-quality datasets. Additionally, Scenimefy’s high parameter architecture presents opportunities for computational optimization. This research introduces NijiGAN, a novel model incorporating Neural Ordinary Differential Equations (NeuralODEs), which offer unique advantages in continuous transformation modeling compared to traditional residual networks. NijiGAN successfully transforms real-world scenes into high fidelity anime visuals using half of Scenimefy’s parameters. It employs pseudo-paired data generated through Scenimefy for supervised training, eliminating dependence on low-quality paired data and improving the training process. Our comprehensive evaluation includes ablation studies, qualitative, and quantitative analysis comparing NijiGAN to similar models. The testing results demonstrate that NijiGAN produces higher-quality images compared to AnimeGAN, as evidenced by a Mean Opinion Score (MOS) of 2.192, it surpasses AnimeGAN’s MOS of 2.160. Furthermore, our model achieved a Frechet Inception Distance (FID) score of 58.71, outperforming Scenimefy’s FID score of 60.32. These results demonstrate that NijiGAN achieves competitive performance against existing state-of-the-arts, especially Scenimefy as the baseline model. keywords Image-to-Image Translation, NeuralODEs, Semi-supervised Learning, Generative Artificial Intelligence † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Figure 1: Generated samples from NijiGAN . We show 1366×768 input images (top) and 1366×768 output images that is translated into anime domain (bottom). The significant advancements in Generative Artificial Intelligence (GenAI) Goodfellow et al. (2020) Karras et al. (2019) Karras et al. (2020) have driven numerous breakthroughs across creative industries, especially in the domains of art and animation. Among the various generative models, Generative Adversarial Networks (GANs) have emerged as one of the most impactful approaches Goodfellow et al. (2020) Cohen and Giryes (2023) Gui et al. (2021) . One of GANs’ strengths compared to other models such as diffusion is that GANs enable systems to generate new data that closely approximates the training data distribution with relatively lower computational cost Sharma et al. (2024) Bengesi et al. (2024) . In reality, GAN-based models have seen applications across many domains, such as image transformation, music generation, and text generation Reed et al. (2016) Zhang et al. (2022) Kim and Park (2020) . However, challenges remain in applying GANs to highly specific tasks, such as style transfer between visually disparate domains. A notable example of this is unpaired image-to-image translation Zhu et al. (2017) Liu et al. (2017) Huang et al. (2018) , where the model must learn style transfer between domains without explicitly paired data. Translating real-world landscape images into anime-style images exemplifies this challenge, given the substantial structural and textural differences between the two visual styles Jiang et al. (2023) . Various approaches have been developed to address these challenges. For example, CycleGAN employs a cycle consistency mechanism to ensure that transformed images can revert to their original domain without significant loss of information Zhu et al. (2017) . Despite its utility, this approach has limitations, particularly due to its reliance on a bijective mapping that may not be suitable for domains as distinct as real-world and anime images. Moreover, finding high-quality datasets specific to the target domain remains a major challenge, which can result in inconsistent visual quality and style in the generated images. To tackle these obstacles, Jiang et al. introduced Scenimefy Jiang et al. (2023) , a model designed to enhance anime-style transfer in real-world landscape images through a semi-supervised learning approach. Scenimefy incorporates pseudo-paired data synthesis, semantic segmentation for data selection, and semi-supervised training, yielding more accurate anime style transfer. However, Scenimefy still encounters challenges, such as reducing visual artifacts, minimizing the substantial domain gap between real-world and anime aesthetics, and balancing these improvements with model complexity and inference time. In this context, we propose NijiGAN, an enhanced model inspired by Scenimefy’s semi-supervised training framework, with several modifications to improve image quality and reduce the computational complexity on scenimefy so that translating real image to anime in real time become much possible. NijiGAN utilizes Neural Ordinary Differential Equations (NeuralODEs) as a replacement for traditional ResNet layers in Scenimefy’s generator, which aims to decrease model complexity while preserving or enhancing visual quality. In summary, our contributions to NijiGAN include: • We introduce NijiGAN, a new framework for anime style transfer that uses Neural Ordinary Differential Equations (NeuralODEs) within the CUT generator from Scenimefy. NeuralODEs layers make NijiGAN smaller and faster without sacrificing image quality, making it better suited for real-time rendering. • We compare NijiGAN with the original CUT generator to show how replacing ResNet blocks with NeuralODEs layers impacts efficiency and image quality. • To support further research in scene stylization, we also add a high-resolution anime scene dataset which aims to enhance future developments in anime-style generative modeling. 2 Related Works 2.1 Scenimefy Scenimefy tackles the challenge of turning real images into anime by handling the unique complexities of anime style, limited high-quality datasets, and the big gap between real and animated visuals. To make this work, Scenimefy blends supervised and unsupervised learning, which helps the model stay consistent by using pseudo-paired data and unpaired data, containing the anime style references. The process has three main steps: creating pseudo-paired data, filtering pseudo-paired data based on semantic segmentation, and semi-supervised training. The pseudo-paired data helps in supervised training, so the model learns to keep essential details during the transition. Semantic segmentation filters out low-quality images that do not match the needed structure. This approach allows the model to pick up both the structure of real-world images and the distinct anime style. On the early stages, the model relies more on guided training, but as it progresses, it gradually learns to handle the task with less supervision Jiang et al. (2023) . 2.2 Neural Ordinary Differential Equations NeuralODEs treat each layer of a neural network as a continuous transformation, rather than a series of discrete layers such in ResNets. This approach can reduce model complexity and, in some cases, produce smoother and more consistent outputs. By learning continuous dynamics, NeuralODEs help models generalize better, especially in tasks like image transformation where smoothness and fine detail are crucial Khrulkov et al. (2021) . In NijiGAN, NeuralODEs replace the ResNet bottleneck on Scenimefy that employs traditional CUT model. It aims to simplify the model without sacrificing quality or detail, especially when translating complex scenes from real-world landscapes to anime. 2.3 Contrastive Learning for Unpaired Image-to-image

Read the complete text