Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model

Donghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim · 2024

Read Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model by Donghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07333.txt.

Introduction

Researchers present a diffusion method that reposes a person in a photo in a paper from 2024. The task, pose-guided person image synthesis, keeps the person's look while changing their stance. Uses range from virtual try-on to sign language videos. Diffusion models do this well, but fine textures often blur and the same person can look inconsistent across poses. The team's framework, FPDM, tackles both problems directly.

A fusion step blends source appearance and target pose into one embedding. Training aligns that embedding with the true target image. The generator then works from appearance, pose, and the fused code together. Tests on fashion and sign language data sets were competitive. Extra tests confirmed the fusion step drives the gains in texture and consistency.

Catalog record

CREATORDonghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model Donghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim Abstract Pose-Guided Person Image Synthesis (PGPIS) aims to synthesize high-quality person images corresponding to target poses while preserving the appearance of the source image. Recently, PGPIS methods that use diffusion models have achieved competitive performance. Most approaches involve extracting representations of the target pose and source image and learning their relationships in the generative model’s training process. This approach makes it difficult to learn the semantic relationships between the input and target images and complicates the model structure needed to enhance generation results. To address these issues, we propose Fusion embedding for PGPIS using a Diffusion Model (FPDM). Inspired by the successful application of pre-trained CLIP models in text-to-image diffusion models, our method consists of two stages. The first stage involves training the fusion embedding of the source image and target pose to align with the target image’s embedding. In the second stage, the generative model uses this fusion embedding as a condition to generate the target image. We applied the proposed method to the benchmark datasets DeepFashion and RWTH-PHOENIX-Weather 2014T, and conducted both quantitative and qualitative evaluations, demonstrating state-of-the-art (SOTA) performance. An ablation study of the model structure showed that even a model using only the second stage achieved performance close to the other PGPIS SOTA models. The code is available at https://github.com/dhlee-work/FPDM. Introduction The generation of realistic virtual human images is an important research area in computer vision. One subfield of person image generation research, pose-guided person image synthesis (PGPIS), aims to generate a person’s image in a given pose while maintaining the appearance of the source person image (Liao et al., 2024). In practice, PGPIS can be useful for data augmentation to improve machine learning (ML) model performance, and commercially, it has a wide range of applications, including virtual reality (VR) and e-commerce (Sha et al., 2023). Since the PGPIS task was proposed by (Ma et al., 2017), research has mainly been based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2020) and Diffusion Models (Ho et al., 2020). The focus of these studies is to synthesize realistic images, and more importantly, images that correspond to the specified poses while maintaining spatial and temporal consistency. Until recently, PGPIS research has been primarily based on GANs (Liu et al., 2022; Ren et al., 2022; Tang et al., 2022; Zhang et al., 2022; Zhou et al., 2022), and the results of the generated models have been impressive. However, as mentioned by (Iglesias et al., 2023), the min-max objective for the adversarial approach is unstable, and the instability of this learning process makes it difficult to produce high-quality images. For example, the results reported by (Albahar et al., 2021) on GAN-based models show blurriness of detailed body shapes and distortion of detailed patterns on clothes. Recently, the diffusion model (Ho et al., 2020) has been actively studied in the field of image generation and has been successful in generating high-quality images. As a result, research on conditional diffusion models (Ho & Salimans, 2021)(Dhariwal & Nichol, 2021) has led to PGPIS model studies such as PIDM (Bhunia et al., 2023), PoCoLD (Han et al., 2023),CFLD(Lu et al., 2024), and PCDM(Shen et al., 2024). These studies condition the target pose and source image to produce images of a person following the target pose while maintaining the appearance of the source image, which surpasses the results of GAN-based models. However, in most approaches, the relationship between the target pose and the source image conditions is learned during the training process of the diffusion model, making it difficult to effectively transfer the semantic information from the source image to the target pose. This difficulty in reflecting conditions in the generated image complicates the model structure and impedes improvements in generation results. In this study, we propose Fusion Embedding for PGPIS with Diffusion Model (FPDM), which is inspired by using the CLIP (Radford et al., 2021) text encoder in the text-to-image diffusion model proposed by (Rombach et al., 2022). Our proposed method consists of a two-stage model. In the first stage In the first stage, the CLIP image encoder is used to obtain image-level embeddings of the source image and the target pose, and the two embeddings are fused using the Combiner module (Baldrati et al., 2023). The aggregated embedding is then trained using contrastive learning to align with the target image embedding, which is extracted using the same encoder as the source image. In the second stage, the conditional diffusion model uses the DINOv2 (Oquab et al., 2023) model to extract features from the source image and a 4-layer CNN module to extract features from the target pose image. The fusion embedding learned in the first stage is then applied. By experimenting with the proposed method on the benchmark datasets DeepFashion and RWTH-PHOENIX-Weather 2014T (Phoenix), we showed that the proposed FPDM model outperforms other SOTA models quantitatively and qualitatively. We also conducted a detailed ablation study on the model to identify an effective structure that achieves performance close to SOTA, even when the model is trained only in the second stage. Our contributions are summarized as follows: • We propose Source-Enhanced Pose Fusion to enable the generative model to produce images that are robust to variations in the condition, achieving SOTA performance on benchmark datasets. • The model trained in only the second stage can achieve performance close to SOTA, making it a useful baseline structure for future PGPIS research. • We tested the proposed model on the Phoenix sign language dataset and demonstrated its ability to generate high-quality and precise body parts, such as hands. Related Work Diffusion Model. Diffusion Model (DM) (Ho et al., 2020) are image generation models that have been actively studied in recent years and have shown remarkable performance in synthesizing high-quality images. The DM consists of a forward process and a reverse process, and through a distribution-based step-by-step generation method, it has achieved high-quality image generation performance. Diffusion models have also been studied for condition-based image generation. (Dhariwal & Nichol, 2021) introduced a classifier-guided diffusion model, Meanwhile (Ho & Salimans, 2021) proposed a classifier-free guidance approach that incorporates the generation conditions without using a classifier module. (Ho et al., 2020) showed that latent denoising is effective in the reverse process of DMs. The Latent Diffusion Model (Kingma et al., 2021) proposed a structure that performs diffusion in the latent space to reduce computational resources and increase the stability of model training and inference. Accordingly, Stable Diffusion (SD) (Rombach et al., 2022) achieved significant image generation results by reducing the input image to latent space using a pre-trained VAE and applying Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020). Additionally, SD successfully generates images using text embeddings from the pre-trained CLIP text encoder. Pose Guided Person Image Synthesis. Since (Ma et al., 2017) first presented the Pose Guided Person Image Synthesis (PGPIS) model, research has been mainly based on GANs. However, the min-max objective approach for competitive learning based on GANs is unstable, and the instability of the learning process makes it difficult to generate high-quality images. Recently, DM-based PGPIS models have been actively researched to

Read the complete text