Real-Time Person Image Synthesis Using a Flow Matching Model

Jiwoo Jeong, Kirok Kim, Wooju Kim, Nam-Joon Kim · 2025

Read Real-Time Person Image Synthesis Using a Flow Matching Model by Jiwoo Jeong, Kirok Kim, Wooju Kim, Nam-Joon Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03562.txt.

Introduction

Jiwoo Jeong, Kirok Kim, Wooju Kim, and Nam-Joon Kim build a fast maker of person images set to a target pose. The task takes a source image of a person and a pose, and paints the same person in that pose. It serves sign language clips, games, and live streams, where the next frame must come at once. Slow paint tools with fine output have blocked such live use.

Their maker rests on flow matching, a newer paint path that trains and samples with more speed and calm. It also works in a shrunk latent space, which trims cost. On a known fashion test set, it ran at near live speed with output near the best slow tools. The trade was a small dip in fidelity for more than twice the speed. The team calls that a fair swap for live work.

Catalog record

CREATORJiwoo Jeong, Kirok Kim, Wooju Kim, Nam-Joon Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03562

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time Person Image Synthesis Using a Flow Matching Model Jiwoo Jeong, Kirok Kim, Wooju Kim, Nam-Joon Kim Abstract Pose-Guided Person Image Synthesis (PGPIS) generates realistic person images conditioned on a target pose and a source image. This task plays a key role in various real-world applications, such as sign language video generation, AR/VR, gaming, and live streaming. In these scenarios, real-time PGPIS is critical for providing immediate visual feedback and maintaining user immersion.However, achieving real-time performance remains a significant challenge due to the complexity of synthesizing high-fidelity images from diverse and dynamic human poses. Recent diffusion-based methods have shown impressive image quality in PGPIS, but their slow sampling speeds hinder deployment in time-sensitive applications. This latency is particularly problematic in tasks like generating sign language videos during live broadcasts, where rapid image updates are required. Moreover, AR/VR, gaming, and live streaming necessitate real-time responsiveness. Therefore, developing a fast and reliable PGPIS model is a crucial step toward enabling real-time interactive systems. To address this challenge, we propose a generative model based on flow matching (FM). Unlike diffusion models, FM employs deterministic updates and avoids the injection of noise in intermediate steps. As a result, our approach enables faster, more stable, and more efficient training and sampling. Furthermore, the proposed model supports conditional generation and can operate in latent space, making it especially suitable for real-time PGPIS applications where both speed and quality are critical. We evaluate our proposed method, Real-Time Person Image Synthesis Using a Flow Matching Model (RPFM), on the widely used DeepFashion dataset for PGPIS tasks. Our results show that RPFM achieves near-real-time sampling speeds while maintaining performance comparable to the state-of-the-art X-MDPT model. Our methodology trades off a slight, acceptable decrease in generated-image accuracy for over a twofold increase in generation speed, thereby ensuring real‑time performance. At 256×256 resolution with a batch size of 8, the fastest model to date—X‑MDPT—requires 1.191s even in its smallest “small” configuration, whereas our medium‑size model completes generation in just 0.489s. In terms of image quality at 512×512 resolution, X‑MDPT achieves an FID of 7.162, an LPIPS of 0.1645, and an SSIM of 0.7522; by contrast, our medium‑size model records an FID of 8.522, an LPIPS of 0.1786, and an SSIM of 0.7742—overall comparable metrics, with an even higher SSIM. Code is available at https://github.com/SONNY2020-c/RPFM-official-code . 1 Introduction The creation of realistic virtual human images constitutes a pivotal area of research within the domains of computer vision and generative models. Among these, Pose-Guided Person Image Synthesis (PGPIS) is particularly focused on generating images that preserve the visual characteristics of a source image while adapting it to a specified target pose [ 14 ] . This technique finds applications in various fields, including virtual reality (VR) and e-commerce [ 28 ] , and holds significance for tasks such as person re-identification [ 35 ] and the generation of sign language videos. The PGPIS task entails transforming an individual’s image to correspond with a designated pose, necessitating the alignment of the source image with the target pose. This process frequently encounters difficulties in synthesizing occluded regions, where the model must infer parts not visible in the source image. For instance, in the first row of Fig. 1 , the model is required to predict the individual’s lower body to generate the target image. Recent research in the field of PGPIS has predominantly focused on diffusion-based methodologies, with prominent examples including PIDM [ 1 ] , PCDM [ 29 ] , X-MDPT [ 23 ] , and CFLD [ 18 ] . Nevertheless, these models encounter challenges related to slow sampling speeds. Diffusion [ 10 ] is a score-based model that employs Stochastic Differential Equations (SDEs), necessitating hundreds to thousands of function evaluations for sampling. This requirement significantly decelerates the process, rendering it computationally inefficient for practical applications. Continuous Normalizing Flows (CNFs) [ 2 ] have recently emerged as a significant category within generative models, capable of modeling arbitrary probability paths. The Flow Matching model facilitates the training of CNFs without the need for simulation [ 15 ] . Furthermore, by mapping images to a latent space, generative models can be effectively applied within this space, thereby enabling conditional generation through flow matching [ 3 ] . Flow matching seeks to learn an ordinary differential equation that delineates a trajectory from a source distribution to a target distribution. Generative models based on ordinary differential equations, such as this, are recognized for offering more rapid training and sampling speeds compared to stochastic differential equation-based models, such as diffusion models. [ 3 ] While flow matching has been recently investigated for real-time generation within the speech domain [ 13 , 16 ] , its application in this specific area remains unexplored. The Motion Flow Matching [ 11 ] paper generates poses from textual input; however, it predominantly produces mannequin-like figures, failing to preserve visual appearance or maintain a specified pose condition. Consequently, this represents a distinct task from PGPIS. We propose the application of flow matching to develop a conditional generation model for PGPIS, with the objective of enhancing sampling speed, and we substantiate our approach through experimental validation. Our major contributions are as follows: • We propose a transformer-based flow matching model for pose-guided person image synthesis, where a pre-trained VAE maps images to a latent space and a DiT-based neural network predicts the latent flow. By conditioning on both source images and target poses, our method efficiently aligns the source to the desired pose. This design enables significant reduction in inference steps, achieving near real-time image and video generation without compromising output quality. • We introduce two complementary conditioning techniques—input concatenation and conditional aggregation—to enhance both the speed and performance of conditional generation in our model. Input concatenation merges the noisy latent, source image, and target pose into a unified representation, enabling the model to directly incorporate spatial features with minimal computational overhead. Conditional aggregation integrates four distinct condition embeddings (local/global features of both source image and target pose) through transformer-based self- and cross-attention. This architecture allows for efficient conditioning, thereby reducing inference time. Our design extends previous works that used only three conditional embeddings, and ablation studies demonstrate that the combined use of both methods not only improves generation quality but also accelerates the PGPIS model, making it suitable for real-time applications. Figure 1 : An illustration of the DeepFashion PGPIS dataset is presented. In this task, the individual depicted in the source image must be modified to align with the target pose. 2 Related Work 2.1 Neural ODE Neural Ordinary Differential Equations (Neural ODEs) [ 2 ] constitute an innovative class of deep neural networks. Conventional deep learning architectures, such as ResNet, consist of multiple discrete, independent layers, with each layer processing values at a single time step. Neural ODEs draw inspiration from the resemblance between the residual connections in ResNet and Euler’s method in solving ordinary differential equations, both of which involve the accumulation of incremental changes. Unlike ResNet [ 7 ] models, which are limited by a finite number of layers (e.g., ResNet50 or ResNet152), Neural ODEs theoretically permit an infinite number of layers by utilizing Euler’s method with smaller step sizes, thereby facilitating a transition from discrete to continuous transformations.

Read the complete text