IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter
Xiaojing Zhong, Zhonghua Wu, Xiaofeng Yang, Guosheng Lin, Qingyao Wu · 2025
Read IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter by Xiaojing Zhong, Zhonghua Wu, Xiaofeng Yang, Guosheng Lin, Qingyao Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15616.txt.
Introduction
Xiaojing Zhong and four co-authors build a system that dresses a 3D person in a chosen garment from photos, in a 2025 paper. The input is just two images: one of the person and one of the clothing item. Shoppers and designers want to see the fit in three dimensions, not as a flat overlay.
Their framework, IPVTON, guides a mixed 3D body model with image prompts fed into a diffusion model. Masks keep the guidance focused on the clothing region so the face and background stay untouched. A generated outline adds shape checks, keeping the person's build while showing the garment truly worn. Tests, both numeric and visual, beat earlier methods on shape and texture. The work moves virtual fitting toward usable online fitting rooms.
Catalog record
| CREATOR | Xiaojing Zhong, Zhonghua Wu, Xiaofeng Yang, Guosheng Lin, Qingyao Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15616
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter Xiaojing Zhong, Zhonghua Wu, Xiaofeng Yang, Guosheng Lin, Qingyao Wu Abstract Given a pair of images depicting a person and a garment separately, image-based 3D virtual try-on methods aim to reconstruct a 3D human model that realistically portrays the person wearing the desired garment. In this paper, we present IPVTON, a novel image-based 3D virtual try-on framework. IPVTON employs score distillation sampling with image prompts to optimize a hybrid 3D human representation, integrating target garment features into diffusion priors through an image prompt adapter. To avoid interference with non-target areas, we leverage mask-guided image prompt embeddings to focus the image features on the try-on regions. Moreover, we impose geometric constraints on the 3D model with a pseudo silhouette generated by ControlNet, ensuring that the clothed 3D human model retains the shape of the source identity while accurately wearing the target garments. Extensive qualitative and quantitative experiments demonstrate that IPVTON outperforms previous methods in image-based 3D virtual try-on tasks, excelling in both geometry and texture. 1 School of Software Engineering, South China University of Technology, China 2 Nanyang Technological University, Singapore 3 SenseTime Research, Singapore 4 Peng Cheng Laboratory, China [email protected], [email protected], [email protected] Introduction Human generation has been a prominent task in the AIGC field, with virtual try-on attracting widespread attention due to its significant commercial and entertainment value. Image-based 2D virtual try-on technology, which generates a realistic photo of a person wearing a desired garment by combining the person’s image with the garment’s image, is valued for its user-friendliness and resource efficiency. However, this method is limited by its reliance on a fixed viewpoint (see Fig. 1 (a)), which poses challenges in real-world applications where users often need to assess the garment from multiple angles. On the other hand, the traditional 3D virtual try-on method provides the advantage of multi-angle views but requires complex processes such as garment-body registration and physics simulations (see Fig. 1 (b)), making it labor-intensive. The challenge of reconstructing accurate 3D models from 2D images, an inherently ill-posed problem, further complicates efforts to integrate image-based and 3D-based virtual try-on techniques. Owing to the remarkable progress in diffusion models for Text-to-Image (T2I) ( Ho, Jain, and Abbeel 2020 ; Sohl-Dickstein et al. 2015 ; Song and Ermon 2019 ) , the field of 3D content generation has seen significant advancements. Recent works ( Chen et al. 2023 ; Wang et al. 2024 ; Qian et al. 2023 ; Zhong et al. 2025 ) leverage 2D generative priors from pre-trained T2I models (e.g., StableDiffusion (SD)) combined with the Score Distillation Sampling (SDS) loss ( Poole et al. 2022 ) to optimize 3D representations, resulting in high-quality 3D objects. Despite the success in synthesizing images with specific concepts ( Ruiz et al. 2023 ; Kumari et al. 2023 ) , extending these techniques to customized 3D object generation remains challenging. For instance, incorporating personalized modules such as LoRAs ( Hu et al. 2021 ) into the SD model diminishes its ability to generate consistent multi-view images ( Xie et al. 2024 ) . Additionally, fine-tuning with only a few images struggles to capture the complex features of garments necessary for 3D virtual try-on. Figure 1: Compared to 2D virtual try-on ( Kim et al. 2024 ) with its fixed viewpoint and 3D virtual try-on ( Li et al. 2024 ) that require complex processes, IPVTON can generate 3D try-on results from just a human image and a garment image. Figure 2: 3D Try-on results. Given a human image, a garment image and a text prompt, IPVTON can generate realistic 3D human models with the desired garment shapes and textures while preserving the source identity. Image prompt adapter (IP-Adapter) ( Ye et al. 2023 ) introduces a cross-attention layer for image prompts in diffusion models, enabling controllable generation based on provided images. In this paper, we propose IPVTON, a data-efficient image-based 3D virtual try-on framework that integrates an image prompt adapter with customized diffusion models to optimize a hybrid 3D human model using SDS loss. Since IP-Adapter is compatible with existing diffusion models, it eliminates the need for additional parameter fine-tuning with limited-viewpoint images, preserving multi-view generative priors for consistent 3D model generation. Moreover, combining textual and visual prompts effectively encapsulates the high-level semantics of garments. Specifically, we adopt a two-stage 3D generation framework that independently optimizes the geometry and texture of a hybrid 3D human model initialized with SMPL-X ( Pavlakos et al. 2019 ) , using an image prompt encoder to extract features from the target garment image and its corresponding normal map to guide the respective optimizations. Unlike using a reference image to influence the entire output ( Zeng et al. 2023 ; Ran et al. 2024 ) , virtual try-on requires preserving the non-try-on regions of the source human image during optimization. To address this problem, we employ mask-guided image prompt embeddings to focus the image prompt features on the targeted region, reducing unintended effects on surrounding areas. Furthermore, while mask guidance mitigates interference, it may limit the effectiveness of image prompts in guiding geometry generation. To overcome this problem, we introduce a Pseudo Silhouette Loss (PSL) to ensure the generated 3D human conforms to the desired garment shapes. Fig. 2 illustrates the 3D try-on results generated from a human image and an in-shop garment image. Overall, our contributions are summarized as follows: • We design a data-efficient image-based 3D virtual try-on framework that generates 3D human models seamlessly wearing the desired garments, which can be observed from any viewpoint. • We combine score distillation sampling with image prompts to optimize a hybrid 3D human representation, using an image prompt adapter to integrate garment features into the diffusion prior. We leverage mask-guided image prompt embeddings to focus the image features on the masked region, preserving the source identity in non-try-on areas. • To ensure the generated model accurately reflects the desired garment shapes, we propose a pseudo silhouette loss to optimize the 3D human geometry. Related Work 2D and 3D Virtual Try-on. Image-based 2D virtual try-on aims to fit an in-shop garment onto a clothed human in an image. Traditional methods primarily rely on Generative Adversarial Networks (GANs) ( Goodfellow et al. 2020 ; Zhong et al. 2023a ; Wu et al. 2020 ; Shi et al. 2021 ) , where the garment is first deformed to align with the person’s pose, followed by a generator that blends the deformed garment with the person’s image ( Zhong et al. 2021 ; Wu et al. 2019 ; Choi et al. 2021 ; Ge et al. 2021 ) . Building on the advancements of diffusion models in image editing, virtual try-on research has increasingly focused on their application, leveraging pre-trained diffusion models to blend garments with human appearances ( Kim et al. 2024 ; Choi et al. 2024 ; Zhu et al. 2023 ) . Despite the success of 2D virtual try-on methods, they struggle to generate multi-view try-on results, which are crucial for real-world applications. With the increasing demand for 3D virtual try-on, ( Bhatnagar et al. 2019 ; Mir, Alldieck, and Pons-Moll 2020 ; Patel, Liao, and Pons-Moll 2020 ; Zhong et al. 2023b ) represent garments