Towards Vision-Language-Garment Models For Web Knowledge Garment Understanding and Generation

Jan Ackermann George Nakayama Guandao Yang Tong Wu Gordon Wetzstein · 2025

Read Towards Vision-Language-Garment Models For Web Knowledge Garment Understanding and Generation by Jan Ackermann George Nakayama Guandao Yang Tong Wu Gordon Wetzstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05210.txt.

Introduction

Jan Ackermann, Kiyohiro Nakayama, Guandao Yang, Tong Wu, and Gordon Wetzstein test whether a web-trained multimodal model can design clothes. General vision-language models reason well about everyday images and text. Fashion, with its cuts, fabrics, and fit logic, is a sterner test of transfer.

Their model, VLG, builds garments from written prompts and pictures. The team probes zero-shot use: no fashion training, just the model's web knowledge aimed at unseen styles and requests. Early results show the knowledge carries over better than expected. The authors argue specialist domains like garment design may ride on general models sooner than assumed, and they sketch where the approach strains.

Catalog record

CREATORJan Ackermann George Nakayama Guandao Yang Tong Wu Gordon Wetzstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Vision-Language-Garment Models For Web Knowledge Garment Understanding and Generation Jan Ackermann George Nakayama Guandao Yang Tong Wu Gordon Wetzstein Abstract Multimodal foundation models have demonstrated strong generalization, yet their ability to transfer knowledge to specialized domains such as garment generation remains underexplored. We introduce VLG, a vision-language-garment model that synthesizes garments from textual descriptions and visual imagery. Our experiments assess VLG’s zero-shot generalization, investigating its ability to transfer web-scale reasoning to unseen garment styles and prompts. Preliminary results indicate promising transfer capabilities, highlighting the potential for multimodal foundation models to adapt effectively to specialized domains like fashion design. The project page is https://www.computationalimaging.org/publications/vision-language-garment-models/ . 1 Introduction Multimodal foundation models have demonstrated remarkable generalization and reasoning capabilities across various tasks, driven by extensive pre-training on web-scale datasets. However, their effectiveness in transferring these general-purpose abilities to specialized domains—such as garment design and generation—remains relatively unexplored. Garment design presents a challenging testbed for these models due to its inherent multimodal nature, requiring the integration of visual aesthetics, textual descriptions, functional constraints, and user-specific preferences. Traditional garment generation approaches have predominantly relied on retrieval-based or narrowly supervised learning methods Liu et al. 2023b ; He et al. 2024 , which often struggle to generalize beyond their training data and exhibit limited flexibility in addressing complex, nuanced user inputs. While recent advancements leveraging large-scale datasets like GarmentCode Korosteleva and Sorkine-Hornung 2023 have improved multimodal garment generation Bian et al. 2025 ; Nakayama et al. 2025 , these methods do not explore model-performance out of their data training distribution. In this work, we explore the potential of vision-language foundation models to transfer their multimodal reasoning skills into garment generation tasks effectively. Specifically, we introduce and evaluate VLG, a Vision-Language-Garment model built upon a pre-trained vision-language foundation model. Our core contributions are as follows: 1. The introduction of VLG, a multimodal garment generation model designed to leverage generalized vision-language reasoning for garment-specific tasks. 2. An evaluation of the reasoning capabilities of VLG against state-of-the-art methods, using a curated benchmark to assess multimodal knowledge transfer. Our study aims to illuminate how multimodal foundation models can adapt their generalized reasoning abilities to specialized, domain-specific applications, laying the groundwork for future research in multimodal knowledge transfer and intelligent garment generation. Figure 1 : Vision-Language-Garment Models. We tune existing vision-language models for garment-specific tasks to predict sewing patterns from multimodal (textual and visual) inputs. Figure 2 : Reasoning Transfer Evaluation. Performance comparison of our proposed model ( red ) against DressCode He et al. 2024 ( blue ), SewFormer Liu et al. 2023b ( orange ) and Sewformer-FT ( green ), across various reasoning dimensions. 2 Related Work Digital Garment Understanding and Generation. Advances in digital garment generation have explored a sewing pattern representation Liu et al. 2023b ; Korosteleva and Lee 2022 ; He et al. 2024 that stands out due to its direct application to real-world manufacturing and garment production processes. The recent availability of larger-scale sewing pattern datasets like GarmentCode Korosteleva and Sorkine-Hornung 2023 has facilitated multimodal learning approaches Bian et al. 2025 ; Nakayama et al. 2025 , significantly enhancing flexibility and cross-modal understanding. Nonetheless, there is no clarity on the pre-trained backbone’s role in the final model performance. Multimodal Foundation Models and Knowledge Transfer. Multimodal foundation models like GPT-4 OpenAI 2024 and LLaMA LLaMA-Team 2023 have demonstrated strong generalization capabilities across diverse tasks. Recent works explore adapting these models to specialized domains through prompting Shen et al. 2024 ; Wu et al. 2023 and domain-specific fine-tuning Liu et al. 2023a ; Zhu et al. 2023 ; Tong et al. 2024 , though fine-tuning often risks catastrophic forgetting. Notably, robotics-focused approaches such as RT-2 Team 2023 and OpenVLA OpenVLA-Team 2024 successfully retain general reasoning during transfer. Motivated by these results, our work investigates how effectively multimodal reasoning transfers to the garment domain by evaluating adapted models on a garment-specific reasoning benchmark. 3 Vision-Language-Garment Models This work explores the capability of multimodal foundation models to transfer their general knowledge and reasoning skills into the specialized domain of garment generation. In the following sections, we detail our approach, including dataset curation for training and evaluating reasoning transfer, and the fine-tuning procedure to adapt pre-trained multimodal foundation models to the garment domain. 3.1 Dataset Curation A critical factor for multimodal reasoning transfer to garment-specific tasks is the curation of suitable training and testing datasets. We contribute two key datasets designed explicitly to train and benchmark multimodal garment reasoning abilities. Garment Prompt Dataset. Existing garment datasets such as AIpparel Nakayama et al. 2025 and DressCode He et al. 2024 predominantly feature detailed and precise textual descriptions, differing significantly from naturally vague and emotionally guided human interactions. To bridge this gap, we leverage GPT-4o’s natural language generation abilities to create a dataset with diverse, context-rich, and realistically ambiguous prompts aligned closely with human interaction patterns. Given sewing pattern images from the GarmentCode dataset, GPT-4o was prompted to generate diverse and nuanced queries, expanding the coverage of naturalistic prompts significantly beyond existing benchmarks. Synthetic Textured Garment Dataset. Evaluating garment reasoning requires data accurately reflecting real-world variability in garment appearances and textures. Previous approaches often rely on manually designed textures, limiting scalability and generalization. We introduce a scalable pipeline to address this and generate diverse, textured garment images. Our method begins with rendered garment images and utilizes the Canny Edge Detector to derive control masks. GPT-4o OpenAI 2024 is then queried to generate plausible textual descriptions for textures, which are subsequently synthesized using ControlNet Zhang et al. 2023 into neural-rendered images. The backgrounds are masked white to maintain consistency and ensure clean visual boundaries. For evaluation, we refine some synthesized images using a Virtual Try-On tool Choi et al. 2024 . Given the limitations of such tools in geometry preservation, we manually curate the final dataset, ensuring suitability for rigorous multimodal reasoning evaluation. 3.2 Model Training and Fine-Tuning For the training, we utilize AIpparel’s Nakayama et al. 2025 garment tokenization scheme to represent garments. We adopt standard fine-tuning practices of multimodal foundation models. Specifically, we train 16 Nvidia H100 GPUs with a per-GPU batch size of 6 and gradient accumulation steps of 5, achieving an effective batch size of 480. Training proceeds for 20 epochs, employing a linear warm-up of the learning rate from 0 to 6\cdot 10^{-5} during the first 1000 steps, followed by linear decay to 0 . To maintain the base multimodal model’s general reasoning capabilities and visual knowledge, we freeze the vision encoder and cross-attention modules. The final layer of the regression head is initialized to zero, stabilizing early-stage optimization by controlling the magnitude of prediction errors. 4 Experiments In this section, we present qualitative and quantitative results of our method, comparing against the following baselines: DressCode He et al. 2024 , SewFormer Liu et al. 2023b , and SewFormer-FT, a SewFormer fine-tuned on GarmentCodeData Korosteleva et al. 2024 . When an input data-point has mismatched modality, we employ GPT-4o to translate it into the correct modality. Metrics. Throughout this section, we quantitavely evaluate the garment predictions by garment accuracy Nakayama et al. 2025 and Vertex L2. For text-based predictions, we ask human evaluators to score the overall-quality and the prompt-alignment. Garment Reasoning Analogous to VLA approaches Team

Read the complete text