AIpparel: A Multimodal Foundation Model for Digital Garments

Kiyohiro Nakayama Jan Ackermann Timur Levent Kesdogan, Yang Zheng Maria Korosteleva Olga Sorkine-Hornung, Leonidas J. Guibas Guandao Yang Gordon Wetzstein · 2024

Read AIpparel: A Multimodal Foundation Model for Digital Garments by Kiyohiro Nakayama Jan Ackermann Timur Levent Kesdogan, Yang Zheng Maria Korosteleva Olga Sorkine-Hornung, Leonidas J. Guibas Guandao Yang Gordon Wetzstein (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03937.txt.

Introduction

Designing a garment still means hours of manual pattern work. Kiyohiro Nakayama and co-authors train a foundation model to do it from words or pictures in this 2024 paper. AIpparel starts from a large multimodal model and fine-tunes it on more than one hundred and twenty thousand real garments, each stored with text, images, and its sewing pattern.

A new token scheme compresses the patterns, which are complex assemblies of panels and stitches, into a form a language model can predict piece by piece. The system leads on text-to-garment and image-to-garment tests, turning a description or a photo into a workable pattern. It also edits interactively, so a user can restyle a garment through conversation. The project aims to bring pattern drafting closer to writing a prompt.

Catalog record

CREATORKiyohiro Nakayama Jan Ackermann Timur Levent Kesdogan, Yang Zheng Maria Korosteleva Olga Sorkine-Hornung, Leonidas J. Guibas Guandao Yang Gordon Wetzstein
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AIpparel: A Multimodal Foundation Model for Digital Garments Kiyohiro Nakayama Jan Ackermann Timur Levent Kesdogan, Yang Zheng Maria Korosteleva Olga Sorkine-Hornung, Leonidas J. Guibas Guandao Yang Gordon Wetzstein Abstract Apparel is essential to human life, offering protection, mirroring cultural identities, and showcasing personal style. Yet, the creation of garments remains a time-consuming process, largely due to the manual work involved in designing them. To simplify this process, we introduce AIpparel, a multimodal foundation model for generating and editing sewing patterns. Our model fine-tunes state-of-the-art large multimodal models (LMMs) on a custom-curated large-scale dataset of over 120,000 unique garments, each with multimodal annotations including text, images, and sewing patterns. Additionally, we propose a novel tokenization scheme that concisely encodes these complex sewing patterns so that LLMs can learn to predict them efficiently. AIpparel achieves state-of-the-art performance in single-modal tasks, including text-to-garment and image-to-garment prediction, and enables novel multimodal garment generation applications such as interactive garment editing. The project website is at https://georgenakayama.github.io/AIpparel/ . Figure 1 : AIpparel. We present a multimodal foundation model for digital garments trained by fine-tuning a large multimodal model on a custom sewing pattern dataset using a novel tokenization scheme for these patterns. AIpparel generates complex, diverse, high-quality sewing patterns based on multimodal inputs, such as text and images, and it unlocks new applications such as language-instructed sewing pattern editing. The generated sewing patterns can be directly used to simulate the corresponding 3D garments. † † ∗ Equal Contribution. † † † Work done as a visiting researcher at Stanford. 1 Introduction Clothing plays a crucial role in society, providing protection from the weather, reflecting societal norms, and serving as a means of personal expression. A key stage in garment production is the development of sewing patterns—a set of flat 2D panels with standardized assembly instructions that form a complete 3D garment [ 4 ] . Pattern making is a challenging task due to the complex geometric relationship between the 2D pattern and the draped 3D shape of the sewn garment. Even an experienced tailor must go through multiple iterations, incorporating feedback from various sources, including verbal descriptions of the garment’s fit and feel, as well as visual references of its appearance. To simplify the pattern-making process, we explore strategies to leverage emerging generative models with multimodal input, such as text and images. State-of-the-art sewing pattern prediction methods are designed to work with one specific input modality, such as 3D points [ 29 , 6 , 14 ] , images [ 50 , 79 , 70 , 46 , 75 ] , or language [ 21 ] . While effective within their respective domains, these single-modal approaches are often challenging to adapt to garment prediction tasks requiring different or combined input modalities. Expanding these methods to multimodal pattern prediction presents two primary challenges. First, no large-scale multimodal sewing pattern dataset is publicly available. Second, the capacity to accurately interpret multimodal inputs typically only emerges in large models with billions of parameters [ 1 , 36 ] . It remains uncertain how to efficiently scale existing methods to models of this size. In this paper, we propose to build such multimodal garment generative models by extending an existing large multimodal model (LMM) [ 74 , 62 , 36 ] to understand sewing patterns with complex geometries. To achieve this, we annotate the largest sewing pattern dataset [ 31 ] with multimodal labels. Our annotated dataset is ten times larger than those used by previous state-of-the-art generative methods [ 29 , 38 , 21 ] , including over 120,000 unique sewing patterns paired with detailed text descriptions, images, edited sewing patterns, and editing instructions. Fine-tuning an LMM to perform multimodal garment generation requires representing complex garments in a format that LMMs can understand—tokens. For this task, we develop a novel tokenization method that is both expressive in representing the complex geometries of sewing patterns and concise enough to fit within the limited context length of existing LMMs, making fine-tuning computationally efficient. Combining these components, we present AIpparel, a large multimodal model for generating sewing patterns. AIpparel can predict sewing patterns with complex geometries and outperforms state-of-the-art methods in single-modal garment prediction, often by a large margin. Moreover, our approach unlocks entirely new multimodal garment generation tasks. Our contributions include: • We present GCD-MM, a multimodal sewing pattern dataset extending the largest public dataset of sewing patterns with multimodal annotations. We plan to publicly release the dataset to inspire innovative garment prediction capabilities and further enable research in multimodal garment generation. • We develop a novel tokenization scheme and a new training objective for fine-tuning LMMs to predict sewing patterns. This tokenization method is critical for retargeting LMMs to multimodal garment prediction tasks efficiently. • We present AIpparel, the first multimodal foundation model for sewing pattern prediction capable of taking language, images, and sewing patterns as input. 2 Related Works Garment Generation. Prior works have studied learning-based garment generation represented in various formats, including images [ 5 , 27 ] , 3D meshes [ 57 , 17 , 44 , 40 , 26 , 60 , 49 , 85 , 41 , 54 , 86 , 83 , 33 , 81 ] , and sewing patterns [ 38 , 29 , 21 , 68 , 50 ] . Our paper focuses on generating sewing patterns, which, compared to other representations, are the industry standard and can be directly used for downstream simulation and manufacturing. Earlier works have explored a variety of different ways to generate and predict sewing patterns, including retrieval-based methods [ 14 , 20 ] , predicting sewing pattern templates with few parameters [ 25 , 76 , 64 , 65 ] , or cutting 3D scans into 2D panels [ 16 , 18 , 37 , 43 , 67 , 55 ] . These algorithms usually require heuristics, such as the output garment templates. This limits their flexibility in extending to different input modalities or more complex garment types. Further, researchers have successfully applied deep learning methods to generate sewing patterns [ 29 , 38 , 21 ] . While these methods can predict accurate sewing patterns based on input conditioning, they are task-specific models designed to work well only in a single modality. Extending these single-modal methods to a novel modality is difficult, in part because of the lack of large-scale multimodal sewing-pattern datasets and the requirement to redesign the network architecture. While Wang et al. [68] can predict sewing patterns from multiple modalities, including images, 3D garments, and body measurements, their method is limited to predicting simple garments with a predefined set of parameters. In this paper, we aim to tackle the challenge of creating a large multimodal generative model by curating the first multimodal garment dataset with complex garment geometries and providing a scalable recipe building on existing large multimodal models. Extending Large Multimodal Models. Large multimodal models have gained significant attention for their ability to understand language and images [ 48 , 2 , 15 , 62 , 58 ] . Efforts to extend LMMs to additional domains typically fall into two categories. Optimization-free approaches [ 39 , 56 , 71 , 24 , 74 , 69 , 77 ] employ prompt engineering.

Read the complete text