Decoupled Data Augmentation for Improving Image Classification
Ruoxin Chen , Zhe Wang , Keyue Zhang , Shuang Wu, Jiamu Sun, Shouli Wang, Taiping Yao, Shouhong Ding · 2024
Read Decoupled Data Augmentation for Improving Image Classification by Ruoxin Chen , Zhe Wang , Keyue Zhang , Shuang Wu, Jiamu Sun, Shouli Wang, Taiping Yao, Shouhong Ding (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02592.txt.
Description
Abstract Recent advancements in image mixing and generative data augmentation have shown promise in enhancing image classification. However, these techniques face the challenge of balancing semantic fidelity with diversity. Specifically, image mixing involves interpolating two images to create a new one, but this pixel-level interpolation can compromise fidelity.
Catalog record
| CREATOR | Ruoxin Chen , Zhe Wang , Keyue Zhang , Shuang Wu, Jiamu Sun, Shouli Wang, Taiping Yao, Shouhong Ding |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoupled Data Augmentation for Improving Image Classification Ruoxin Chen , Zhe Wang , Keyue Zhang , Shuang Wu, Jiamu Sun, Shouli Wang, Taiping Yao, Shouhong Ding Abstract Recent advancements in image mixing and generative data augmentation have shown promise in enhancing image classification. However, these techniques face the challenge of balancing semantic fidelity with diversity. Specifically, image mixing involves interpolating two images to create a new one, but this pixel-level interpolation can compromise fidelity. Generative augmentation uses text-to-image generative models to synthesize or modify images, often limiting diversity to avoid generating out-of-distribution data that potentially affects accuracy. We propose that this fidelity-diversity dilemma partially stems from the whole-image paradigm of existing methods. Since an image comprises the class-dependent part (CDP) and the class-independent part (CIP), where each part has fundamentally different impacts on the image’s fidelity, treating different parts uniformly can therefore be misleading. To address this fidelity-diversity dilemma, we introduce Decoupled Data Augmentation (De-DA), which resolves the dilemma by separating images into CDPs and CIPs and handling them adaptively. To maintain fidelity, we use generative models to modify real CDPs under controlled conditions, preserving semantic consistency. To enhance diversity, we replace the image’s CIP with inter-class variants, creating diverse CDP-CIP combinations. Additionally, we implement an online randomized combination strategy during training to generate numerous distinct CDP-CIP combinations cost-effectively. Comprehensive empirical evaluations validate the effectiveness of our method. A Preprint Figure 1: Visualization of different data augmentation methods. Column 2-4: CutMix, MixUp, and SaliencyMix interpolate two images at the pixel level. Column 5-7: DiffuseMix, DA-Fusion, and Diff-Mix utilize generative models to make semantic modifications to the input image. Column 8: Our proposed method, De-DA, fuses image-mixing with generative data augmentation. De-DA edits the class-dependent part of one image using a generative model, then mixes it with another image’s class-independent part to create a realistic and diverse image. 1 Introduction Data augmentation is extensively employed to enhance neural network performance. Traditional data augmentation, such as random shifting, cropping, and rotation, are widely used due to their simplicity and effectiveness, becoming standard practice in nearly all training algorithms. Recently, two innovative types of data augmentation have shown potential for improving image classification: • Image-Mixing Data Augmentation. Generate augmented images by integrating two or more randomly picked natural images at the pixel or feature level, creating virtual data between classes. The online combination paradigm allows for the efficient production of many images with extensive pixel-level variations at a low cost, yet the images often look unrealistic and face fidelity problems, as noted by ( Kang & Kim, 2023 ; Islam et al., 2024 ) . • Generative Data Augmentation. This method leverages generative models to create images using prompts generated manually or via textual inversion to align with class labels. However, as noted by ( Islam et al., 2024 ) , this method is not yet mature for data-rich learning scenarios. Crafting prompts that ensure model-generated images match the actual data distribution is difficult, requiring expert knowledge to describe class objects and challenges in capturing the dataset’s style. Additionally, textual inversion often leads to limited image diversity due to information loss, reducing the diversity of the generated images, as mentioned by ( Wang et al., 2024 ) . Both forms of prompt guidance encounter issues of misalignment or limited variation, resulting in limited performance improvements. Readers can refer to Figure 1 for examples of various data augmentation methods. It is evident that a trade-off exists between semantic fidelity and diversity in these methods. Naturally, the question arises: ’How can semantic fidelity be preserved while simultaneously enhancing diversity?’ The prevailing practice of treating images as indivisible units in existing data augmentation methods presents a fundamental obstacle to achieving both fidelity and diversity. This whole-image paradigm, while enriching diversity, often results in excessive and detrimental variations to class-dependent objects, severely compromising fidelity. In contrast, viewing images from a disentangled perspective could alleviate this challenge by applying distinct strategies to class-dependent parts and class-independent parts: a conservative strategy on CDPs to maintain fidelity and an aggressive strategy on CIPs to enhance diversity. Based on this insight, we propose a novel data augmentation framework, Decoupled Data Augmentation (De-DA), which addresses the fidelity-diversity dilemma through a decoupling strategy. Specifically, we first separate images into class-dependent parts (CDPs) and class-independent parts (CIPs) using SAM ( Kirillov et al., 2023 ) , and then tailor our adaptive strategies for respective parts according to their distinct characteristics. To preserve semantic fidelity, we use class identifiers derived from intra-class CDPs as conditions to edit real CDPs with controlled strength, elaborately varying them while preserving their semantic consistency. To encourage diversity, we replace the original CIP of the images with a random CIP sampled from an inter-class image. Furthermore, we adopt an online randomized combination strategy, pairing one CDP (real or synthetic) with one CIP (cross-class real CIPs) at random positions and transformations to provide the model with various combinations during the training stage, further enhancing diversity. In summary, both conservatively translated CDPs and real CIPs align with the actual data, ensuring that the generated images maintain fidelity, while the semantic edits on CDPs and diverse CDP-CIP combinations significantly enrich variety. Compared to previous image-mixing methods, De-DA fuses CDPs and CIPs at the semantic level rather than the pixel level, thereby enhancing fidelity. De-DA also distinguishes itself from other generative methods via applying textual inversion ( Gal et al., 2022 ) and SDEdit ( Meng et al., 2021 ) to isolated CDPs instead of the entire image, thus avoiding the negative effects of noisy information in the image. Furthermore, De-DA’s decouple-and-combine paradigm enables the production of more images at a lower cost than prior generative methods. Our contributions include: • De-DA shows a solution to the fidelity-diversity dilemma in previous data augmentation methods by decoupling images into class-dependent parts and class-independent parts and managing these parts adaptively. • To our knowledge, we are the first to apply textual inversion and SDEdit to isolated CDPs instead of entire images in the field of data augmentation, which minimizes the negative impact from the noisy information in the images. Additionally, we propose truncated-timestep textual inversion to reduce the computational burden, enhancing practicability. • Extensive experiments on domain-specific classification, multi-label classification, and data-scarce learning scenarios comprehensively validate the effectiveness of De-DA. Table 1: Comparing data augmentation methods on fidelity and diversity. Image-Mixing Generative Image-Mixing + Generative Mixup 2018 CutMix 2019 Real-Guidance 2023 DA-Fusion 2024 Diff-Mix 2024 DiffuseMix 2024 De-DA (ours) Mixing Pixel-Wise Patch-Wise — — — Mask-Wise Semantic-Wise Prompt — — Label Description Derived from Intra-Class Images Derived from Inter-Class Images Style Prompt Derived from Intra-Class CDPs Fidelity Low Low High High Medium High High Diversity High High Medium Low High Medium High Figure 2: Illustration of the mechanisms of different data augmentation methods. Row 1: Image-mixing methods, such as Mixup ( Zhang et al., 2018 ) and CutMix ( Yun et al., 2019 ) create mixed images through pixel-level interpolation. DiffuseMix ( Islam et al., 2024 ) uses style prompts (e.g., ”Sunset”) to transform input images, generating varied-style images which are then concatenated to form a hybrid image. DA-Fusion ( Trabucco et al., 2024 ) uses the intra-class identifier {\color[rgb]{0,0.42,0.24}\textbf{V1}_{\text{image}}} , while Diff-Mix ( Wang et