Backbone Augmented Training for Adaptations

Jae Wan Park, Junhyeok Kim, Youngjun Jun, Hyunah Ko, Seong Jae Hwang · 2025

Read Backbone Augmented Training for Adaptations by Jae Wan Park, Junhyeok Kim, Youngjun Jun, Hyunah Ko, Seong Jae Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04288.txt.

Introduction

Jae Park and co-authors fed adaptation with the backbone's own data in two thousand twenty-five. Big models adapt cheaply with small add-on training. Scarce adaptation data makes that training shaky. Their fix reuses the vast pretraining data itself. Backbone Augmented Training mixes it into adaptation sets. Two mathematical claims ground the method's validity.

One claim shows when the mixing truly helps. A selection scheme picks backbone samples that qualify. Their algorithm, ALBAT, automates that picking. Tests covered image personalization and language generation. Gains appeared without new external data. Compute stayed modest throughout the trials. The trick recycles sunk pretraining costs wisely.

Catalog record

CREATORJae Wan Park, Junhyeok Kim, Youngjun Jun, Hyunah Ko, Seong Jae Hwang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Backbone Augmented Training for Adaptations Jae Wan Park, Junhyeok Kim, Youngjun Jun, Hyunah Ko, Seong Jae Hwang Abstract Adaptations facilitate efficient training of large backbone models, including diffusion models for image generation and transformer-based language models. While various adaptation techniques enhance performance with minimal computational resources, limited adaptation data often lead to challenges in training. To address this, we focus on the enormous amount of backbone data used to pre-train the backbone models. We propose B ackbone A ugmented T raining (BAT), a method that leverages backbone data to augment the adaptation dataset. First, we formulate and prove two mathematical key propositions: one establishes the validity of BAT, while the other identifies a condition under which BAT benefits adaptation. Furthermore, we introduce an advanced data selection scheme that satisfies these propositions and present AL gorithm for BAT (ALBAT) to implement this approach. ALBAT efficiently enhances adaptation training in both personalization and language generation tasks with scarce data. 1 Introduction Recently, large foundation models have demonstrated exceptional performance across various tasks [ 1 , 2 , 3 , 4 , 5 ] . To adapt these models for specific downstream tasks, a variety of adaptation techniques have been introduced. These approaches typically involve updating only a small portion of parameters from the large model, often called backbone models. Some have leveraged rank decomposition [ 6 , 7 , 8 ] of the backbone weights, while others employing fixed text embeddings [ 9 , 10 ] to maintain identity consistency in image generation. Adaptation methods have enabled any user, equipped with an appropriate adaptation dataset, to fine-tune large models for tasks defined by themselves. However, without sufficient data, the adaptation is likely to suffer from various complications. Primarily, underfitting may occur, and at the same time, efforts to alleviate underfitting by increasing the number of training steps can result in overfitting [ 11 , 12 , 13 ] . Thus, obtaining sufficient amount of adaptation data is crucial, but imposes a significant burden on users [ 14 , 15 , 16 ] . Meanwhile, recent studies have highlighted the importance of the distribution of backbone [ 17 ] in dealing with data scarcity. There has been a work that utilized pre-train dataset in language model [ 18 ] , then in vision tasks [ 19 ] . In adaptations, DreamBooth [ 9 ] utilizes synthetic data generated by the backbone model for regularization, while DoRA [ 8 ] proposes an adaptation method that aligns training behavior with that of the backbone. Inspired by these approaches, we propose a method called Backbone Augmented Training (BAT) to mitigate data scarcity. BAT augments adaptation data by leveraging the backbone data used during the pre-training of the backbone model. Although approaches similar to BAT have recently been proposed within the community [ 20 ] , they perform augmentation using backbone data without any theoretical validation. For instance, [ 20 ] arbitrarily selects backbone data that appear likely to enhance adaptation performance. Consequently, while these methods occasionally yield favorable results, they often perform worse than cases without augmentation. To overcome these fundamental flaws, we establish a mathematical foundation of BAT for the first time, demonstrating its theoretical validity. Figure 1: Framework of our Backbone Augmented Training. (A) The surrogate model is first trained using the adaptation dataset, while the backbone model is pre-trained separately. (B) With the help of the surrogate model, backbone data are selected using a selection scheme grounded in our solid mathematical foundation. This approach differentiates itself from traditional adaptation methods that do not use backbone data and existing methods [ 20 ] that arbitrarily select backbone data. To establish a solid mathematical foundation for BAT, we begin by adopting reasonable assumptions proposed in [ 21 ] . Building on these assumptions, we derive two key propositions to verify the rationale and efficiency of BAT. The first proposition demonstrates that BAT allows the adaptation parameters to converge to the optimal parameters, justifying the use of backbone data in adaptations. The second proposition offers a fundamental condition that helps to monitor the convergence rate of BAT. According to the second proposition, BAT outperforms traditional adaptation methods [ 6 , 9 , 8 ] when the selected backbone data satisfy specific conditions. An effective strategy for selecting backbone data that meets the condition of our second proposition necessarily facilitates adaptation training with BAT. Drawing inspiration from data selection studies under weak supervision [ 22 ] , we propose a biased selection scheme that thoroughly selects a subset of the backbone data to augment the adaptation dataset. While the biased scheme has been shown to be more effective than random selection or an unbiased scheme [ 22 ] , it faces the challenge of high impractical computational costs due to Hessian derivatives. We address this limitation by adopting mathematical methodologies inspired by [ 23 ] , reducing the computational cost from O(D^{3}) to O(D) , for D adaptation parameters. Finally, we devise a complete pipeline, referred to as ALBAT (Fig. 1 ), which implements this selection scheme. To sum up, our main contributions are as follows: • As a method to augment the adaptation dataset, we propose Backbone Augmented Training (BAT). Through Propositions 4.1 and 4.2 , we establish the theoretical foundation of BAT and define a condition under which it is effective. • Building on our mathematical foundation, we propose ALBAT, an algorithm for selecting useful backbone data. To this end, we make data influence computation feasible by integrating multiple mathematical techniques. • Beyond theoretical validation, we conduct experiments using ALBAT on personalized image generation (Fig. 2 ) and language generation tasks (Table 1 ) in data-scarce scenarios, demonstrating that applying BAT improves adaptation performance. 2 Related Works Figure 2: Application Results of BAT in Personalized Image Generation Task. The comparison is conducted between two approaches: augmenting a text-to-image model personalized via DreamBooth [ 9 ] with arbitrary selected backbone data (Random) and utilizing our BAT data selection scheme (BAT). The sample ratio represents the proportion of backbone dataset used for data selection while equal amount of backbone data is selected for each sample ratio. Each metric evaluates the performance of personalized image generation. While the random approach shows better performance only for few metrics and no correlation with the sample ratio, BAT exhibits consistent performance improvement as the sample ratio increases, achieving results that surpass others and coinciding with data selection theory [ 22 ] . For more details, please refer to Sec. 6.1 . Adaptations. Fine-tuning a large-scale model to solve a downstream task is extremely expensive. To mitigate this challenge, many have come up with methods that train a small portion of parameters, also known as adaptations. Adaptation methods are widely distinguished as additive fine-tuning [ 24 , 25 ] , selective fine-tuning [ 26 , 27 ] , reparameterized fine-tuning [ 28 , 29 ] . In the following part, we introduce eminent types of adaptations. LoRA. Low-Rank Adaptation [ 6 ] has gained significant attention among early adaptations for its ability to efficiently train a small portion of parameters through weight decomposition, without any additional inference burden. Despite utilizing only a small set of parameters, LoRA achieves performance comparable to full

Read the complete text