DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression

Wei Huang, Huang Wei, Yinggui Wang Ant Group, China{hw378176,lingxin.hw}@antgroup.com, [email protected] · 2025

Read DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression by Wei Huang, Huang Wei, Yinggui Wang Ant Group, China{hw378176,lingxin.hw}@antgroup.com, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01221.txt.

Introduction

Many open language models exist, but picking the best one to fine-tune for a specialist task is slow and costly to test directly. Wei Huang, Huang Wei, and Yinggui Wang introduce DaMoC, a framework that chooses by compressing both the data and the candidate models. Smaller stand-ins let the team judge fit before committing to full training.

On the data side, they sort filtering methods into three families and raise the density of key tokens, then rewrite text with a model's help to sharpen it. On the model side, they score each layer's importance, drop weak layers, and merge what remains sparsely to keep skill. Wide experiments back the approach. DaMoC aims to make model choice fast and grounded.

Catalog record

CREATORWei Huang, Huang Wei, Yinggui Wang Ant Group, China{hw378176,lingxin.hw}@antgroup.com, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression Wei Huang, Huang Wei, Yinggui Wang Ant Group, China{hw378176,lingxin.hw}@antgroup.com, [email protected] Abstract Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. With many open-source LLMs available, selecting the best model for fine-tuning downstream tasks is challenging, primarily focusing on how to quickly identify the optimal LLM. We introduce a Da ta and Mo del C ompression Framework (DaMoC) that addresses this challenge by: 1) Data Level: A systematic categorization of data filtering methodologies for LLMs is first established, classifying them into three distinct paradigms: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. Further, we enhance the density of key tokens in the text achieving token compression. Subsequently, we use an LLM to iterative rewrite the text to optimize its expression. 2) Model Level: We use layer similarity scores to assess each layer’s importance and remove those with lower importance. Then, we introduce a sparse merging paradigm to preserve as much of the original model’s capability as possible. Extensive experiments on four datasets, medical Q&A, financial Q&A, general Q&A, and reading comprehension, show that we can select the optimal LLM while saving approximately 20-fold in training time. 1 Introduction In recent years, LLMs have rapidly developed, showcasing strong general capabilities after being pre-trained on large-scale datasets, and can handle diverse tasks through few-shot prediction or n-context learning GLM et al. 2024 ; Huang et al. 2024 . However, in specific fields such as law and medicine, the performance of LLMs is not entirely satisfactory Yang et al. 2024b . To improve the model’s performance in these areas, users typically fine-tune the model using domain-specific data. Over the past few years, numerous institutions have released various open-source LLMs, such as Llama3 Dubey et al. 2024 , Qwen2.5 Yang et al. 2024a , etc. This brings a challenge: when fine-tuning LLMs with domain data, how should we choose the optimal open-source LLM? Although existing research has focused on the issue of hyperparameter selection in fine-tuning LLMs Liu et al. 2024a , no studies have yet addressed the problem of selecting an optimal LLM for fine-tuning. The core issue we need to address is how to design an algorithm that enables rapid training and ensures relative performance stability among different models. Figure 1: Illustration optimal model selection problem, the desired objectives, and methods for accelerating training. Currently, common methods for accelerating the training or inference of LLMs include knowledge distillation Xu et al. 2024 , quantization Wan et al. 2023 , pruning Zhu et al. 2023 , data filtering Chen et al. 2024 , and token compression Jiang et al. 2023b . Knowledge distillation transfers the knowledge from a teacher model to a smaller one, but the process is time-consuming. Quantization compresses the model by converting floating-point numbers to integers or fixed-point numbers. Most existing research focuses on the post-training phase, while quantization methods in the fine-tuning phase primarily focus on parameter-efficient fine-tuning, such as PEQA Kim et al. 2024 and Qlora Dettmers et al. 2024 . Therefore, these methods are not very friendly for full fine-tuning and require hardware support. Pruning accelerates models by removing unimportant parameters, requiring a large amount of pre-training data for retraining, and there is limited exploration in scenarios where fine-tuning on downstream tasks. Data filtering selects important or similarly distributed data from the original dataset, but our experiments found that a high filtering ratio can affect the relative stability among models (see Section 5.1). Token compression speeds up processes by shortening the input text length, mainly used in the inference phase, and its application in fine-tuning still requires further exploration. A summary can be seen in Figure 1 . To quickly and accurately select the best open-source LLMs for fine-tuning, we propose DaMoC. DaMoC addresses the model selection problem from both data and model perspectives. On the data level, we categorize LLM data filtering methods into three types: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. We have established a dedicated data compression framework based on these categories to evaluate the relationship between data filtering methods and optimal model selection. Through our research, we found that distribution-aware methods perform better than hybrid approaches considering both dimensions, which in turn outperform quality-aware methods(see Section 5.1 for details). Additionally, we found that data filtering methods struggle to select the optimal model when the filtering ratio is too high (>90%). We also employed perplexity and Question-related perplexity to enhance the density of critical information in the input text. Subsequently, we used an LLM to iterative rewrite the text to optimize its expression achieving a compression rate of approximately 50%. At the model level, we focused on model pruning to balance time efficiency and implementation generality. We adopted a layer-wise pruning approach. We began by collecting input and output activations for each layer using a small calibration dataset and calculated their cosine similarity, deeming layers with higher similarity as less important. To preserve model accuracy and parameter distribution, we introduce a sparse merging paradigm to merge the parameters of pruned layers with those of the remaining ones. Our method successfully prunes approximately 25% of the layers while identifying the optimal LLM. Extensive experiments on different LLMs and various datasets show that by jointly applying data filtering, token compression, and model pruning, we can select the optimal LLM while reducing training time by approximately 20-fold. Further analysis revealed that the DaMoC can also identify the optimal fine-tuning method from full fine-tuning and LoRA fine-tuning, providing valuable insights for future research. Base Model PubMedQA BillSum Full LoRA Zero-shot One-shot Two-shot Full LoRA Zero-shot One-shot Two-shot Llama3.1-8B 0.5978(2) 0.6083(2) 0.4901(8) 0.5125(8) 0.5274(8) 0.7017(3) 0.6958(1) 0.5689(6) 0.5943(5) 0.6002(5) Qwen2.5-7B 0.5973(3) 0.6090(1) 0.5233(5) 0.5289(5) 0.5300(7) 0.7060(1) 0.6901(4) 0.5673(7) 0.5921(6) 0.5933(7) Gemma2-9B 0.5898(8) 0.6076(4) 0.5202(6) 0.5287(7) 0.5358(1) 0.7016(4) 0.6893(5) 0.5509(8) 0.5782(8) 0.5891(8) GLM4-9B 0.5914(5) 0.6046(6) 0.5253(4) 0.5318(3) 0.5341(4) 0.7021(2) 0.6948(3) 0.5798(3) 0.6012(3) 0.6161(1) Ministral-8B 0.5954(4) 0.6078(3) 0.5296(1) 0.5344(2) 0.5353(2) 0.6979(5) 0.6952(2) 0.5698(5) 0.5835(7) 0.5990(6) Phi-3-small-7B 0.6063(1) 0.6038(8) 0.5283(2) 0.5349(1) 0.5350(3) 0.6881(8) 0.6866(8) 0.5947(1) 0.6124(1) 0.6120(2) Yi1.5-9B 0.5911(7) 0.6065(5) 0.5171(7) 0.5288(5) 0.5319(5) 0.6957(6) 0.6871(7) 0.5801(2) 0.6033(2) 0.6057(3) Internlm2.5-7B 0.5914(5) 0.6043(7) 0.5261(3) 0.5302(4) 0.5303(6) 0.6936(7) 0.6874(6) 0.5772(4) 0.5985(4) 0.6007(4) Table 1: The impact of few-shot on selecting the most suitable LLM. Full represented Full fine-tuning. LoRA represented LoRA fine-tuning. Full and LoRA serve as baselines trained on all models. The optimal model is considered to have been selected if the results of few-shot are consistent with these baselines. Numerical values in parentheses indicate the ranking of the current model’s accuracy among all evaluated models, and the optimal model is denoted in red. 2 Motivation Inspired by in-context learning Dong et al. 2024 , we explore whether it is possible to choose the most suitable LLM for domain fine-tuning through a few-shot approach without additional training. Our experiments involve selected examples from the training data, structured as follows: instruct+example_{1}+...+example_{N}+question+answer . where N represents the number of included examples. For our experiments, we utilize two domain-specific datasets and eight LLMs. The selected datasets include PubMedQA Jin et al. 2019 for medical tasks and BillSum Kornilova and

Read the complete text