LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection

Jian Wu 1,2 , Hang Yu 1 ∗ , Bingchang Liu 1 , Wenjie Yang 1 , Peng Di 1 , Jianguo Li 1 , and Yue Zhang 2 Email: {wujian, zhangyue}@westlake.edu.cn, {hyu.hugo, bingchang.lbc, ywj439780, dipeng.dp, lijg.zero}@antgroup.com 1 Ant Group 2 Westlake University · 2025

Read LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection by Jian Wu 1,2 , Hang Yu 1 ∗ , Bingchang Liu 1 , Wenjie Yang 1 , Peng Di 1 , Jianguo Li 1 , and Yue Zhang 2 Email: {wujian, zhangyue}@westlake.edu.cn, {hyu.hugo, bingchang.lbc, ywj439780, dipeng.dp, lijg.zero}@antgroup.com 1 Ant Group 2 Westlake University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06524.txt.

Introduction

Choosing which data should fine-tune a language model for one special field is the bottleneck addressed by Jian Wu and six co-authors in a 2025 machine learning paper. Raw data is plentiful, but training on all of it adds noise and can hurt results. Existing selection methods struggle to be both accurate and cheap.

Their method, LAMDAS, treats selection as asking whether a candidate belongs to the target domain, using the pretrained model itself as the judge. A small reference set defines the domain, and no separate feature engineering or heavy optimization is needed. Across many tests, models trained on the selected fraction beat models trained on everything, and the method outperformed nine leading rivals. The authors stress its balance: strong gains at modest computing cost, which suits teams with limited budgets.

Catalog record

CREATORJian Wu 1,2 , Hang Yu 1 ∗ , Bingchang Liu 1 , Wenjie Yang 1 , Peng Di 1 , Jianguo Li 1 , and Yue Zhang 2 Email: {wujian, zhangyue}@westlake.edu.cn, {hyu.hugo, bingchang.lbc, ywj439780, dipeng.dp, lijg.zero}@antgroup.com 1 Ant Group 2 Westlake University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection Jian Wu 1,2 , Hang Yu 1 ∗ , Bingchang Liu 1 , Wenjie Yang 1 , Peng Di 1 , Jianguo Li 1 , and Yue Zhang 2 Email: {wujian, zhangyue}@westlake.edu.cn, {hyu.hugo, bingchang.lbc, ywj439780, dipeng.dp, lijg.zero}@antgroup.com 1 Ant Group 2 Westlake University Abstract Adapting large language models (LLMs) to specific domains often faces a critical bottleneck: the scarcity of high-quality, human-curated data. While large volumes of unchecked data are readily available, indiscriminately using them for fine-tuning risks introducing noise and degrading performance. Strategic data selection is thus crucial, requiring a method that is both accurate and efficient. Existing approaches, categorized as similarity-based and direct optimization methods, struggle to simultaneously achieve these goals. In this paper, we introduce LAMDAS ( L LM A s an i M plicit classifier for domain-specific DA ta S election), a novel approach that leverages the pre-trained LLM itself as an implicit classifier, thereby bypassing explicit feature engineering and computationally intensive optimization process. LAMDAS reframes data selection as a one-class classification problem, identifying candidate data that ”belongs” to the target domain defined by a small reference dataset. Extensive experimental results demonstrate that LAMDAS not only exceeds the performance of full-data training using a fraction of the data but also outperforms nine state-of-the-art (SOTA) baselines under various scenarios. Furthermore, LAMDAS achieves the most compelling balance between performance gains and computational efficiency compared to all evaluated baselines. 1 1 1 Code will be available upon publication. 1 Introduction The unprecedented capabilities of modern large language models (LLMs) rely on their ability to learn from vast and diverse datasets during pre-training, followed by adaptation to specialized domains or tasks via continual pre-training (CPT) and supervised fine-tuning (SFT). However, a key challenge arises when adapting LLMs to new scenarios: while high-quality, human-curated data is often scarce, large volumes of unchecked, automatically collected data, such as web-scraped content, crowd-sourced annotations, or synthetic examples, are readily accessible. Indiscriminately using this unchecked data for CPT or SFT carries the risk of introducing noise, misalignment with the target task, or even harmful patterns, leading to degraded model performance and increased hallucinations ( Zhang et al. 2023 ; Li et al. 2023 ) . Furthermore, training LLMs on unnecessarily large datasets incurs significant computational costs, requiring substantial GPU resources that are not readily available to all researchers and practitioners ( He et al. 2024 ) . It is therefore imperative to strategically select the most relevant training examples from the vast pool of unchecked candidate data, particularly when working with a small but meticulously curated reference dataset that accurately represents the target domain or task. The core challenge in domain-specific data selection lies in identifying a method that is both accurate – maximizing LLM performance on the reference dataset after training on the selected data – and efficient – capable of processing massive candidate datasets, especially for CPT. (a) (b) Figure 1: Performance gains versus data selection efficiency (selection time) for all methods in both (a) CPT and (b) SFT scenarios. A larger perpendicular distance, indicated by the dashed lines, from a method to the regression line—and above the regression line—indicates a more favorable trade-off. The regression line is estimated using linear regression on the x and y values of all methods—suggests a more favorable trade-off. The zigzag lines mean scale breaks. Unfortunately, existing approaches often struggle to achieve both of these goals simultaneously. As discussed in Section 2 , the current literature on domain-specific data selection can be divided into two primary groups: similarity-based methods and direct optimization methods. Similarity-based methods typically extract features from both candidate and reference data, selecting candidate data based on their similarity to the reference data according to specific measures. Within this category, methods using easily obtained features like lexicon or embedding-based representations ( Liu et al. 2024c ; Xie et al. 2023 ; Thrush et al. 2025 ) and computationally cheap measures like cosine similarity or correlation are fast. However, selecting data based on such superficial characteristics can be ineffective or even detrimental to model performance, as pointed out in ( Engstrom et al. 2024b ) . To tackle this issue, more recent methods utilize more informative features, such as gradients ( Gu et al. 2025 ; Yu et al. 2024 ) or model weights ( Xia et al. 2024 ; Huang et al. 2025 ) , and employ advanced measures like optimal transport ( Liu et al. 2024c ) and KL reduction ( Xie et al. 2023 ) , achieving improved performance gains at the cost of efficiency. On the other hand, direct optimization methods aim to enhance the LLM’s performance on the reference data by selecting suitable subsets of candidate data based on frameworks like optimal control theory ( Gu et al. 2025 ) , data models ( Engstrom et al. 2024b ) , conditional loss ( Brandfonbrener et al. 2024 ) , or Shapley values ( He et al. 2024 ) . However, the high computational burden of these methods typically requires them to estimate selection scores for only a representative subset of the candidate data. A smaller or simplified model is then trained to extrapolate these scores to the entire candidate dataset, adding a layer of approximation that can degrade performance. In essence, these techniques often remain computationally expensive. In this paper, we introduce LAMDAS , innovatively using L LM A s an i M plicit classifer for domain-specific DA ta S election. The proposed method is applicable to both data selection for CPT and SFT. Our approach effectively addresses the trade-off between performance and efficiency. The key insight is to reframe data selection as a one-class classification (OCC) problem: given a small reference dataset representing the target domain (positive samples), how can we identify candidate data that ”belongs” to the same class or domain? Unlike prior works, LAMDAS leverages the pre-trained LLM itself as an implicit classifier, bypassing the need for explicit feature engineering or computationally intensive retraining. We first demonstrate that a prefix tuned on the reference dataset acts as a concise representation of the target domain – a ”domain prefix”. LAMDAS scores candidate examples by comparing their likelihood under the LLM with and without this domain prefix. Candidates exhibiting significantly higher likelihoods when conditioned on the prefix are prioritized, indicating a stronger alignment with the reference distribution. Since prefix tuning only modifies the prefix tokens while leaving the LLM’s weights untouched, LAMDAS effectively preserves the LLM’s general knowledge and prevents overfitting to the limited reference data. Moreover, given the relative simplicity of binary classification, a smaller LLM can often achieve good classification performance (as shown in our experiments), thereby ensuring the efficiency of our approach. Extensive experimental results demonstrate that LAMDAS not only surpasses full data training while utilizing only a small fraction of the data but also outperforms seven SOTA baselines for CPT data selection and nine baselines for SFT data selection. Moreover, as depicted in Figure 1 , LAMDAS achieves the most favorable trade-off between performance gains and efficiency among all baselines evaluated. In summary, the key contributions of this work are: • We innovatively formulate the domain-specific data selection problem as an OCC problem. • We propose

Read the complete text