FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation

Chaitali Bhattacharyya, Yeseong Kim · 2025

Read FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation by Chaitali Bhattacharyya, Yeseong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00624.txt.

Introduction

Chaitali Bhattacharyya and co-authors shrink big language models for special fields in a 2025 paper. Training a domain model from scratch is costly, and plain pruning of a large model dulls its skill. Their framework, FineScope, uses sparse autoencoders to read which features a domain needs. Structured pruning then cuts the model under domain constraints, keeping what the field requires.

Lost skill is restored by fine-tuning on data the autoencoder itself curates from the domain. The pruned models rival far larger systems on specialty tasks, they report. The curated data even sharpens unpruned models. The route promises cheaper, sharper tools for narrow jobs.

Catalog record

CREATORChaitali Bhattacharyya, Yeseong Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation Chaitali Bhattacharyya, Yeseong Kim Abstract Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA ( Touvron et al. 2023b ; Touvron et al. 2023a ) have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations ( Templeton et al. 2024 ) , to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach. The code will be released. 1 Introduction Transformer-based Large Language Models (LLMs) have demonstrated remarkable generalization across diverse natural language processing (NLP) tasks by leveraging broad, heterogeneous datasets ( Vaswani 2017 ) . However, deploying these models for domain-specific applications remains challenging. Foundation models are typically large, often with hundreds of billions to trillions of parameters ( RBC Borealis AI 2024 ; Adler et al. 2024 ; Dubey et al. 2024 ) , requiring substantial computational resources ( Kim et al. 2023 ; Dubey et al. 2024 ; Wang et al. 2024 ) . Additionally, general-purpose LLMs often fail to capture fine-grained domain knowledge due to their generic training data ( Xu et al. 2022 ; Nijkamp et al. 2022 ) . This challenge has driven the demand for domain-specialized, resource-efficient models that retain strong task performance while reducing computational costs ( Talmor et al. 2018 ; Clark et al. 2018 ; Gu et al. 2024 ) . Recent advances in model optimization techniques, such as pruning and parameter-efficient fine-tuning, have enabled the adaptation of LLMs to specialized domains with reduced resource overhead ( Ma et al. 2023 ) . However, these approaches rely heavily on the availability of high-quality domain-specific datasets, which are often scarce or expensive to curate. The bottleneck in domain adaptation thus lies not only in optimizing model architectures but also in constructing an effective domain-specific dataset from limited initial supervision. Given the wide range of potential application domains, manually curating datasets for every scenario is impractical, raising the critical question: How can we efficiently construct a domain-specific training dataset using minimal human supervision while maintaining adaptability and scalability? Recent work has shown that large-scale instruction tuning may not always be necessary for effective fine-tuning. For instance, LIMA ( Zhou et al. 2024 ) demonstrated that fine-tuning an LLM on just 1,000 carefully curated examples can yield competitive performance, suggesting that pretrained models already encode rich knowledge, and domain adaptation primarily requires high-quality, task-specific supervision rather than extensive data volume. However, obtaining such a high-quality dataset remains a bottleneck: existing methods either require manual annotation or rely on human experts, which are hard to be applicable across numerous domains requiring various language,e models. In this paper, we propose an alternative learning paradigm, self-data cultivation , which automates dataset construction by expanding a small, user-defined seed set into a high-quality, domain-relevant corpus . We hypothesize that a small, carefully chosen seed dataset can serve as the foundation for automatically constructing a domain-specific dataset. If we can curate high-quality training examples from a broader corpus using a small number of representative samples, we can significantly reduce the manual effort required for domain adaptation. However, dataset curation in this context is not merely selecting a subset – it must also refine the dataset to ensure seamless integration into the fine-tuning process. By structuring data in a way that aligns with the pretrained model’s learning dynamics, we can improve both adaptation efficiency and task-specific generalization. The effectiveness of self-data cultivation depends on overcoming several key technical challenges. First, extracting domain-relevant samples efficiently from a large corpus is non-trivial, as brute-force similarity search is computationally infeasible. Second, training an effective data selection model requires a mechanism that can generalize from a small seed set to retrieve semantically meaningful examples. Third, the dataset must ensure that fine-tuning on the curated dataset remains stable and effective, preserving the generalization capabilities of the pretrained model while improving task-specific performance. To address these challenges, we introduce FineScope , a novel self-data cultivation framework that learns computation-efficient domain-specific language models by automating domain-specific dataset construction. FineScope utilizes a Sparse Autoencoder (SAE) to learn a compressed representation of domain-relevant features ( Templeton et al. 2024 ; Kissane et al. 2024 ; Yan et al. 2024 ) . Unlike typical SAEs which model all neuron activities, FineScope learns our SAE using only top- k activations, which selects semantically similar data points from a large, general-purpose corpus, forming a domain-optimized dataset. It allows FineScope to learn latent domain structures rather than static embedding similarity, making it more scalable and adaptable across diverse application domains. The curated dataset is then used to fine-tune a pruned domain-specialized LLM , ensuring both computational efficiency and high task relevance. To aid the fine-tuning process of the pruned models, the FineScope further refines the curated dataset using a self-data distillation technique ( Yang et al. 2024 ) , allowing the models with fewer parameters to learn the target domain knowledge seamlessly. We evaluate FineScope across multiple domain-specific NLP tasks and demonstrate that it achieves strong domain adaptation while significantly reducing the computational cost. Our results show that models fine-tuned with SAE-curated datasets consistently outperform conventional fine-tuning approaches, achieving higher accuracy with smaller model sizes. Furthermore, FineScope enables fine-grained multi-domain specialization, allowing LLMs to adapt to different subdomains without retraining from scratch. For example, testing on the Math dataset showed an average performance improvement of 11.45 across all models. Our key contributions are as follows: • We introduce FineScope , a novel framework for self-supervised domain dataset curation , enabling scalable domain adaptation from a minimal seed dataset. • We formally propose the concept of self-data cultivation , hypothesizing that a small, representative seed dataset can be expanded into a high-quality domain corpus with minimal supervision. • We develop an SAE-guided dataset selection mechanism , which learns latent domain structures and selects high-relevance training samples efficiently. • We demonstrate that structured pruning with curated datasets yields efficient, domain-specialized LLMs that retain high task performance while reducing computational cost. 2 Related Work 2.1 Domain-Specific Language Models Domain-adapted LLMs have been developed to enhance performance in specialized fields by training on domain-specific datasets. Notable examples include PharmaGPT-13B/70B ( Chen et al. 2024 ) for biopharmaceutical and chemical applications, SaulLM-54B/141B ( Colombo et al. 2024 ) for legal tasks,

Read the complete text