Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder
Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Karn, Rui Hou, Madian Khabsa, Yuning Mao · 2025
Read Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder by Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Karn, Rui Hou, Madian Khabsa, Yuning Mao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14050.txt.
Introduction
Xianjun Yang and co-authors tackle data choice for tuning language models in a 2025 paper. Tuning sets are now vast, so picking the right subset matters, yet most pickers chase quality alone. This team argues variety and complexity count just as much, and they need a good way to measure variety.
Their tool is a sparse autoencoder, a network that learns a compact code for each text. Codes make variety easy to gauge and also explain odd findings, such as why picking the longest replies can work well. Models trained on their chosen subset beat rival methods, cost less to train, and gave the team more control over behaviour.
Catalog record
| CREATOR | Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Karn, Rui Hou, Madian Khabsa, Yuning Mao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14050
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Karn, Rui Hou, Madian Khabsa, Yuning Mao Abstract Instruction tuning data are often quantity-saturated due to the large volume of data collection and fast model iteration, leaving data selection important but underexplored. Existing quality-driven data selection methods, such as LIMA (NeurIPS 2023 ( Zhou et al. 2024 ) ) and AlpaGasus (ICLR 2024 ( Chen et al. ) ) generally ignore the equal importance of data diversity and complexity. In this work, we aim to design a diversity-aware data selection strategy and creatively propose using sparse autoencoders (SAEs) to tackle the challenge of data diversity measure. In addition, SAEs can also provide more interpretability of model behavior and explain, e.g., the surprising effectiveness of selecting the longest response (ICML 2024 ( Zhao et al. ) ). Using effective data selection, we experimentally prove that models trained on our selected data can outperform other methods in terms of model capabilities, reduce training cost, and potentially gain more control over model behaviors. We prove that SAEs can serve as a good alternative to diversity measure and design our method to be scalable for potential industrial large-scale pruning, and we will also release our trained SAEs for use by the broader community. † † affiliation: Meta GenAI † † date: August 9, 2026 † † correspondence: [email protected] , [email protected] 1 Introduction Scaling large language models (LLMs) has been shown to significantly enhance performance ( Kaplan et al. 2020 ; Achiam et al. 2023 ) , augmented by alignment ( Bai et al. 2022 ; Rafailov et al. 2024 ; Schulman et al. 2017 ) to make LLMs follow human instructions. Instruction fine-tuning (IFT) ( Wei et al. ; Longpre et al. 2023 ; Sanh et al. ; Ouyang et al. 2022 ) has become an essential step in adapting LLMs to perform effectively across diverse tasks, and it is believed that the data quality and diversity are the most important factors during IFT. Recently, there is a growing interest on data-centric AI. For example, innovations in data engineering enable scaling to vast contexts, extending the model’s capacity to handle extensive data sequences ( Fu et al. ) . Alpaca ( Taori et al. 2023b ) can elicit Llama’s instruction-following by distilling 52 k instruction conversations from ChatGPT ( OpenAI 2023 ) . Importance resampling for selection has been developed to optimize data relevance ( Xie et al. 2023 ) . Although it is well acknowledged that the critical to effective instruction tuning is the training data’s quantity ( Ding et al. 2023 ) , quality ( Chen et al. ) , diversity ( Bukharin and Zhao 2023 ) , and complexity ( Wang et al. 2023 ) , as also highlighted by recent industrial technical reports, such as Llama-3 ( Dubey et al. 2024 ) and QWen-2 ( Yang et al. 2024 ) , it is still a mystery of how to accurately measure those features in data. Generally, ensuring data quantity and quality are easier to achieve with either human feedback ( Ouyang et al. 2022 ) or automated AI feedback ( Lee et al. ) . However, evaluating data diversity and complexity are challenging. For instance, a recent call ( Zhao et al. 2024 ) urge a quantifiable approach to dataset diversity to ensure that claims of diversity are substantiated and meaningful: " Measure Dataset Diversity, Don’t Just Claim It ". Previous work has proved the effectiveness of small data quantity for achieving either general instruction following ( Chen et al. 2023 ; Zhou et al. 2024 ) , or task-oriented abilities ( Xia et al. ) . Zhang et al. 2024a shows some preliminarily results to claim that instruction diversity drives generalization to unseen scenarios through symbolic tasks. And Liu et al. perform controlled studies to measure data across three dimensions: complexity, quality, and diversity (measured by cosine similarity of sentence representation). Besides, data pruning based on concept complexity has been shown effective in both pre-training ( Abbas et al. ) and post-training ( Lu et al. ) . However, many previous text diversity metrics ( Shaib et al. 2024 ) are not adopted due to their intrinsic limitations. Therefore, there is still a lack of reliable diversity measure toward more efficient, effective, and targeted data selection in the instruction tuning landscape. And it is also unclear about the underlying mechanism such as what contributes to an accurate data diversity measure. Recently, SAEs ( Cunningham et al. 2023 ) has emerged as a powerful tool for interpreting the activated features in language models. The sparsity of the atomic monosemanticity guarantees the features are highly independent and accurate. Inspired by this advance, we first train an SAE and then propose a new approach to use the activated features from SAE to measure data diversity . Depending on the number of selected data, we then proposed two algorithms for effective data selection. The first one is greedy sampling using features from SAE for limited data ( SAE-GreedSelect ) and the second one is similarity-based sampling using features from SAE for scaling up the data selection ( SAE-SimScale ). We conduct comprehensive experiments on the widely used Alpaca ( Taori et al. 2023a ) and WizardLM_evol_instruct_70k ( Xu et al. 2023 ) datasets and our approach witnesses significant superiority over several previous results such as #InsTag ( Lu et al. ) , Longest ( Zhao et al. ) and Repr Filter ( Liu et al. ) across various models and data scale. Our approach also explains why some previous methods work by looking at the extracted features in the selected data. In summary, we propose a novel method to use SAEs for diversity measure, and design new algorithms for diversity-driven data selection, contributing to more effective and interpretable instruction tuning. 2 Related Work Sparse Autoencoders (SAEs) are powerful for understanding neural representations ( Gao et al. 2024 ; Paulo et al. 2024 ; Braun et al. 2024 ) , as well as in scaling and enhancing the interpretability of LLMs ( Cunningham et al. 2023 ; Foote et al. 2023 ) . The foundational work in sparse coding, also known as dictionary learning, dates back to an overcomplete basis set proposed around 30 years ago ( Olshausen and Field 1997 ) . Based on this, K-SVD ( Aharon et al. 2006 ) was developed as an algorithm for designing overcomplete dictionaries, facilitating sparse representations in signal processing. Furthermore, K-SAEs ( Makhzani and Frey 2013 ) introduced a K-sparse constraint to enforce sparsity, leading to more accurate data representations. Bau et al. 2020 highlights the importance of understanding unit-specific functions for the interpretability of the model. Furthermore, Tonolini et al. 2020 proposes that the visual cortex can employ such strategies to efficiently represent natural images. Recently, sparse dictionary learning has been applied to visualize transformer models ( Elhage et al. 2022 ; Henighan et al. 2023 ) , revealing that contextualized embeddings can be expressed as linear combinations of transformer factors ( Yun et al. 2021 ) . Research has also investigated polysemanticity and capacity in neural networks, finding that neurons often encode multiple concepts,