A Survey on Efficient Large Language Model Training: From Data-centric Perspectives
Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang · 2025
Read A Survey on Efficient Large Language Model Training: From Data-centric Perspectives by Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25817.txt.
Introduction
Junyu Luo leads ten co-authors in a survey of how to train large language models with less data after their first training phase. Post training is the extra training that helps a model follow tasks and fit a field. Good post training data is costly to label by hand, and adding more data helps less and less over time.
Luo and the team sort data saving methods into five groups, from picking and improving data to making synthetic data and compressing data sets. They sum up leading work in each group and point to open problems and future paths. The survey helps readers see how to use data well at large scale.
Catalog record
| CREATOR | Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25817
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Survey on Efficient Large Language Model Training: From Data-centric Perspectives Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang Abstract Post-training of Large Language Models (LLMs) is crucial for unlocking their task generalization potential and domain-specific capabilities. However, the current LLM post-training paradigm faces significant data challenges, including the high costs of manual annotation and diminishing marginal returns on data scales. Therefore, achieving data-efficient post-training has become a key research question. In this paper, we present the first systematic survey of data-efficient LLM post-training from a data-centric perspective. We propose a taxonomy of data-efficient LLM post-training methods, covering data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. We summarize representative approaches in each category and outline future research directions. By examining the challenges in data-efficient LLM post-training, we highlight open problems and propose potential research avenues. We hope our work inspires further exploration into maximizing the potential of data utilization in large-scale model training. 1 Introduction Large Language Models (LLMs) post-training has emerged as a crucial stage for unlocking their domain adaptation capabilities and task generalization potential Luo et al. 2025b . This phase has effectively enhanced models’ abilities in long-context reasoning Zelikman et al. 2022 ; Yuan et al. 2024c , human alignment Rafailov et al. 2024 , instruction tuning Zhang et al. 2023b , and domain-specific adaptation Cheng et al. 2024 . Figure 1: Illustration of the data flywheel in Data-Efficient LLM Post-Training , depicting the iterative cycle of data selection, data quality enhancement, synthetic data generation, knowledge distillation, and self-evolving data ecosystems to maximize model performance with minimal data requirements. During the LLM post-training phase, data is the essential driver of model evolution. However, the current paradigm faces a severe data dilemma : the cost of manually annotating high-quality data is rapidly growing, while simply scaling data volume yields diminishing returns. Moreover, static datasets inherently limit models from adapting to evolving real-world knowledge. The linear dependency between data volume and model performance fundamentally stems from the inefficient data usage in traditional post-training paradigms. The recent success of DeepSeek-R1 Guo et al. 2025 , which leverages reinforcement learning for data-efficient post-training, further demonstrates the effectiveness and necessity of data-efficient approaches in achieving superior LLM performance. Our work establishes the first systematic survey on data-efficient training of LLMs, providing a unified, taxonomized framework to address the fragmented research landscape. Our survey reveals that breaking through efficiency bottlenecks requires establishing value extraction across the lifecycle, rather than merely expanding data scale. Researchers have explored various approaches to fully exploit the data potential in LLM post-training Jeong et al. 2024 ; Wang et al. 2024a ; Pan et al. 2024b . While these methods have made notable progress in improving data efficiency, the field still lacks a comprehensive review. In this paper, we provide a comprehensive survey of data-efficient LLM post-training from a data-centric perspective. Specifically, we introduce the concept of a data value flywheel (as illustrated in Figure 1 ), which consists of five key components: data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. Using this framework, we present a taxonomy of existing work, summarize key components, and identify promising research directions. We hope our work serves as both a useful roadmap for newcomers and a guide for future advancements in the field. Differences from previous surveys. While several surveys have explored a few aspects of LLMs post-training, including data selection Wang et al. 2024b , synthetic data generation Long et al. 2024 ; Tan et al. 2024 , model self-feedback Liang et al. 2024a ; Pan et al. 2023 , self-evolution Tao et al. 2024 , trustworthiness Liu et al. 2023 , and time-efficiency Wan et al. 2023 , these studies primarily focus on individual aspects rather than a holistic perspective. Our survey fills this gap by systematically examining these methods through the lens of data efficiency, offering critical insights into maximizing data value extraction. {forest} Figure 2: A taxonomy of Data-Efficient LLM Post Training. 2 Taxonomy This section categorizes data-efficient post-training methods for LLMs into five core methodologies: • Data Selection : Filtering high-value subsets from raw data. ❶ Static Filtering: Offline selection based on data properties; ❷ Dynamic Selection: Adjusting weights based on model uncertainty; ❸ Agent Strategy: Multi-model voting for reliable selection; ❹ Labeling Efficiency: Combining active learning and semi-supervised strategies for cost-effective sample coverage. • Data Quality Enhancement : Improving the utility of existing data. ❶ Semantic Rewriting: Enhancing expression diversity through semantic-preserving transformations and generating variants while maintaining original meaning; ❷ Toxicity Control: Correcting harmful content; ❸ Distribution Stabilization: Adjusting data characteristics for robustness • Synthetic Data Generation : Creating new training data. ❶ Instruction-Driven: Model-generated instruction-response pairs; ❷ Knowledge-Guided: Generation with structured knowledge; ❸ Adversarial Generation: Producing challenging samples. • Data Distillation and Compression : Extracting core knowledge for efficient training. ❶ Model Distillation: Transferring large model output distributions to smaller models while preserving key knowledge; ❷ Data Distillation: Extracting high information density samples to construct compact datasets equivalent to full-scale data; ❸ Joint Compression: Combining model architecture compression with data selection strategies for end-to-end efficiency optimization • Self-Evolving Data Ecosystem : Building self-evolution mechanisms. ❶ Self-Iterative Optimization: Using current model to generate data; ❷ Dynamic Evaluation Feedback: Real-time monitoring and adjustment; ❸ LLM-as-a-Judge: Feedback-Driven Data Optimization; Category Data Compute Model Data Value Dependency Cost Dependency Mining Data Selection ++ + + +++ Quality Enhance. ++ ++ ++ ++ Synthetic Generation + +++ +++ + Distill. & Compress. + + +++ +++ Self-Evolving + +++ +++ +++ Table 1: Comparison of different data-efficient post-training methods across key dimensions. Table 1 compares the five methodologies across key dimensions, where more ’+’ indicates higher requirements or better performance. Data selection shows high data efficiency but requires quality source data. Quality enhancement maintains balanced requirements across dimensions. Synthetic generation and self-evolving approaches demand more compute and model resources but reduce dependency on original manually annotated datasets, as they primarily rely on teacher model outputs or self-generated data. Distillation methods excel in data efficiency while depending on model capabilities. These five dimensions complement each other: selection filters quality data, enhancement improves utility, generation expands coverage, distillation concentrates knowledge, and self-evolution enables continuous improvement. Together, they pursue the goal of less data, higher returns . 3 Data Selection Data selection is crucial for enhancing LLM post-training efficiency by identifying high-value data subsets. As shown in Figure 3 , we divide existing approaches into four dimensions: (1) static filtering based on inherent data properties, (2) dynamic selection that adapts during training, (3) agent strategy using collaborative mechanisms, and (4) labeling efficiency through human-AI collaboration. 3.1 Static Filtering Static filtering evaluates inherent data properties offline to identify samples with high information density and representativeness. Quality-based Filtering. Alpagasus Chen et al. 2023 achieves comparable performance using only 17% of the original data through complexity-based filtering (instruction length, diversity, and perplexity). MoDS Du et al. 2023 employs multi-dimensional indicators and density peak clustering, while Kang et al. 2024 uses KL-divergence-driven selection to align domain distributions. Information-theoretic approaches Kim and Baek 2024 leverage entropy