Data-Efficient Training by Evolved Sampling
Ziheng Cheng, Zhong Li, Jiang Bian · 2025
Read Data-Efficient Training by Evolved Sampling by Ziheng Cheng, Zhong Li, Jiang Bian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23461.txt.
Introduction
Training on every data point costs time that smarter sampling can save. Ziheng Cheng and 2 co-authors propose Evolved Sampling, a method that picks data batch by batch while training runs. The picks rest on loss values and on how losses change when data are altered. How often the picks refresh can be tuned, which cuts backward-pass time while keeping model quality.
A second version adds set-level pruning for still more speed. Across pre-training and post-training tasks, the framework sped training with no loss in results, saving up to nearly half the wall-clock time. The authors present it as a plug-in that fits many training setups.
Catalog record
| CREATOR | Ziheng Cheng, Zhong Li, Jiang Bian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23461
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data-Efficient Training by Evolved Sampling Ziheng Cheng, Zhong Li, Jiang Bian Abstract Data selection is designed to accelerate learning with preserved performance. To achieve this, a fundamental thought is to identify informative data samples with significant contributions to the training. In this work, we propose Evolved Sampling ( ES ), a simple yet effective framework for dynamic sampling along the training process. This method conducts batch level data selection based on the dynamics of losses and augmented loss differences , which enables flexible frequency tuning , and hence significantly reduces the back propagation time with maintained model performance. Due to its conciseness, ES is also readily extensible to incorporate set level data selection (to form ES with pruning, ESWP ) for further accelerations. As a plug-and-play framework, ES(WP) consistently achieves lossless training accelerations across various pre-training and post-training tasks, saving up to nearly 45% wall-clock time. Our results motivate further investigations on the data efficiency aspect of modern large-scale machine learning. 1 Introduction Deep learning has showcased remarkable performance across a variety of real-world applications, particularly leading to unparalleled successes of large “foundation” models ( Touvron et al. 2023 ; Rombach et al. 2022 ). On the other hand, since these large models are usually trained on web-scale datasets, the overall computation and memory loads are considerably increasing and unsustainable, calling for more efficient developments of modern large-scale machine learning. Efficient learning involves several aspects, centering around models, data, optimization, systems, and so on ( Shen et al. 2023 ). For data -efficient machine learning, the core is to properly evaluate the importance per data sample in the original (large) datasets. A broad array of methods is applied in a static manner, or known as the offline (coreset) selection, where the samples’ importance is determined before the formal training. By leveraging feature representations of data ( Swayamdipta et al. 2020 ; Xie et al. 2023b ), this importance can be either evaluated based on a variety of metrics such as distances ( Huang et al. 2023 ; Xia et al. 2023 ; Abbas et al. 2023 ), uncertainties ( Coleman et al. 2020 ; Margatina et al. 2021 ), errors ( Toneva et al. 2019 ; Paul et al. 2021 ), etc., or learned via procedures from the meta optimization ( Killamsetty et al. 2021c ; Killamsetty et al. 2021b ; Jain et al. 2024 ; Wang et al. 2022 ) and dataset distillation ( Nguyen et al. 2021 ; Wang et al. 2022 ; Zhao and Bilen 2023 ), or directly assessed by LLMs ( Sachdeva et al. 2024 ). See more detailed discussions in Appendix A . However, these approaches can be prohibitively expensive to apply in practice, since their potential dependence on feature representations requires additional (pre-)training in advance. Another array of methods lies in a dynamic sense, or known as the online (batch) selection, where the samples’ importance is simultaneously evaluated along the training process. Dynamic sampling methods can be further divided into two categories: set level selection, to prune the whole dataset at the beginning of each epoch ( Qin et al. 2024 ; Raju et al. 2021 ; Thao Nguyen et al. 2023 ; Attendu and Corbeil 2023 ), and batch level selection, to sample subsets from original batches for back propagation ( Kawaguchi and Lu 2020 ; Katharopoulos and Fleuret 2017 ; Katharopoulos and Fleuret 2018 ; Mindermann et al. 2022 ). Nevertheless, these dynamic sampling methods leverage similar strategies to evaluate the samples’ importance. Based on the naive intuition that samples’ contributions to the learning are directly associated with gradient updates, it is natural to re-weight data samples with scales of gradients or losses during training. Sampling methods based on the gradients ( Hanchi et al. 2022 ; Wang et al. 2024b ; Gu et al. 2025 ; Wang et al. 2025 ; Wang et al. 2024a ) usually suffer from significant computation and memory loads. Sampling methods based on the loss dynamics can involve current losses ( Jiang et al. 2019 ; Loshchilov and Hutter 2016 ; Qin et al. 2024 ; Thao Nguyen et al. 2023 ; Kumar et al. 2023 ; Balaban et al. 2023 ) and historical losses ( Attendu and Corbeil 2023 ; Raju et al. 2021 ; Sagawa et al. 2020 ) and also adopt reference models ( Mindermann et al. 2022 ; Deng et al. 2023 ; Xie et al. 2023a ). See more detailed discussions in Appendix A . However, these approaches exploit the information of losses inadequately by only involving “absolute” loss values, without finer considerations on their dynamical “variations” during training. To tackle these issues, we propose a simple novel dynamic sampling framework, Evolved Sampling ( ES ). Unlike previous sampling methods, ES determines the importance/weights of data samples based on both (zero-order) losses and additional (first-order) loss differences along the training dynamics. By augmenting and balancing these two orders, ES can flexibly tune the portion of oscillations (high frequencies) presented in loss signals , and conducts batch level selection without the demand of pre-trained reference models. Importantly, ES employs an equivalent dynamical scheme to compute sampling weights without explicitly storing historical losses , and only computations regarding losses are involved to implicitly calculate the required loss differences , implying the negligible memory costs and mild computation overhead additionally introduced by weight calculations. Due to its simplicity, ES is effortless to implement, while significantly reducing the number of samples used for back propagations (BPs) and consequently saving the overall wall-clock time without degrading the overall performance. Moreover, ES facilitates convenient extensions to data pruning on the set level, i.e., Evolved Sampling with Pruning ( ESWP ), leading to further accelerations with lossless learning performance. We demonstrate the differences in details between our proposed methods (ES/ESWP) and previous dynamic sampling methods in Table 1 . Table 1 : Comparison of different dynamic sampling methods. The “history” denotes whether the method uses historical (loss) information along the training. The “dif” column stands for whether the method uses dynamical variations of losses during the training. The last column summarizes the ratio of samples used for back propagations (BPs) relative to the standard training. Here, r stands for the pruning ratio for set level methods (pruning data samples of the whole epoch), and b/B represents the pruning ratio for batch level methods (selecting a mini-batch \mathfrak{b} (subset) from a meta-batch \mathcal{B} ). set batch history dif # of samples for BP UCB ( Raju et al. 2021 ) ✓ ✓ 1-r KA ( Thao Nguyen et al. 2023 ) ✓ 1-r InfoBatch ( Qin et al. 2024 ) ✓ 1-r Loss ( Katharopoulos and Fleuret 2017 ) ✓ b/B Order ( Kawaguchi and Lu 2020 ) ✓ b/B ES (ours) ✓ ✓ ✓ b/B ESWP (ours) ✓ ✓ ✓ ✓ \bm{(1-r)b/B} Our contributions can be summarized as follows: • On the theoretical side, we provide quantitative convergence analysis of the loss re-weighted gradient descent (GD) under idealized settings. Motivated by this, we propose a simple novel dynamic sampling framework ES(WP) that can implicitly incorporate (and balance) additional dynamical differences of losses without explicitly storing historical values and calculating variations . By further injecting higher-order dynamical information,