On the Difficulty of Learning a Meta-network for Training Data Selection

Zilin Du, Junqi Zhao, Boyang Albert Li · 2026

Read On the Difficulty of Learning a Meta-network for Training Data Selection by Zilin Du, Junqi Zhao, Boyang Albert Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00571.txt.

Introduction

Du, Zhao, and Li name why a meta-net that picks train data by worth is hard to train well. Fake data now train many nets, yet their spread can clash with real data. A meta-net can weigh each fake case by its worth. In live use, those picks most oft fall short of hopes.

Two walls block the picks, the team find. Slope marks for worth are too noisy to learn from well. Traits that track data worth are also thin on the ground. Math on worth flow ties poor noise marks to split data worth. A plain fix is to grow the batch size in training. New traits log where a case sits in its spread and run. Tests on four sets lift scores past no-pick and top base runs.

Catalog record

CREATORZilin Du, Junqi Zhao, Boyang Albert Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE131 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00571

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Difficulty of Learning a Meta-network for Training Data Selection Zilin Du, Junqi Zhao, Boyang Albert Li Abstract Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49% over training without selection and 2.89% over the strongest baseline. 1 1 1 Our code is available at https://github.com/ZILIN003/MTS . 1 Introduction It is often desirable to selectively train on a subset of available data, especially when the training and test data are not identically distributed ( 27 ; 22 ; 34 ; 61 ) . A prominent use case scenario these days is training with synthetic data ( 10 ; 15 ; 30 ) , which has become increasingly practical as generative models improve. Nevertheless, synthetic data often exhibit distributional shifts or artifacts that distinguish them from real data ( 17 ; 18 ; 33 ) . A popular approach for training data selection is to assign soft weights w_{i} to each training sample (x_{i},y_{i}) ( 44 ; 48 ) in order to select effective training data or align the training and validation/test distributions. We define w_{i}:=s_{\phi}(x_{i}) , where s_{\phi} is a data selection network parameterized by \phi . The main network f_{\theta}(x) is parameterized by \theta . With training data points (x_{i},y_{i}) , validation data (x^{\text{v}}_{i},y^{\text{v}}_{i}) , and per-instance loss \ell(\cdot) , the mini-batch training loss and validation loss are defined as \displaystyle\mathcal{L}_{\text{train}}(\theta,\phi) \displaystyle:=\Big(\sum_{i=1}^{N}w_{i}\Big)^{-1}\sum_{i=1}^{N}w_{i}\ell(\theta,x_{i},y_{i}), (1) \displaystyle\mathcal{L}_{\text{val}}(\theta) \displaystyle:=N^{-1}\sum_{i=1}^{N}\ell(\theta,x^{\text{v}}_{i},y^{\text{v}}_{i}), (2) where N denotes the batch size. To optimize \phi , prior works adopt a one-step approximation of the complex bi-level optimization. At iteration t : \phi_{t+1}=\phi_{t}-\eta_{\phi}\nabla_{\phi_{t}}\mathcal{L}_{\text{val}}(\theta_{t}-\eta_{\theta}\nabla_{\theta_{t}}\mathcal{L}_{\text{train}}(\theta_{t},\phi_{t})) (3) and \eta_{\theta} and \eta_{\phi} are the learning rates. We refer to this framework as Meta-learning for Training-data Selection (MTS). Despite its intuitive appeal, MTS often performs under expectation. In Table 2 , we show that naive training of MTS with raw image input to s_{\phi}(\cdot) and small batches fails to outperform training without data selection. What causes performance issues of MTS? We identify two obstacles in MTS. First, the gradient signal-to-noise ratio (GSNR) of the selection network is exceedingly low (Figure 1 ). Though moderate levels of variance in the stochastic gradients can serve as implicit regularization ( 50 ; 39 ; 67 ) , excessive variance may overwhelm the gradient signal and harm optimization and generalization ( 46 ; 32 ) . We present a comprehensive analysis of the GSNR issue, which shows that the MTS training tends to suppress most data weights w_{i} , which in turn leads to high sensitivity of the gradient update to \phi to the highest data weight. This directly results in low GSNR. The analysis also presents a simple remedy: increasing the batch size. The second obstacle is that input features to the selection network do not correlate well with data effectiveness in training. Even if the network can be trained to convergence due to a good GSNR, it may still lack the information to distinguish between high-quality and low-quality data. We propose a set of informative features, representing the positions of training data in the training and validation distributions and their interaction with the training process. These distributional and dynamic features may be difficult to learn from the raw input x_{i} alone, as they may involve relations among multiple data points and the training process. Good GSNR and high-quality features are both necessary for effective training of the selection network and complementary to each other. Intuitively, high-quality features ensure good solutions exist in the hypothesis space, or the set of all functions representable by the selection network; decent GSNR allows us to find such good solutions quickly. Empirically on the task of synthetic data selection, we demonstrate substantial enhancement in MTS performance when the two solutions are combined, but weak or no improvements when they are employed in isolation. On 4 different datasets and 15 different settings, our method attains strong performance, surpassing the baseline without data selection by 5.49% and the strongest baseline by 2.89%. Figure 1 : The gradient signal-to-noise ratio (GSNR) of the selection network is lower than that of the ResNet classification network by roughly one order of magnitude. However, increasing the batch size can help. Early and middle training stages correspond to 20% and 60% of the total training steps. Results are computed as averages over 100 batches. Our contributions are summarized as follows: • We provide a mathematical analysis revealing a fundamental obstacle that hinder effective training of MTS: a low gradient signal-to-noise ratio (GSNR) in the selection network. The theoretical analysis identifies the cause of low GSNR and a simple and effective remedy: increasing the batch size. • We design informative features that correlate with the effectiveness of training data and allow the data selection network to function as intended. 2 Related Work 2.1 Synthetic Images for Image Classification Recent advances in diffusion models have made synthetic images a practical resource for training deep models ( 13 ; 60 ; 8 ) , and this capability has been widely leveraged in image classification ( 5 ; 21 ; 49 ; 53 ) . However, synthetic images often exhibit distributional shifts and artifacts relative to real data, which can degrade generalization if used indiscriminately ( 17 ; 18 ) . To mitigate these issues, prior work has largely focused on generation-side improvements, including fine-tuning generative models toward the target distribution ( 23 ; 1 ; 58 ) , conditioning on real images ( 20 ; 33 ; 55 ) , and prompt engineering strategies ( 62 ; 47 ; 9 ; 65 ) . 2.2 Weighing Training Data The notion of assigning distinct weights to data points in the training loss finds applications in diverse problems such as data pruning and domain adaptation. Data pruning aims to find a subset of training data that are as effective as the whole training set ( 54 ; 40 ; 41 ; 12 ; 59 ; 66 ) . This is equivalent to binary 0/1 weights. In domain adaptation, the data weights may help align the training data distribution with the validation distribution ( 56 ; 51 ; 61 ; 4 ; 6 ) , often with continuous weights, rather than 0/1 weights. A closely related line of work is the selection of synthetic training images, which often employs CLIP similarity ( 9 ; 2 ; 18 ; 65 ; 28 ; 58 ) or auxiliary classifier confidence ( 2 ; 26 ) .

Read the complete text