LiLAW: Lightweight Learnable Adaptive Weighting to Meta-Learn Sample Difficulty and Improve Noisy Training

Abhishek Moturu · 2025

Read LiLAW: Lightweight Learnable Adaptive Weighting to Meta-Learn Sample Difficulty and Improve Noisy Training by Abhishek Moturu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20786.txt.

Introduction

Noisy labels and mixed-quality data make neural network training unreliable. Abhishek Moturu and three co-authors introduce LiLAW in a 2025 paper on learning sample difficulty. LiLAW stands for lightweight learnable adaptive weighting, and it uses only three learned numbers.

Each training sample gets a loss weight based on its current difficulty, classed as easy, moderate, or hard. Three global parameters control that weighting. After each training batch, one step on a validation batch updates those parameters. The method needs no clean, unbiased validation set. The authors test it across general and medical image datasets, many noise types and levels, loss functions, and network designs, reporting steadier training than fixed weighting schemes.

Catalog record

CREATORAbhishek Moturu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20786

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiLAW: Lightweight Learnable Adaptive Weighting to Meta-Learn Sample Difficulty and Improve Noisy Training Abhishek Moturu Abstract Training deep neural networks in the presence of noisy labels and data heterogeneity is a major challenge. We introduce Lightweight Learnable Adaptive Weighting (LiLAW), a novel method that dynamically adjusts the loss weight of each training sample based on its evolving difficulty level, categorized as easy, moderate, or hard. Using only three learnable parameters, LiLAW adaptively prioritizes informative samples throughout training by updating these weights using a single mini-batch gradient descent step on the validation set after each training mini-batch, without requiring excessive hyperparameter tuning or a clean validation set. Extensive experiments across multiple general and medical imaging datasets, noise levels and types, loss functions, and architectures with and without pretraining demonstrate that LiLAW consistently enhances performance, even in high-noise environments. It is effective without heavy reliance on data augmentation or advanced regularization, highlighting its practicality. It offers a computationally efficient solution to boost model generalization and robustness in any neural network training setup. 1 Introduction The increasing availability of very large labeled datasets has played a major role in advancing machine learning and computer vision in recent years. However, imaging datasets often contain samples with varying levels of quality, affecting the efficiency and effectiveness of model training. This issue is particularly significant in medical imaging datasets, which typically have smaller sample sizes, exhibit greater heterogeneity, and often require specialized expertise for accurate labeling. Ensuring that models make the best use of noisy, heterogeneous data remains a key challenge, as not all samples are equally informative or beneficial for model performance. At varying rates at different points during training, samples can benefit the model, hurt the model, or not affect the model much at all. Several methods have been proposed to quantify the difficulty or importance of individual samples ( Agarwal et al. 2022 ; Paul et al. 2021 ; Baldock et al. 2021 ; Maini et al. 2022 ; Siddiqui et al. 2022 ; Rabanser et al. 2022 ; Toneva et al. 2018 ; Pliushch et al. 2022 ; Xu et al. 2021 ; Kong et al. 2021 ; Dong 2023 ; Jiang et al. 2020 ; Seedat et al. 2024b ) . Understanding which samples a model finds difficult to predict is essential for safe model deployment, sample selection for human-in-the-loop auditing, and gaining insights into model behavior ( Agarwal et al. 2022 ) . Knowing or estimating example difficulty can help separate misclassified, mislabeled, and rare examples ( Maini et al. 2022 ) , prune data ( Paul et al. 2021 ) , quantify uncertainty ( Dong 2023 ) , improve generalization ( Xu et al. 2021 ) , increase convergence speed ( Kong et al. 2021 ) , enhance out-of-distribution detection ( Agarwal et al. 2022 ) , and provide insights into example memorization and forgetting ( Toneva et al. 2018 ) . While existing methods provide insights into individual data points and how models learn, they often do not adjust the training process in a dynamic, efficient manner. This may restrict potential improvements in model performance, robustness, and generalization. Focusing on the most relevant data at different stages of training can help leverage datasets effectively. A common limitation in many existing approaches is the static or uniform treatment of all data points during training. Standard practice does not apply additional weights to samples or take sample difficulty into account. Existing sample weighting methods usually require a clean validation set, a separate model to learn sample difficulty, and/or a higher training complexity (see Section 2.2 ). In this paper, we propose a novel method called Li ghtweight L earnable A daptive W eighting ( LiLAW ), which addresses these challenges with a lightweight, adaptive mechanism to adjust the weight of each sample loss based on its difficulty. Our approach introduces a simple yet powerful parametrization, using three parameters pertaining to learning easy, moderate, and hard samples (see Figure 1 ). These parameters are used to calculate the weight of each sample loss and are jointly optimized during training with a single gradient descent step on the validation set after every training mini-batch. Figure 1: Given noisy training and validation data, LiLAW learns to adaptively weight the loss of each sample based on three trainable parameters, \alpha,\delta,\beta , pertaining to learning easy , moderate , and hard samples, respectively, at different stages of training using meta-learning on the validation data. Importantly, our method does not require the validation set to be clean and unbiased. Unlike static weighting schemes or hyperparameter-based weighting search schemes, LiLAW learns to adaptively prioritize different samples as training progresses. By integrating sample difficulty assessments into the training process, LiLAW helps the model focus on the most informative data when it is most beneficial. This approach offers a flexible solution for incorporating sample difficulty into any training process for classification problems, while maintaining computational efficiency. We first introduce our theoretical framework and then perform extensive experiments with and without LiLAW in various settings. Our results show that LiLAW improves test performance across several general and medical imaging datasets, models, loss functions, noise types, noise levels, with and without pretraining, with and without calibration, and with and without a clean and unbiased validation set. This is done without the need for excessive hyperparameter tuning and without increasing the time and space complexity. In summary, we introduce a new method, LiLAW, that: – only adds three extra trainable parameters and does not require more models for difficulty-learning or pruning (easy-to-implement and maintains similar space complexity); – only needs one additional forward and backward pass on a single validation mini-batch after each training mini-batch (maintains similar time complexity); – uses weighting to measure difficulty, inform model training, and improve test performance without pruning (avoids reducing dataset size, which may already be small in certain settings); – improves noisy learning with label and input noise (prevalent in medical imaging datasets); – uses meta-learning to avoid the need for extensive hyperparameter tuning (saves time); – does not require a clean, unbiased meta-validation set (may be hard to obtain in health domain); – offers an intuitive graphical and mathematical understanding of the weighting; and, – is adaptive, lightweight, and easy-to-implement on top of any model training/fine-tuning. 2 Related Work The effectiveness of machine learning models, especially deep neural networks, heavily depends on the quality and proper use of training data. Recent literature has explored strategies to enhance model performance and robustness to address challenges brought forward by noisy, mislabeled, underrepresented, or otherwise hard-to-learn data. 2.1 Data-Centric AI Pleiss et al. 2020 proposed using the area under the margin ranking to identify mislabeled data and improve model performance by filtering out mislabeled data, which requires “dataset cleaning.” Toneva et al. 2018 conducted an empirical study on example forgetting during deep neural network training that shed light on data quality highlighting that some samples that are forgotten frequently, some are not forgotten at all, and some could be omitted from training without affecting the model greatly. Paul et al. 2021 highlight that simple scores such as the Gradient Normed (GraNd) and Error L2-Norm (EL2N) can be used to identify important examples very early in training and

Read the complete text