Step-E: A Differentiable Data Cleaning Framework for Robust Learning with Noisy Labels

Du Wenzhang · 2025

Read Step-E: A Differentiable Data Cleaning Framework for Robust Learning with Noisy Labels by Du Wenzhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17040.txt.

Introduction

Wenzhang Du addresses a common flaw in training data: labels gathered in the wild are often wrong, and those errors drag down neural network accuracy. Standard cleaning happens before training and cannot react to what the model itself is learning. Du's framework, Step-E, merges sample selection into training as a single process instead.

At each pass through the data, Step-E ranks examples by loss and gradually excludes more of the highest-loss ones after a short warm-up. The model thus learns first from easy, consistent examples and ends up ignoring stubborn outliers. On a noisy version of the CIFAR-100 image set, accuracy rose from about 43 percent to about 50 percent, clearly beating rival cleaning methods. Gains on CIFAR-10 were smaller but real, at modest extra training cost.

Catalog record

CREATORDu Wenzhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Step-E: A Differentiable Data Cleaning Framework for Robust Learning with Noisy Labels Du Wenzhang Abstract Training data collected in the wild often contain noisy labels and outliers that substantially degrade the performance and reliability of deep neural networks [ 18 , 3 ] . While data cleaning is commonly applied as a separate preprocessing stage [ 15 , 9 , 16 ] , such two-stage pipelines neither fully exploit feedback from the downstream model nor adapt to unknown noise patterns. We propose Step-E, a simple framework that integrates sample selection and model learning into a single differentiable optimization process. In each epoch, Step-E down-weights or discards samples with persistently high loss while retaining low-loss samples for gradient updates. The keep/drop ratio is gradually annealed after a short warm-up, yielding an online curriculum that focuses on easy and consistent examples and eventually ignores hard outliers. Step-E improves accuracy on CIFAR-100N [ 10 , 19 ] from 43.3% for standard training to 50.4% on average over three seeds, surpassing loss truncation, self-paced learning, and one-shot filtering baselines with better noise identification quality and only moderate training-time overhead. On CIFAR-10N (aggre) [ 19 ] , Step-E also improves over the noisy baseline (85.3% vs. 83.9%) and approaches the clean-label oracle (85.9%). These results demonstrate that integrating data cleaning and training via differentiable sample selection is an effective route toward robust learning on real noisy datasets. 1 Introduction Modern deep networks achieve impressive performance when trained on large, clean, and carefully curated datasets. In realistic data mining scenarios, however, labels come from heterogeneous sources such as crowdsourcing, weak supervision, or heuristic rules and are therefore noisy [ 18 , 3 ] . Human annotation errors, ambiguous images, and domain shifts all contribute to mislabeled or outlier samples that can harm generalization. In image classification, for example, web-scale datasets often contain wrong tags or near-duplicate images with conflicting labels; in user-generated content analysis, spam or off-topic posts corrupt the training distribution. Data cleaning is widely recognized as crucial [ 15 ] but is typically performed before model training using hand-crafted rules or separate anomaly detectors [ 9 , 16 ] . This two-stage design has two drawbacks: (i) it requires domain expertise or extra supervision to specify cleaning rules and thresholds; (ii) it decouples cleaning from model optimization, so the decisions do not directly leverage discriminative feedback from the task model. Some high-loss samples may still be informative “hard cases,” whereas others are truly corrupted and should be discarded. We explore a different paradigm: can the model learn which samples to trust during training, treating data cleaning as an integral, differentiable part of optimization? If such a framework works, it would reduce manual effort, adapt to unknown noise patterns, and align cleaning with the task objective. We propose Step-E (Stepwise Elimination), a lightweight framework that inserts a differentiable sample-selection module into standard SGD. At each epoch, we compute per-example losses using the current model, sort samples by loss, and drop a controlled fraction of high-loss ones. The drop ratio is gradually increased from zero after a short warm-up until it reaches a maximum \rho_{\max} estimated from the noisy dataset. Step-E performs online data cleaning: it keeps early training stable, allows the model to discover the main signal, and progressively suppresses persistent outliers. Contributions. We (i) introduce Step-E, a differentiable data cleaning framework that integrates sample selection with model training and requires no clean validation set or auxiliary model; (ii) provide a simple analysis showing that under separation assumptions on loss distributions, the optimal solution recovers the clean subset and approaches the clean-label oracle; (iii) conduct extensive experiments on CIFAR-100N and CIFAR-10N, where Step-E yields substantial gains in test accuracy, noise identification quality, and training efficiency. 2 Related Work 2.1 Robust learning under label noise Robust losses such as Huber loss, truncated loss, or loss clipping [ 7 , 4 , 20 ] dampen the influence of large errors but require manual thresholds and provide limited control over how many examples are discarded. Meta-learning approaches estimate per-sample weights using a small clean validation set [ 17 ] , but rely on extra trusted data. Some methods explicitly model label corruption using EM-style procedures [ 14 , 1 ] ; they assume a parametric noise model and can be sensitive to misspecification. Another important line of work uses small-loss selection in multi-network or teacher–student frameworks such as Co-teaching [ 5 ] , MentorNet [ 8 ] , and DivideMix [ 12 ] , which combine sample selection with label correction or semi-supervised training but typically require two networks or an auxiliary mentor. Step-E does not recover clean labels; it only decides whether a sample contributes to gradients, making it applicable to both label noise and feature outliers while retaining a single-network, deterministic training pipeline. 2.2 Data cleaning and anomaly detection Data cleaning methods range from rule-based constraint checking to unsupervised outlier detection [ 15 , 9 , 16 ] , but are typically divorced from the downstream learning objective. Their decisions may not optimally serve the target model. Step-E can be viewed as a lightweight, task-aware cleaning process learned jointly with the classifier rather than a separate filter. 2.3 Curriculum learning and self-paced learning Curriculum and self-paced learning adjust the order or subset of data according to difficulty, often measured by loss [ 2 , 11 ] . Classic self-paced learning gradually increases the keep ratio to eventually include all samples, which is inappropriate when a nontrivial fraction of data is corrupted. Step-E adopts the opposite philosophy: a fraction \rho_{\max} of samples are never learned and are permanently dropped once identified as persistent outliers. 3 Method 3.1 Problem formulation We consider supervised learning with potentially noisy labels. The training set \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} , y_{i}\in\mathcal{Y} , mixes clean samples from the target distribution with corrupted samples whose labels or features are unreliable. Let f_{\theta} denote the model with parameters \theta and \ell(f_{\theta}(x),y) a differentiable loss (e.g., cross-entropy). We introduce binary variables \gamma_{i}\in\{0,1\} indicating whether a sample is kept ( \gamma_{i}=1 ) or dropped ( \gamma_{i}=0 ). Assuming at most a fraction \rho_{\max} of samples are corrupted, we enforce \sum_{i}(1-\gamma_{i})\leq\rho_{\max}n . 3.2 Joint objective We optimize a regularized empirical risk J(\theta,\gamma)=\frac{1}{n}\sum_{i=1}^{n}\gamma_{i}\,\ell(f_{\theta}(x_{i}),y_{i})+\lambda\cdot\frac{1}{n}\sum_{i=1}^{n}\gamma_{i}, (1) subject to \gamma_{i}\in\{0,1\} and \sum_{i}\gamma_{i}\geq(1-\rho_{\max})n . The first term is the empirical loss over selected samples; the second encourages keeping more data to avoid over-pruning. When no noise is present, the optimum sets all \gamma_{i}=1 , recovering standard empirical risk minimization. Directly solving this mixed-integer program is intractable. Following self-paced learning, we alternate: for a given model \theta , we update \gamma based on current per-sample losses; for the updated \gamma , we update \theta using SGD on the selected subset. 3.3 Per-epoch sample selection For fixed \theta , optimizing J with respect to \gamma decomposes over samples. In practice, we bypass Lagrange multipliers and directly determine a loss threshold via the desired keep ratio. At each epoch t : 1. Probe pass. Run a forward pass to compute per-sample losses \ell_{i}(t) for all training examples. 2. Sorting and thresholding. Sort samples by loss and keep the lowest-loss fraction 1-\rho_{t} , where \rho_{t} is the drop ratio at epoch t . This yields a kept set K_{t} .

Read the complete text