NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training
Fang Wu, Haokai Zhao, Da Xing, Tinson XU, Hanqun Cao, Yanchao Li, Zeqi Zhou, Xiangru Tang, Hanchen Wang, Hongbin Lin, Zehong Wang, Kuan Pang, Xia Peng, Yinxi Li, Aaron Tu, Molei Tao, Li Erran Li, Aditya Joshi, Jure Leskovec, Yejin Choi · 2026
Read NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training by Fang Wu, Haokai Zhao, Da Xing, Tinson XU, Hanqun Cao, Yanchao Li, Zeqi Zhou, Xiangru Tang, Hanchen Wang, Hongbin Lin, Zehong Wang, Kuan Pang, Xia Peng, Yinxi Li, Aaron Tu, Molei Tao, Li Erran Li, Aditya Joshi, Jure Leskovec, Yejin Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08144.txt.
Introduction
Haokai Zhao and colleagues ask a neglected question about diffusion model training: does the exact random noise drawn for a sample matter? Training randomness has two parts, the timestep and the noise itself. Timesteps are heavily studied through schedules and weights, while noise draws are usually treated as interchangeable.
Their NoiseRater network scores each noise instance for a given sample and timestep. It is learned by bilevel optimization: its scores reweight the training loss, and it is updated to lower validation loss. Top-scored noise improved training over random draws, while bottom-scored noise hurt, showing that noise quality varies and can be selected for gain.
Catalog record
| CREATOR | Fang Wu, Haokai Zhao, Da Xing, Tinson XU, Hanqun Cao, Yanchao Li, Zeqi Zhou, Xiangru Tang, Hanchen Wang, Hongbin Lin, Zehong Wang, Kuan Pang, Xia Peng, Yinxi Li, Aaron Tu, Molei Tao, Li Erran Li, Aditya Joshi, Jure Leskovec, Yejin Choi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08144
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training Fang Wu, Haokai Zhao, Da Xing, Tinson XU, Hanqun Cao, Yanchao Li, Zeqi Zhou, Xiangru Tang, Hanchen Wang, Hongbin Lin, Zehong Wang, Kuan Pang, Xia Peng, Yinxi Li, Aaron Tu, Molei Tao, Li Erran Li, Aditya Joshi, Jure Leskovec, Yejin Choi Abstract Diffusion models have achieved remarkable success across a wide range of generative tasks, yet their training paradigm largely treats injected noise as uniformly informative. In this work, we challenge this assumption and introduce NoiseRater, a meta-learning framework for instance-level noise valuation in diffusion model training. We propose a parametric noise rater that assigns importance scores to individual noise realizations conditioned on data and timestep, enabling adaptive reweighting of the training objective. The rater is trained via bilevel optimization to improve downstream validation performance after inner-loop diffusion updates. To enable efficient deployment, we further design a decoupled two-stage pipeline that transitions from soft weighting during meta-training to hard noise selection during standard training. Extensive experiments on FFHQ and ImageNet demonstrate that not all noise samples contribute equally, and that prioritizing informative noise improves both training efficiency and generation quality. Our results establish noise valuation as a complementary and previously underexplored axis for improving diffusion model training. Our code is available at: https://anonymous.4open.science/r/NoiseRater-DEB116 . 1 Introduction Diffusion models [ 42 , 44 , 17 ] have emerged as a dominant paradigm for generative modeling, achieving state-of-the-art performance across image [ 34 , 39 ] , video [ 32 ] , biology [ 51 ] , and multimodal generation tasks [ 38 , 52 ] . A key factor behind their success is the iterative denoising process, which transforms random noise into structured data through a sequence of refinement steps. Recently, there has been a growing interest in test-time compute optimization for diffusion models [ 15 , 37 , 24 , 53 , 29 , 45 ] . In particular, noise has increasingly become a central object of control at inference time [ 11 , 26 ] . Techniques such as guidance scaling [ 2 ] , adaptive step-size schedules [ 8 ] , and noise resampling strategies [ 1 , 41 ] explicitly manipulate the noise trajectory to trade off between generation quality, diversity, and computational cost. These methods demonstrate that the choice of noise—its magnitude, structure, and evolution—plays a critical role in shaping the final output. Existing approaches primarily treat noise as a test-time control variable . During training, however, noise is typically sampled from a fixed Gaussian distribution and incorporated into the objective in a largely uniform, sample-agnostic manner. While prior work has shown that different noise levels can contribute unequally to learning [ 49 , 33 , 35 , 46 , 13 ] , major methods largely focus on timestep-level reweighting or schedule design, leaving the role of instance-level noise variation underexplored. This raises a fundamental question: are all noise realizations equally useful for learning diffusion models? More specifically, even at the same timestep, different noise instances may carry varying levels of learning signal. Some may provide clear supervision for denoising, while others may be ambiguous, redundant, or less informative for optimization. If this is the case, then uniformly treating noise during training may lead to suboptimal learning dynamics. In this work, we shift the focus from test-time noise manipulation to training-time noise valuation . We propose to explicitly model the importance of individual noise samples and use this information to guide the training process. Concretely, we introduce a meta-learned noise rater , a parametric function that assigns a score to each noise instance conditioned on the data sample and timestep. These scores are used to reweight the diffusion loss, allowing the model to focus on more informative noise while downweighting less useful ones. To learn the noise rater, we formulate the problem as a bilevel optimization framework [ 30 , 9 ] . The diffusion model is trained using weighted noise samples in the inner loop, while the rater is optimized in the outer loop to minimize validation loss. This meta-learning setup enables the rater to capture the contribution of noise samples to generalization performance directly, rather than relying on heuristic criteria. Our approach is simple, flexible, and compatible with modern diffusion frameworks [ 43 ] . By operating purely at the level of the training objective, it does not require modifying the model architecture or inference procedure. The method overview is in Fig. 1 . Figure 1 : Illustration of our meta-Learned noise valuation approach for diffusion training. We first train a noise rater using bi-level optimization, and then guide diffusion training with the rater’s scores. We summarize our contributions as follows: • We identify and formalize the problem of training-time noise valuation in diffusion, an underexplored dimension complementary to timestep-level noise design and test-time control. • We propose a meta-learned noise rater that adaptively weights noise samples during training via a bilevel optimization framework. • We introduce a decoupled two-stage training pipeline that separates noise evaluation from model training, enabling efficient deployment of meta-learned noise policies. • We demonstrate that not all noise realizations are equally useful, and that selectively emphasizing informative noise improves learning efficiency and model performance. 2 Related Work Noise Optimization at Inference Time. Recent work demonstrates that optimizing initial noise at inference time can significantly improve generation quality. One line of research directly optimizes the noise latent through backpropagation across denoising steps. These methods iteratively update the noise to maximize human-preference rewards [ 48 ] , satisfy specific motion constraints [ 22 ] , or align with internal attention scores [ 12 ] . While effective and training-free, they incur massive computational overhead at inference due to repeated denoising loops. To amortize this cost, other approaches train auxiliary models to directly predict optimal noise. For instance, FIND [ 5 ] uses reinforcement learning to adjust the mean and variance of the initial Gaussian, while GoldenNoise [ 54 ] trains a network to predict high-quality noise perturbations. Crucially, all treat noise as a test-time control variable. In contrast, we explore the untapped potential of evaluating and weighting noise during training. Training-Time Loss Reweighting and Noise Selection. Various loss reweighting and noise scheduling strategies improve diffusion training efficiency. Most methods operate at the timestep level , adjusting the relative importance or sampling frequency of different noise magnitudes. For loss reweighting, P2 [ 6 ] prioritizes highly corrupted timesteps to learn global concepts, Min-SNR [ 14 ] balances inter-timestep optimization conflicts, and 46 dynamically adjusts weights based on the variance of loss distributions across log-SNR levels. Concurrently, noise scheduling approaches optimize the distribution of sampled timesteps: 13 introduces a fixed schedule focusing on the \log\text{SNR}\approx 0 region. 25 uses curriculum learning to progressively introduce harder timesteps, and 35 proposes a dynamic, information-guided allocation based on entropy reduction. 49 theoretically analyzes different total weighting, time, and noise schedules. In addition, EDM [ 20 ] provides a unifying perspective on diffusion design, showing that choices of noise parameterization, preconditioning, and loss scaling implicitly define effective weighting across noise levels. However, these methods treat all noise instances at a given timestep as equally informative. Recent work has also begun to question more fundamental assumptions about noise in diffusion