Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds

Jiaxin Shi, Michalis K. Titsias · 2025

Read Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds by Jiaxin Shi, Michalis K. Titsias (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19664.txt.

Introduction

Jiaxin Shi and Michalis K Titsias give a new theoretical reading of the reweighted training losses used for diffusion models, the generators behind much current image AI. Practitioners have long reweighted the loss across noise levels because it works, without a full justification. The authors construct a cascade of time-dependent variational bounds on the data likelihood and prove the cascade improves on the standard bound, shrinking the gap between model and data distributions.

Combining those bounds produces exactly the reweighted objectives in wide use, and the framework covers both continuous Gaussian diffusion and masked, discrete diffusion. Showcasing it on masked diffusion for pixel-level image modelling, they report large gains over previous losses, approaching the sample quality of continuous models. A folk recipe gains a principled foundation.

Catalog record

CREATORJiaxin Shi, Michalis K. Titsias
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19664

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds Jiaxin Shi, Michalis K. Titsias Abstract We derive a new theoretical interpretation of the reweighted losses that are widely used for training diffusion models. Our method is based on constructing a cascade of time-dependent variational lower bounds on the data log-likelihood, that provably improves upon the standard evidence lower bound and results in reduced data-model KL-divergences. Combining such bounds gives rise to reweighted objectives that can be applied to any generative diffusion model including both continuous Gaussian diffusion and masked (discrete) diffusion models. Then, we showcase this framework in masked diffusion and report significant improvements over previous training losses in pixel-space image modeling, approaching sample quality comparable to continuous diffusion models. Our results also provide a theoretical justification for the simple weighting scheme widely used in masked image models. 1 Introduction Diffusion models ( Sohl-Dickstein et al. 2015 , Song and Ermon 2019 , Ho et al. 2020 , Song et al. 2020 ) have become the prominent generative modelling approach for image generation ( Rombach et al. 2022 , Ramesh et al. 2022 , Saharia et al. 2022 ) , audio ( Chen et al. 2021 , Kong et al. 2021 ) and video generation ( Ho et al. 2022 , Villegas et al. 2023 ) . These methods have been also extended to discrete data ( Sohl-Dickstein et al. 2015 , Austin et al. 2021 , Hoogeboom et al. 2021 , Campbell et al. 2022 , Lou et al. 2023 ) and applied to domains such as language modeling ( Nie et al. 2025 , Ye et al. 2025 ) , graph generation ( Vignac et al. 2023 ) , text-to-sound generation ( Yang et al. 2023 ) or protein design ( Wang et al. 2025 ) . Very recent work on masked diffusion ( Shi et al. 2024 ) suggests that the use of discrete diffusion enables a multimodal generative framework that unifies the treatment of diverse data, including images and text. The standard framework to train a diffusion model, either continuous or discrete, is to view it as as a probabilistic latent-variable model and apply approximate Maximum-Likelihood learning by optimizing the Evidence Lower Bounds (ELBOs) on the data log-likelihood. A detailed presentation of the ELBO and its continuous-time limit can be found in Kingma et al. 2021 , while corresponding continuous-time ELBOs for discrete masked diffusions were derived more recently ( Shi et al. 2024 , Sahoo et al. 2024 , Ou et al. 2024 ) . However, modern experimental practices in continuous Gaussian diffusion have converged to optimizing not the ELBO itself but a reweighted version of the ELBO. The reason for this switch is noticeably higher perceptual quality reported widely in literature on image generation ( Ho et al. 2020 , Nichol and Dhariwal 2021 ) . Although the reweighted loss is widely adopted in practice, its theoretical understanding has been quite limited. Kingma and Gao 2023 proposed to interpret the reweighted loss as a weighted sum of ELBOs of augmented data - where the data augmentation corresponds to Gaussian noises employed in continuous diffusion models. However, this derivation does not explain why ELBOs on the noise-perturbed data provide better signals for learning the denoiser. It is also unclear how to extend such result to discrete diffusion models. In this work, we give a new interpretation of the reweighted loss. Our initial motivation is to reduce the accumulation of errors when approximating (during training) the backward denoising process. To this end, we prove that the standard ELBO on the overall log-likelihood is not the best Maximum-Likelihood objective to train the model up to a given denoising time, but instead there is another improved time-dependent ELBO having smaller Kullback-Leibler divergence. Based on this observation, we suggest to train a full diffusion model not with the standard ELBO, but instead a weighted sum of all time-dependent ELBOs, visualized in Figure 1 . This analysis provides a theoretical interpretation of the reweighted loss that is not limited to the scope of continuous-time diffusion or Gaussian noise processes. Following this, we propose to adapt the reweighted schemes from continuous diffusion models to train masked diffusion models. As we demonstrate in our results, the new training objectives for masked diffusion yield significant improvements in image sample quality, measured by Fréchet inception distance (FID), over previous standard ELBO based losses. \mathbf{x} \mathbf{z}_{t(0)} \mathbf{z}_{t(1)} \dots \mathbf{z}_{t(T-1)} \mathbf{z}_{t(T)} q p_{\theta} p_{\theta} p_{\theta} p_{\theta} \mathbf{x} \mathbf{z}_{t(1)} \dots \mathbf{z}_{t(T-1)} \mathbf{z}_{t(T)} q p_{\theta} p_{\theta} p_{\theta} \vdots \vdots \mathbf{x} \mathbf{z}_{t(T-1)} \mathbf{z}_{t(T)} q p_{\theta} w_{1} w_{2} w_{T} Improved ELBO \mathcal{L}^{(1)}(\mathbf{x}) \mathcal{L}^{(2)}(\mathbf{x}) \mathcal{L}^{(T)}(\mathbf{x}) Tractable Sampling C.D.F Diffusion objectives: \mathcal{L}^{\tilde{w}}(\mathbf{x})={\displaystyle\lim_{T\to\infty}}\sum_{i=1}^{T}w_{i}\mathcal{L}^{(i)}(\mathbf{x})=\int_{0}^{1}{\color[rgb]{0,0,1}\tilde{w}(t)}\mathbb{E}_{q(\mathbf{z}_{t}|\mathbf{x})}\left[L_{\text{denoise}}(\mathbf{z}_{t},\mathbf{x},t)\right]dt+\mathrm{C} Figure 1: Diffusion objectives viewed as a weighted sum of the ELBOs of a sequence of models with optimal decoders (defined in Section 3 ). For continuous Gaussian diffusion models: L_{\text{denoise}}(\mathbf{z}_{t},\mathbf{x},t)=\frac{1}{2}\lambda^{\prime}(t)\|{\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t)}\|_{2}^{2} . For masked diffusion models: L_{\text{denoise}}(\mathbf{z}_{t},\mathbf{x},t)=-\frac{\alpha_{t}^{\prime}}{1-\alpha_{t}}\delta_{\mathbf{z}_{t},m}\cdot\mathbf{x}^{\top}\log\mu_{\theta}(\mathbf{z}_{t}) . 1.1 Related work The standard justification for using a reweighted loss over a likelihood-based loss (such as the ELBO) is that it prioritizes perceptually relevant signals over high frequency details ( Dieleman 2024 ) . A similar argument is often used to explain the suboptimal sample quality of autoregressive image models despite their strong likelihood performance. We offer an alternative perspective: the efficacy of reweighted loss is due to a more fundamental improvement in the ELBOs, specifically achieved by yielding smaller KL divergences. Parallel with the development of discrete diffusion models, masked image models such as MaskGIT also proposed using a weighted sum of cross-entropy denoising losses at many noise levels ( Chang et al. 2022 ) . Li et al. 2024 further extended this method to modeling image latent-space ( Rombach et al. 2022 , e.g., Stable Diffusion latents,) by switching from cross-entropy to euclidean losses resulted from local continuous diffusion. Despite the similarity between such models and masked diffusion models ( Zheng et al. 2024 ) , they often use a simple heuristic weighting scheme—denoising losses on mask inputs are summed over minibatches, divided by the total number of masks in the batch. You et al. 2025 summarizes the connection and differences of such models and masked diffusion models and proposed a hybrid model that uses simple weighting. In Section 5 we show that the simple weighting alone can be explained as a special case of our framework, and, when it is applied to masked diffusion models in isolation, it leads to a significant improvement in sample quality without requiring any other modifications as in masked image models ( Chang et al. 2022 , Li et al. 2024 , Li et al. 2025 ) and You et al. 2025 . 2 Background: Diffusion Models We consider the task of generative modeling: Given a dataset of observations \mathbf{x} with an underlying distribution q(\mathbf{x}) , we aim to train a probabilistic model p_{\theta}(\mathbf{x}) that approximates q(\mathbf{x}) . After training, we can draw novel samples from p_{\theta}(\mathbf{x}) that resemble observations in the dataset. The class of models we will look into is diffusion models ( Sohl-Dickstein et al. 2015 , Song and Ermon 2019 , Ho et al. 2020 , Song et al. 2020 ) . Typically, we construct such models by first introducing a “forward” noising process that gradually transforms \mathbf{x} to

Read the complete text