Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps

Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, Saining Xie · 2025

Read Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps by Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, Saining Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09732.txt.

Introduction

Nanye Ma and co-authors find new ways to spend compute at generation time in diffusion models. Their 2025 paper notes that image diffusion models usually scale only by adding denoising steps, and gains flatten after a few dozen. Language models, by contrast, keep improving when given more time to think.

The authors reframe the task as a search over starting noise. Two design axes organize the space: the verifiers that grade candidate results, and the search methods that hunt for better noise. Trials on class-based and text-based image benchmarks show real gains from the extra compute, beyond what more steps alone deliver. Their message is that sampling strategy, not just model size, is a lever for quality that the field has barely pulled.

Catalog record

CREATORNanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, Saining Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, Saining Xie Abstract Generative models have made significant impacts across various domains, largely due to their ability to scale during training by increasing data, computational resources, and model size, a phenomenon characterized by the scaling laws. Recent research has begun to explore inference-time scaling behavior in Large Language Models (LLMs), revealing how performance can further improve with additional computation during inference. Unlike LLMs, diffusion models inherently possess the flexibility to adjust inference-time computation via the number of denoising steps, although the performance gains typically flatten after a few dozen. In this work, we explore the inference-time scaling behavior of diffusion models beyond increasing denoising steps and investigate how the generation performance can further improve with increased computation. Specifically, we consider a search problem aimed at identifying better noises for the diffusion sampling process. We structure the design space along two axes: the verifiers used to provide feedback, and the algorithms used to find better noise candidates. Through extensive experiments on class-conditioned and text-conditioned image generation benchmarks, our findings reveal that increasing inference-time compute leads to substantial improvements in the quality of samples generated by diffusion models, and with the complicated nature of images, combinations of the components in the framework can be specifically chosen to conform with different application scenario. 1 Introduction Figure 1: Inference scaling beyond increasing denoising steps. We demonstrate the performance with respect to FID \boldsymbol{\downarrow} , IS \boldsymbol{\uparrow} on ImageNet, and CLIPScore \boldsymbol{\uparrow} , Aesthetic Score \boldsymbol{\uparrow} on DrawBench. Our search framework exhibits substantial improvements in all settings over purely scaling NFEs with increasing denoising steps. Generative models have transformed various fields, including language [ 1 , 78 , 80 ] , vision [ 61 , 60 ] , and biology [ 86 ] , by learning to sample from the underlying data distributions. A key factor in their success is their ability to scale up during training by increasing data volumes, computational resources, and model sizes. This training-time scaling behavior, often described as Scaling Laws [ 31 , 27 ] , predicts how performance improves as the models grow larger, consume more data, and are trained for longer time, guiding the development of increasingly capable generative models. Recently, in Large Language Models (LLMs), the study on scaling has expanded to inference-time [ 69 , 7 , 90 ] . By allocating more compute during inference, often through sophisticated search processes, these works show that LLMs can produce higher-quality and more contextually appropriate responses [ 87 , 85 , 95 , 20 , 74 ] . Inference-time scaling opens new avenues for improving model performance when additional resources become available after training. Diffusion models [ 70 , 26 , 72 ] , trained to remove noises from data, are a class of generative models that dominates the continuous data domains such as images [ 15 ] , audio [ 67 ] , and videos [ 56 ] . To generate a single sample, their generation process usually starts from pure noise and requires multiple forward passes of trained models to denoise and obtain clean data. These forward passes are thus dubbed denoising steps . Since the number of denoising steps can be adjusted to trade sample quality for computational cost, the generation process of diffusion models naturally provides flexibility in allocating inference-time computation budget. Under the context of generative models, such computation budget is also commonly measured by the number of function evaluations (NFE), to ensure a reasonable comparison with other families of models that use iterative sampling processes but without denoising capabilities [ 79 , 96 ] , Empirical observations [ 72 , 32 , 71 ] have indicated that by investing compute into denoising steps alone, performance gains tend to plateau after a certain NFEs, limiting the benefits of scaling up computation during inference. Therefore, previous work on diffusion models has long focused on maintaining high performance while making NFEs as small as possible for efficiency during inference time [ 64 , 73 ] . We, on the other hand, are interested in the opposite frontier. Compared to LLMs, diffusion models work with explicit randomness that comes from the noises injected either as initial samples or during the sampling process [ 72 , 93 ] . It has been shown that these noises are not created equal [ 2 , 57 ] , i.e., some lead to better generations than others. This observation extends an additional dimension to scale NFEs other than increasing denoising steps - searching for better noises in sampling. Rather than solely allocating NFEs for denoising steps, which often leads to a quick performance plateau, this work investigates methods to effectively utilize compute during inference through search, thereby improving the performance and scalability of diffusion models at inference time (Figure 1 ). We primarily consider two design axes in our search framework: verifiers used to provide feedback in search, and algorithms used to find better noise candidates, following terminologies used in LLMs [ 69 ] . For verifiers , we consider the three different settings, which are meant to simulate three different use cases: (1) where we have privileged information about how the final evaluation is carried out; (2) where we have conditioning information for guiding the generation; (3) where we have no extra information available. For algorithms , we consider (1) Random Search, which simply selects the best from a fixed set of candidates; (2) Zero-Order Search, which leverages verifiers feedback to iteratively refine noise candidates; (3) Search over Paths, which leverages verifiers feedback to iteratively refine diffusion sampling trajectories. We first walk over these design choices in the relatively simple setting of class-conditioned generation on ImageNet and demonstrate their effectiveness, providing an instantiation of our framework. Then we carry these design choices over to the larger-scale text-conditioned generation setting and evaluate our proposed framework. Due to the complex nature of images and the rich information text conditionings contain, more holistic evaluations of generation quality are required [ 42 ] . We therefore employ multiple verifiers for scaling inference-time compute in search. This also enables us to probe into the “biases” each verifier possesses, and how well they are aligned with the generation tasks. To alleviate overfitting to a single verifier, we also experiment with an ensemble of verifiers and showcase its good generalizability across different benchmarks. Our contributions are summarized as follows: • We propose a fundamental framework for inference-time scaling of diffusion models. We show that scaling NFEs through search can lead to substantial improvement across generation tasks and model sizes beyond increasing denoising steps. In addition, we conduct a comprehensive empirical analysis of how inference-time compute budgets affect scaling performance. • We identify two key design axes in the proposed search framework: verifiers, which provide feedback, and algorithms, which find better noise candidates. We examine how different verifier–algorithm combinations perform across various tasks, and our findings indicate that no single configuration is universally optimal; each task instead necessitates a distinct search setup to achieve the best scaling performance. • We conduct extensive analysis on the alignment between verifiers

Read the complete text