Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models

Luca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, Zeynep Akata · 2025

Read Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models by Luca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, Zeynep Akata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09968.txt.

Introduction

Luca Eyring and four co-authors tackle the high computing cost of improving image models at use time in a 2025 paper. Diffusion models make images by refining noise, and spending extra steps at use time can raise quality a lot. That extra work also makes each image slow to produce, which limits real use.

Their fix trains a small extra network that reshapes the starting noise, baking in what the slow search would have found. A learning rule keeps the model faithful to its base while steering it toward traits users want. Tests recover much of the quality gain from the slow method at a small share of its cost.

Catalog record

CREATORLuca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, Zeynep Akata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models Luca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, Zeynep Akata Abstract The new paradigm of test-time scaling has yielded remarkable breakthroughs in Large Language Models (LLMs) (e.g. reasoning models) and in generative vision models, allowing models to allocate additional computation during inference to effectively tackle increasingly complex problems. Despite the improvements of this approach, an important limitation emerges: the substantial increase in computation time makes the process slow and impractical for many applications. Given the success of this paradigm and its growing usage, we seek to preserve its benefits while eschewing the inference overhead. In this work we propose one solution to the critical problem of integrating test-time scaling knowledge into a model during post-training. Specifically, we replace reward guided test-time noise optimization in diffusion models with a Noise Hypernetwork that modulates initial input noise. We propose a theoretically grounded framework for learning this reward-tilted distribution for distilled generators, through a tractable noise-space objective that maintains fidelity to the base model while optimizing for desired characteristics. We show that our approach recovers a substantial portion of the quality gains from explicit test-time optimization at a fraction of the computational cost. Code is available at https://github.com/ExplainableML/HyperNoise . 1 Introduction Recently, inference-time scaling has made remarkable breakthroughs in Large Language Models [ 78 , 24 , 36 ] and generative vision models, enabling models to spend more computation during inference to solve complex problems effectively. Drawing from the success and growing usage of test-time compute in LLMs, several methods have attempted to apply similar ideas in the context of diffusion models for generation [ 55 , 18 , 6 , 91 , 86 , 87 , 62 , 84 , 82 , 61 , 75 ] . The goal of this process is to spend additional compute during inference to obtain generations that better reflect desired output properties. Diffusion model test-time techniques that optimize the initial noise or intermediate steps of the diffusion process, often guided by feedback from pre-trained reward models [ 97 , 44 , 96 , 95 , 101 , 50 ] , have demonstrated significant promise in improving critical attributes of the generated outputs, such as prompt following, aesthetics, quality and composition [ 55 , 40 , 18 , 9 , 91 , 62 , 61 ] . These methods generally fall into two broad categories: gradient-based optimization, which typically requires substantial GPU memory for backpropagation through the full model [ 91 , 6 , 18 , 62 , 42 ] , and gradient-free optimization, which often necessitates a very large number of function evaluations (NFEs), sometimes thousands, of the computationally expensive denoising network [ 40 , 55 , 87 ] . While both strategies can effectively boost output quality, they introduce considerable latency (exceeding 10 minutes for one generation), severely limiting their practical utility, particularly for real-time applications. This is an instantiation of a global problem of test-time scaling methods that we seek to tackle in this work. The core hypothesis of our work is whether it is possible to capture a portion of test-time scaling benefits by integrating this knowledge into a neural network during training time? Figure 1 : The same initial random noise is used for the base generation and the initialization of noise hypernetwork. HyperNoise significantly improves upon the initially generated image with respect to both prompt faithfulness and aesthetic quality for both SANA-Sprint and FLUX-Schnell. To address this, one might consider directly fine-tuning the diffusion model using reward signals [ 12 , 69 , 15 , 85 , 83 , 49 , 102 , 10 ] or with Direct Preference Optimization (DPO) [ 72 , 92 , 48 , 41 , 30 ] . The objective here can be formulated as learning a tilted distribution (Equation 4 ), which upweights samples with high reward while maintaining fidelity to a base model’s distribution. These methods are usually expensive to train due to the need for backpropagation through the sampling process. Instead, one might consider directly fine-tuning a step-distilled generative model to learn this target distribution. However, this approach typically involves a KL regularization to the base model that is intractable for distilled models. An imbalance or poor estimation of this can lead to the model "reward-hacking", superficially maximizing the reward metric while significantly deviating from the desired underlying data distribution, thus not achieving the genuine desired improvements. In this work, we propose a different path to realize the benefits of the target tilted distribution (Equation 3 ), particularly for step-distilled generative models. Our core hypothesis is that instead of modifying the parameters of the base generator, we can achieve the desired output distribution by learning to predict an optimal initial noise distribution. We first show that such an optimal tilted noise distribution p_{0}^{\star} exists (characterized by Equation 5 ). When samples from this p_{0}^{\star} are passed through the frozen generator, they naturally produce outputs that are distributed according to the target data-space tilted distribution. To learn this tilted noise distribution, we introduce a lightweight network, f_{\phi} , that transforms standard Gaussian noise into a modulated, improved noise latent. The crucial advantage of this approach lies in its optimization objective. In particular, the regularization term, a KL divergence between the modulated noise distribution and the standard Gaussian prior, is defined entirely in the noise space. We show that this noise-space KL divergence can be made tractable and effectively approximated by an L_{2} penalty on the magnitude of the noise modification. This lightweight network forms the core of our approach, which we term Noise Hyper networks. It functions akin to a hypernetwork [ 26 , 2 , 27 , 59 , 35 , 67 , 89 , 99 ] as rather than generating the final image, it produces a specific, optimized starting latent for the main frozen generative model. This effectively guides the output of the base model without any changes to its parameters. Broadly, a hypernetwork is an auxiliary model trained to generate crucial inputs or parameters of a primary model. Our f_{\phi} embodies this concept by learning to predict the optimized initial noise as input to the frozen generator. Consequently, our proposed approach is effectively training a Noise Hypernetwork to perform the task of test-time noise optimization , by learning to directly output an optimized noise latent in a single step sidestepping the need for expensive, iterative test-time optimization. Our practical implementation of the noise hypernetwork utilizes Low-Rank Adaptation (LoRA), ensuring it remains parameter-efficient and adds negligible computational cost during inference. We apply our method to text-to-image generation, conducting evaluations with an illustrative "redness" reward task to demonstrate core mechanics, as well as complex alignments using sophisticated human-preference reward models. We demonstrate the efficacy of our approach by applying it to distilled diffusion models SD-Turbo [ 77 ] , SANA-Sprint [ 11 ] , and FLUX-Schnell. Overall, our experiments show that we can recover a substantial portion of the quality gains from explicit test-time optimization at a fraction of the computational inference cost. In summary, our contributions are: 1. We introduce HyperNoise, a novel framework that learns to predict an optimized initial noise for a fixed distilled generator, effectively moving test-time noise optimization benefits and

Read the complete text