Gradient-Free Noise Optimization for Reward Alignment in Generative Models

Jeongsol Kim, Hongeun Kim, Jian Wang, Jong Chul Ye, KAIST AI, Snap Inc. {jeongsol, hongeun}@kaist.ac.kr, [email protected], [email protected] * Equal contribution · 2026

Read Gradient-Free Noise Optimization for Reward Alignment in Generative Models by Jeongsol Kim, Hongeun Kim, Jian Wang, Jong Chul Ye, KAIST AI, Snap Inc. {jeongsol, hongeun}@kaist.ac.kr, [email protected], [email protected] * Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11347.txt.

Introduction

Kim and three colleagues present ZeNO, a gradient-free way to tune noise for reward alignment. Existing methods need multi-step stochastic paths or backpropagation through generator and reward. That fails for deterministic generators. ZeNO frames noise tuning as path-integral control, estimated from reward values alone. With an Ornstein-Uhlenbeck reference and a one-step horizon, its updates link to Langevin dynamics, and it scales inference well across generators, including protein structure generation.

Catalog record

CREATORJeongsol Kim, Hongeun Kim, Jian Wang, Jong Chul Ye, KAIST AI, Snap Inc. {jeongsol, hongeun}@kaist.ac.kr, [email protected], [email protected] * Equal contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52594
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11347

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient-Free Noise Optimization for Reward Alignment in Generative Models Jeongsol Kim, Hongeun Kim, Jian Wang, Jong Chul Ye, KAIST AI, Snap Inc. {jeongsol, hongeun}@kaist.ac.kr, [email protected], [email protected] * Equal contribution Abstract Existing reward alignment methods for diffusion and flow models rely on multi-step stochastic trajectories, making them difficult to extend to deterministic generators. A natural alternative is noise-space optimization, but existing approaches require backpropagation through the generator and reward pipeline, limiting applicability to differentiable settings. To address this, here we present ZeNO ( Ze roth-order No ise O ptimization), a gradient-free framework that formulates noise optimization as a path-integral control problem, estimable from zeroth-order reward evaluations alone. When instantiated with an Ornstein–Uhlenbeck reference process, the update connects to Langevin dynamics implicitly targeting a reward-tilted distribution. ZeNO enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible. Figure 1: Representative results of ZeNO across diverse one-step generators and reward functions. Our noise optimization consistently improves image quality and prompt alignment without generator modification or backpropagation. Generator specification and the corresponding text prompts are reported in Appendix F . 1 Introduction Post-training aims to align a pre-trained generative model with desired objectives, often specified by a reward function such as prompt fidelity, aesthetic quality, human preference, etc. A broad class of recent post-training methods improves diffusion or flow models by optimizing them toward such rewards while keeping them close to a reference model. For example, DDPO 1 , Flow-GRPO 18 , PCPO 16 , and SQDF 10 formulate alignment from a sequential Markov decision process perspective, whereas VGG-Flow 19 and Residual \nabla -DB 20 are motivated by optimal control and GFlowNet-style principles. Despite their different motivations, these methods are primarily developed for settings in which generation is represented as a multi-step stochastic trajectory. This trajectory-level structure is central to their formulation, and consequently, extending these approaches to deterministic or one-step generators is not straightforward, since such generators do not naturally expose the same stochastic trajectory structure. Recent advances in generative modeling have led to the development of few-step or one-step deterministic generators, such as consistency models 26 ; 21 and distilled diffusion models 34 ; 33 ; 24 ; 7 , which achieve high-quality generation at a fraction of the computational cost of their multi-step counterparts 34 . Aligning these models with reward functions is therefore of growing practical importance, yet existing post-training methods are not directly applicable to this. A natural alternative is to optimize the input noise of a deterministic generator directly 5 ; 22 . Given a generator G_{\phi} and a reward function r , one could backpropagate the terminal reward through the generator and perform gradient ascent in the noise space. However, this approach requires differentiating through the generator and reward pipeline, and can be unreliable when gradients reflect spurious local sensitivities rather than meaningful semantic improvement. Moreover, it is inapplicable when the reward is non-differentiable or when the generator is too large to support efficient backpropagation. For example, ReNO 5 and ORIGEN 22 operate in the noise space and connects noise optimization to gradient ascent or Langevin dynamics targeting a reward-tilted distribution, but still require backpropagation through the reward-generator composition, limiting its applicability to differentiable pipelines. In this paper, we retain the noise-space perspective but replace direct gradient ascent with a zeroth-order stochastic control formulation, which we call ZeNO ( Ze roth-order No ise O ptimization). Specifically, we treat noise evolution as a controllable process and optimize it via a path-integral control (PIC) formulation, using the final sample reward as the objective, which yields a practical zeroth-order estimator. In particular, we adopt the Ornstein-Uhlenbeck (OU) process as the reference dynamics, which provides initial noises compatible with the pre-trained generator. The PIC formulation situates this estimator within a controlled SDE framework, connecting the noise update to Langevin sampling targeting the reward-tilted distribution p^{\star}({\bm{z}})\propto p({\bm{z}})\exp(r(G_{\theta}({\bm{z}}))/\lambda) without requiring gradients through the generator or reward model. We further introduce a variance reduction approximation that achieves significantly higher reward improvement under the same compute budget. Empirically, ZeNO achieves competitive or superior performance compared to gradient-based methods across a range of generative models and reward functions, including non-differentiable reward, and enables effective inference-time scaling as more computation is allocated. We further validate the broad applicability of ZeNO on a protein structure generation task, where the reward model is too large to support efficient backpropagation, which is a setting where gradient-based methods are inapplicable, demonstrating strong performance even under this challenging constraint. 2 Background Stochastic Differential Equation. For a stochastic process {\bm{x}}_{t} , a stochastic differential equation (SDE) is written as d{\bm{x}}_{t}=f({\bm{x}}_{t},t)dt+g(t)d{\bm{w}}_{t} (1) where f({\bm{x}}_{t},t) and g(t) denote the drift and diffusion coefficients, and {\bm{w}}_{t} denotes a Wiener process. A representative example is the Ornstein–Uhlenbeck (OU) process, obtained when f({\bm{x}}_{t},t)=-a{\bm{x}}_{t} and g(t)=b for a>0 and b>0 . This process has the Gaussian stationary distribution {\mathcal{N}}(0,b^{2}/2a) 36 . In particular, when 2a=b^{2} , the stationary distribution becomes {\mathcal{N}}(0,{\mathbf{I}}) . Reward Tilted Distribution. In post-training generative models to maximize rewards, the alignment problem is often formulated as an entropy-regularized reinforcement learning objective over the generative trajectory. Let P^{\mathrm{ref}} denote the trajectory distribution induced by the pretrained generative model, and let Q^{\theta} denote the trajectory distribution induced by the finetuned model parameterized by \theta . The objective can then be written as \mathcal{L}(\theta)=\mathbb{E}_{{\bm{x}}_{0}\sim q_{0}^{\theta}}\left[r({\bm{x}}_{0})\right]-\lambda D_{\mathrm{KL}}\left(Q^{\theta}\,\|\,P^{\mathrm{ref}}\right), (2) where the first term encourages the model to generate high-reward samples, while the KL regularization term penalizes excessive deviation from the pretrained model. The coefficient \lambda>0 controls the trade-off between reward maximization and staying close to the reference generative process. Equivalently, this KL term can often be decomposed into a sum or integral of per-step policy divergences along the sampling trajectory, depending on the specific generative process. The marginal distribution induced by the optimal policy of Eq. ( 2 ) is given by 29 p_{t}^{\star}({\bm{x}}_{t})\propto p_{t}^{\mathrm{ref}}({\bm{x}}_{t})e^{V({\bm{x}}_{t})},\quad V({\bm{x}}_{t}):=\log\mathbb{E}_{P^{\mathrm{ref}}}\left[\exp\left(\frac{r({\bm{x}}_{0})}{\lambda}\right)\middle|{\bm{x}}_{t}\right], (3) for t\in[0,1] , where {\bm{x}}_{0} refers to the clean sample. This shows that the optimal marginal is obtained by tilting the reference marginal according to the conditional desirability of future high-reward samples. Notably, this tilted form also applies at t=1 , corresponding to the initial noise distribution, which is particularly relevant to settings where optimization is performed directly in the initial noise space. Path Integral Control. Let {\bm{u}}({\bm{x}}_{t},t) denote a control input that guides the reference SDE d{\bm{x}}_{t}=f({\bm{x}}_{t},t)dt+g(t)\left({\bm{u}}({\bm{x}}_{t},t)dt+d{\bm{w}}_{t}\right). (4) We seek a control that steers the trajectory toward high-reward terminal states while regularizing control effort. To this end, we consider the terminal-cost optimal control objective J_{\bm{u}}({\bm{x}}_{t},t)=m({\bm{x}}_{t})+\int_{t}^{T}\frac{1}{2}\|{\bm{u}}({\bm{x}}_{\tau},\tau)\|^{2}d\tau, (5) where m({\bm{x}}_{t}) denotes the terminal cost and the running cost term penalizes the control energy. Since the objective contains no intermediate-state cost, the optimal policy is driven solely by the terminal desirability of the trajectory subject to minimum control effort. Under this formulation, the optimal control has the path-integral form 11 ; 9 ; 28 . \displaystyle{\bm{u}}^{\star}({\bm{x}}_{t},t)=\frac{\mathbb{E}_{{\bm{x}}_{t:T}}\left[\exp\left(-m({\bm{x}}_{t})/\lambda\right)d{\bm{w}}_{t}\right]}{\mathbb{E}_{{\bm{x}}_{t:T}}\left[\exp\left(-m({\bm{x}}_{T})/\lambda\right)\right]}, (6) where the expectation is taken over trajectories of the uncontrolled reference SDE in Eq. ( 4 ). This expression shows that the control can be estimated from reward-weighted trajectory samples and depends only on terminal-cost evaluations, without requiring differentiation through the reward or the generator. A practical difficulty is that evaluating {\bm{u}}^{\star} over the full

Read the complete text