Training-Free Reward-Guided Image Editing via Trajectory Optimal Control
Jinho Chang, Jaemin Kim & Jong Chul YeGraduate School of Artificial IntelligenceKorea Advanced Institute of Science and TechnologySeoul, South Korea {jinhojsk515,kjm981995,jong.ye}@kaist.ac.kr · 2025
Read Training-Free Reward-Guided Image Editing via Trajectory Optimal Control by Jinho Chang, Jaemin Kim & Jong Chul YeGraduate School of Artificial IntelligenceKorea Advanced Institute of Science and TechnologySeoul, South Korea {jinhojsk515,kjm981995,jong.ye}@kaist.ac.kr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25845.txt.
Introduction
Jinho Chang, Jaemin Kim, and Jong Chul Ye edit images with reward guidance and no retraining, in a 2025 paper. Image generators can be steered toward goals during creation, but editing is harder: the result must keep the source image's meaning while raising a target score. Earlier editing methods lean on reversing the generator, which can drift.
The authors treat editing as steering a moving path, borrowing optimal control theory. The generator's reverse process becomes the path, and helper variables updated step by step nudge it toward the reward. Across several editing tasks the method beat training-free rivals, gaining reward without the tricks that game scores while wrecking the image. It kept the source's content more faithfully at the same time.
Catalog record
| CREATOR | Jinho Chang, Jaemin Kim & Jong Chul YeGraduate School of Artificial IntelligenceKorea Advanced Institute of Science and TechnologySeoul, South Korea {jinhojsk515,kjm981995,jong.ye}@kaist.ac.kr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25845
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training-Free Reward-Guided Image Editing via Trajectory Optimal Control Jinho Chang, Jaemin Kim & Jong Chul YeGraduate School of Artificial IntelligenceKorea Advanced Institute of Science and TechnologySeoul, South Korea {jinhojsk515,kjm981995,jong.ye}@kaist.ac.kr Abstract Recent advancements in diffusion and flow-matching models have demonstrated remarkable capabilities in high-fidelity image synthesis. A prominent line of research involves reward-guided guidance, which steers the generation process during inference to align with specific objectives. However, leveraging this reward-guided approach to the task of image editing, which requires preserving the semantic content of the source image while enhancing a target reward, is largely unexplored. In this work, we introduce a novel framework for training-free, reward-guided image editing. We formulate the editing process as a trajectory optimal control problem where the reverse process of a diffusion model is treated as a controllable trajectory originating from the source image, and the adjoint states are iteratively updated to steer the editing process. Through extensive experiments across distinct editing tasks, we demonstrate that our approach significantly outperforms existing inversion-based training-free guidance baselines, achieving a superior balance between reward maximization and fidelity to the source image without reward hacking. Figure 1: Reward-guided image editing samples with unconditional diffusion and flow-matching models. Reward-guided edited samples across various tasks, such as (a) Human preference, (b) Style transfer, (c) Counterfactual generation, and (d) Text-guided image editing. 1 Introduction Following the advancement of diffusion and flow-matching models that led to remarkable success in high-fidelity image synthesis ( Ho et al. 2020 ; Dhariwal & Nichol 2021 ; Lipman et al. 2022 ) , various methods have been developed to edit real-world images ( Meng et al. 2021 ; Hertz et al. 2023 ) by leveraging their pre-trained image priors. However, most editing techniques remain limited to concepts that exist within the model’s pre-trained distribution ( i.e. , applying a “Van Gogh style” is only possible if the model has been trained in such styles). While text-to-image models ( Rombach et al. 2022 ; Esser et al. 2024 ) provide diverse conditional distributions, abstract human preferences or subtle stylistic nuances are often difficult to specify clearly using natural language. Meanwhile, reward-guided sampling methods have been proposed as a promising, training-free framework that operates during inference ( Chung et al. 2023 ; Yu et al. 2023 ; Song et al. 2023 ; Ye et al. 2024 ; Geng & Owens 2024 ) , which leverages off-the-shelf differentiable reward functions to steer the generation process toward a desired objective. The primary advantage of this approach is its ability to generate images toward a novel target distribution defined by the reward function, moving beyond the original sample distribution. Despite the progress of reward-guided image generation, its potential for image editing techniques has been under-explored, and there is room for improvement. Reward-guided editing is more challenging, as it requires both maximizing a reward and preserving the core identity of the source image. The most intuitive approach is to first invert the source image into the noise space and then apply a reward-guided generation algorithm during the reverse process. Unfortunately, this method often fails because most guidance techniques rely on the reward gradient to the intermediate noised image or one-step approximation of the clean image, but for complex and non-linear reward functions, this indirect guidance degrades the structural faithfulness of the source image ( Chung et al. 2023 ; Yu et al. 2023 ; Ye et al. 2024 ) . Figure 2: Methodology overview. Given a source image {\bm{x}}_{1} , our method first generates its corresponding initial trajectory. We then progressively refine this trajectory by solving a reward-guided optimal control problem. This process steers the path into an optimized trajectory, whose endpoint is the final edited image {\bm{x}}_{1}^{u^{*}} . To address these challenges, here we propose a training-free framework by reformulating reward-guided image editing as a trajectory optimal control problem (Figure 2 ). Specifically, we treat the reverse diffusion process, originating from the source image, as a controllable trajectory. Our goal is then to find the optimal control signal that steers this entire trajectory to a terminal state that maximizes the reward. To solve this control problem, we develop an iterative adjoint-state update algorithm based on the principles of Pontryagin’s Maximum Principle (PMP) ( Levine 1972 ) . We comprehensively demonstrate the effectiveness of our approach across four distinct editing tasks (Figure Training-Free Reward-Guided Image Editing via Trajectory Optimal Control ). By optimizing the entire path, our approach shows that the resulting edits are not only effective in terms of the target reward but also structurally coherent with the source image. Our main contributions are threefold: • We present a training-free reward-guided image editing framework by formulating it as a trajectory optimal control problem, applicable to both diffusion and flow-matching models. • Based on the PMP necessary conditions, we develop an iterative adjoint-state optimization procedure to find the optimal trajectory that maximizes the target reward. • Through extensive experiments across diverse tasks, we demonstrate that our method outperforms existing inversion-based guidance baselines, achieving superior results without reward hacking or structural degradation. 2 Related works 2.1 Training-free image editing with diffusion and flow-matching models The exploitation of the pre-trained distribution from pre-trained models enabled various techniques for the image editing task. One of the most popular approaches is inversion-based methods ( Meng et al. 2021 ; Mokady et al. 2023 ; Huberman-Spiegelglas et al. 2024 ) , which map a source image to noise space through the forward process and then edit it through a modified reverse trajectory. Another direction employs distillation-based optimization ( Poole et al. 2023 ; Hertz et al. 2023 ; Nam et al. 2024 ) , which guides the source image without an explicit sampling step. Some used empirical feature alignment, such as cross-attention map ( Hertz et al. 2022 ; Cao et al. 2023 ) , to ensure the sampling output retains the source image feature. More recently, flow-matching approaches such as FlowEdit ( Kulikov et al. 2024 ) achieve optimization-free editing by directly steering text-conditional flows. Leveraging the semantic coverage of large-scale text-to-image models like Stable Diffusion ( Rombach et al. 2022 ; Esser et al. 2024 ) , editing is typically specified with natural language prompts. However, these approaches are restricted by the scope of the model’s pre-trained distribution, making it difficult to edit beyond the concepts it has trained. 2.2 Reward-Guided Image Generation Recently, modifying the generative process to align with a user-defined objective, often encapsulated by a reward function, is a central goal in controllable generation. While this can be done by explicit training for the reward-aligned distribution ( Black et al. 2023 ; Wallace et al. 2024 ) , several works aim to apply training-free guidance that steers the sampling process ( Chung et al. 2023 ; Ye et al. 2024 ; Yu et al. 2023 ; He et al. 2023 ; Song et al. 2023 ) . Leveraging off-the-shelf differentiable predictors ( e.g. , a classifier or a reward model), these approaches modify the denoising samples or their posterior mean during inference to achieve a higher reward at the end of the sampling. Nevertheless, their potential for editing has been underexplored since these methods were fundamentally designed for sampling from the noise distribution. 2.3 Steering