⁠ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control

Shelly Golan, Michael Finkelson, Ariel Bereslavsky, Yotam Nitzan, Or Patashnik, [0.6em] Tel Aviv University Lightricks Adobe Research · 2026

Read ⁠ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control by Shelly Golan, Michael Finkelson, Ariel Bereslavsky, Yotam Nitzan, Or Patashnik, [0.6em] Tel Aviv University Lightricks Adobe Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20816.txt.

Introduction

Golan, Finkelson, Bereslavsky, Nitzan, and Patashnik present ParetoSlider, a multi-goal reinforcement learning framework that trains a diffusion model to match the Pareto front. Early scalarization commits the model to one trade-off at training time, with no inference-time control over conflicting goals. By training with steadily varying preference weights, ParetoSlider lets users explore optimal trade-offs at inference time without new training or keeping multiple checkpoints. Their single conditioned model matches or exceeds baselines trained apart for fixed trade-offs, while giving fine-grained control over goals.

Catalog record

CREATORShelly Golan, Michael Finkelson, Ariel Bereslavsky, Yotam Nitzan, Or Patashnik, [0.6em] Tel Aviv University Lightricks Adobe Research
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80923
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20816

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

⁠ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control Shelly Golan, Michael Finkelson, Ariel Bereslavsky, Yotam Nitzan, Or Patashnik, [0.6em] Tel Aviv University Lightricks Adobe Research Abstract Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of “early scalarization” collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals – such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider , a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, we enable users to navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints. We evaluate ParetoSlider across three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2. Our single preference-conditioned model matches or exceeds the performance of baselines trained separately for fixed reward trade-offs, while uniquely providing fine-grained control over competing generative goals. Figure 1 : ParetoSlider enables smooth inference-time control over competing rewards trade-off via a single preference-conditioned model. Top: Text-to-image generation sliding between photorealism and sketch. Bottom: Image editing sliding between source preservation and prompt adherence. 1 Introduction Reinforcement Learning (RL) has emerged as the cornerstone for aligning Large Language Models with nuanced human intent, transforming them into highly capable systems across a vast array of real-world applications [ 32 , 38 , 5 ] . This success is now driving a parallel paradigm shift in visual generative modeling [ 11 , 3 , 7 , 42 ] . Recent advances [ 28 , 47 , 52 ] have successfully applied RL methods to diffusion and flow matching models to optimize intricate and occasionally subjective objectives such as aesthetic appeal, style fidelity, and precise adherence to complex text prompts [ 21 , 45 , 30 , 46 ] . In practice, these models are not judged based on a single metric. Rather, their utility depends on the simultaneous satisfaction of multiple rewards that jointly define the quality and utility of the generated content. In instruction-based editing, for example, a model must maximize a reward for adherence to the edit prompt while satisfying a second reward for faithfulness to the source image. These objectives are inherently in tension: pushing for more aggressive editing often degrades structural preservation. This conflict is a hallmark of multi-objective optimization (MOO), where there is typically no single “perfect” solution [ 31 , 9 ] . In practice, MOO problems are often addressed through early scalarization , which collapses multiple objectives into a single fixed weighted sum [ 31 , 9 , 37 , 16 ] . This approach requires a costly search for weighting coefficients and freezes the model at a single, static operating point, thereby forcing a permanent compromise that precludes the flexibility required to adapt to varying user-defined preferences at inference time. A more principled approach to MOO is identifying the Pareto front : the set of optimal trade-offs where no single criterion can be improved without degrading another. While recent efforts have begun addressing multi-objective alignment in visual generative models through RL (MORL), they typically face a trilemma of flexibility, efficiency, and scalability. Some methods utilize Pareto-based selection during training, but remain limited to a fixed, static trade-off at inference-time [ 25 , 26 ] . Others achieve inference-time control through model interpolation, yet this requires training and storing multiple checkpoints – a cost that scales linearly with the number of objectives [ 6 , 35 ] . Finally, training-free steering approaches offer flexibility but suffer from heavy per-step sampling overhead [ 19 ] . To bridge this gap, we introduce ParetoSlider, a multi-objective reinforcement learning (MORL) framework for diffusion alignment. Rather than restricting the model to a fixed balance of rewards during training, we introduce a preference vector \omega that specifies the desired trade-off between objectives and is provided to the model as a conditioning signal. Since the model is conditioned on the preference vector \omega , it has the foresight to produce results that abide by the current relative reward preference. Consequently, the model learns to approximate the entire Pareto front within a single set of parameters, rather than converging to a single static solution. Still, the training process requires an aggregate scalar signal to guide gradient updates. Directly aggregating raw rewards, even based on preference \omega , is suboptimal, as objectives with naturally high magnitudes or variances can dominate the learning signal, effectively overshadowing other rewards and the preference conditioning \omega . To prevent such “reward hijacking”, we introduce a late-scalarization strategy that normalizes per-reward advantages independently before the weighted aggregation with preference vector \omega . Doing so ensures that the optimization landscape is defined strictly by the target preference vector rather than the arbitrary raw scales of the underlying reward functions. To efficiently optimize this continuous formulation, we build upon DiffusionNFT [ 52 ] , an RL fine-tuning framework for flow-matching models. We compute NFT losses for each reward independently and aggregate them based on preferences \omega only at the final step. At inference time, ParetoSlider enables users to continuously control a trade-off slider, navigating between competing goals without the overhead of training multiple checkpoints. We evaluate ParetoSlider on three state-of-the-art flow-matching backbones: Stable Diffusion 3.5 [ 40 ] for text-to-image synthesis, FluxKontext [ 23 ] for instruction-based image editing, and LTX-2 [ 13 ] for text-to-video generation. Across all three domains, our single preference-conditioned model matches or exceeds the performance of multiple separately tuned early-scalarization baselines at their respective operating points, while enabling continuous inference-time control over reward weights. Through extensive ablations, we compare conditioning mechanisms for injecting the preference vector into diffusion transformers, identifying design choices that enable a smooth and continuous transition between reward trade-offs. Additionally, we compare scalarization strategies and loss formulations, demonstrating that late scalarization with per-reward losses yields more faithful adherence to the requested trade-off than early-scalarization alternatives. 2 Related Work RL Fine-Tuning of Diffusion Models. RL-based post-training for generative models falls into two paradigms: offline methods, which rely on static datasets of human preferences (e.g., DPO [ 42 ] ), and online methods, which actively sample from the model’s current policy and query explicit reward functions during training. Because online methods continuously explore the generation space, they are not bounded by the coverage of a pre-collected dataset. Early methods [ 3 , 11 ] applied REINFORCE-style policy gradients over the full denoising trajectory, whereas others [ 7 ] traded generality for efficiency by back-propagating directly through differentiable rewards. More recently, GRPO [ 38 ] was adapted to flow-matching models [ 28 , 47 , 27 ] , reducing variance via group-based normalization instead of learned value networks. DiffusionNFT [ 52 ] shifts optimization to the forward process, using implicit velocity steering (§ 3 ) and a flow-matching loss, it avoids storing or differentiating through sampled trajectories. However, all of these online methods optimize a single scalar objective or a fixed weighted combination of rewards, producing a single operating point with no inference-time control. ParetoSlider extends DiffusionNFT to

Read the complete text