A Unified Density Operator View of Flow Control and Merging Riccardo De Santi, Malte Franke, Ya-Ping Hsieh, Andreas Krause Abstract Recent progress in large-scale flow and diffusion models raised two fundamental algorithmic challenges: ( i ) (i) control-based reward adaptation of pre-trained flows, and ( i ​ i ) (ii) integration of multiple models, i.e., flow merging. While current approaches address them separately, we introduce a unifying probability-space framework that subsumes both as limit cases, and enables reward-guided flow merging , allowing principled, task-aware combination of multiple pre-trained flows (e.g., merging priors while maximizing drug-discovery utilities). Our formulation renders possible to express a rich family of operators over generative models densities , including intersection (e.g., to enforce safety), union (e.g., to compose diverse models), interpolation (e.g., for discovery), their reward-guided counterparts, as well as complex logical expressions via generative circuits . Next, we introduce R eward-Guided F low M erging ( RFM ), a mirror-descent scheme that reduces reward-guided flow merging to a sequence of standard fine-tuning problems. Then, we provide first-of-their-kind theoretical guarantees for reward-guided and pure flow merging via RFM . Ultimately, we showcase the capabilities of the proposed method on illustrative settings providing visually interpretable insights, and apply our method to high-dimensional de-novo molecular design and low-energy conformer generation. 1 Introduction Large-scale generative modeling has recently progressed at an unprecedented pace, with flow ( 34 ; 35 ) and diffusion models ( 54 ; 56 ; 28 ) delivering high-fidelity samples in chemistry ( 29 ) , biology ( 10 ) , and robotics ( 9 ) . However, adoption in real-world applications like scientific discovery led to two fundamental algorithmic challenges: (i) reward-guided fine-tuning, i.e., adapting pre-trained flows to maximize downstream utilities (e.g., binding affinity) ( 17 ; 60 ; 15 , e.g.,) , and (ii) flow merging (FM): integrating multiple pre-trained models into one ( 55 ; 36 ) , e.g., to incorporate safety constraints ( 12 ) , or unify diverse priors ( 36 ) . Crucially, so far these two problems have been treated via fundamentally distinct formulations and methods. On the contrary, in this work we ask: Can control and merging be cast in a single framework enabling task-aware merging of multiple flows? Intuitively, this would allow to merge flows in a task-aware manner, as well as adapt flows to optimize downstream rewards while leveraging information from multiple prior models. Answering this would contribute to the algorithmic-theoretical foundations of flow adaptation . Our approach To address this challenge, we first introduce a probability-space optimization framework (see Fig. 1(b) ) that recovers reward-guided fine-tuning and pure model merging as limit cases, and provably enables reward-guided flow merging (Sec. 3 ). Our formulation allows to express a rich family of operators over generative models, covering practical needs such as enforcing safety (e.g., via intersection), composing diverse models (e.g., via union), and discovery in data-scarce regions (e.g., via interpolation). However, these operators are expressed via non-linear functionals that cannot be optimized via classic RL or control schemes, as shown by 15 . To overcome this challenge, we introduce R eward-Guided F low M erging ( RFM ), a mirror descent (MD) ( 44 ) scheme that solves reward-guided and pure flow merging via a sequential adaptation process implementable via established fine-tuning methods ( 17 ; 60 , e.g.,) (Sec. 4 ). Next, we extend the algorithm proposed, to operate on the space of entire flow processes, enabling scalable and stable computation of the intersection operator (Sec. 5 ). We provide a rigorous convergence analysis of RFM , yielding first-of-its-kind theoretical guarantees for reward-guided and pure flow merging (Sec. 6 ). Ultimately, we showcase our method’s capabilities on illustrative settings, as well as on high-dimensional molecular design and conformer generation tasks (Sec. 7 ). Our contributions To sum up, in this work we contribute • A formalization of reward-guided flow merging via density operators, which unifies and generalizes reward-guided adaptation and flow merging (Sec. 3 ). • R eward-Guided F low M erging ( RFM ) , a probability-space optimization algorithm that provably solves reward-guided flow merging, and pure flow merging problems (Sec. 4 ), and a stability-enhancing extension for flow intersection, implementing a scalable mirror-descent scheme over the space of flow processes (Sec. 5 ). • Theoretical convergence guarantees for the proposed algorithms, leveraging recent insights on mirror flows, and a novel technical result of independent interest. (Sec. 6 ). • An experimental evaluation of RFM showcasing its practical relevance on both synthetic, yet illustrative settings, as well as on scientific discovery tasks, namely molecular design and conformer generation. (Sec. 7 ). 2 Background and Notation We denote the set of Borel probability measures on a set \mathcal{X} with {\bm{P}}(\mathcal{X}) , and the set of functionals over {\bm{P}}(\mathcal{X}) as {\bm{F}}(\mathcal{X}) . Generative Flow Models. Generative models aim to approximately sample novel data points from a data distribution p_{data} . Flow models tackle this problem by transforming samples X_{0}=x_{0} from a source distribution p_{0} into samples X_{1}=x_{1} from the target distribution p_{data} ( 35 ; 21 ) . Formally, a flow is a time-dependent map \psi:[0,1]\times\mathbb{R}^{d}\to\mathbb{R} such that \psi:(t,x)\to\psi_{t}(x) . A generative flow model is a continuous-time Markov process \{X_{t}\}_{0\leq t\leq 1} obtained by applying a flow \psi_{t} to X_{0}\sim p_{0} as X_{t}=\psi_{t}(X_{0}) , t\in[0,1] , such that X_{1}=\psi_{1}(X_{0})\sim p_{data} . In particular, the flow \psi can be defined by a velocity field u:[0,1]\times\mathbb{R}^{d}\to\mathbb{R}^{d} , which is a vector field related to \psi via the following ordinary differential equation (ODE), typically referred to as flow ODE : \frac{\mathrm{d}}{\mathrm{d}t}\psi_{t}(x)=u_{t}(\psi_{t}(x)) (1) with initial condition \psi_{0}(x)=0 . A flow model X_{t}=\psi_{t}(X_{0}) induces a probability path of marginal densities p=\{p_{t}\}_{0\leq t\leq 1} such that at time t we have that X_{t}\sim p_{t} . We denote by p^{u} the probability path of marginal densities induced by the velocity field u . Flow matching (FM) ( 35 ) can estimate a velocity field u^{\theta} s.t. the induced marginal densities p^{u_{\theta}} satisfy p^{u_{\theta}}_{0}=p_{0} and p^{u_{\theta}}_{1}=p_{data} , where p_{0} denotes the source distribution, and p_{data} the target data distribution. Typically FM are rendered tractable by defining p^{u}_{t} as the marginal of a conditional density p^{u}_{t}(\cdot|x_{0},x_{1}) , e.g.,: X_{t}\;|\;X_{0},X_{1}=\kappa_{t}X_{0}+\omega_{t}X_{1} (2) where \kappa_{0}=\omega_{1}=1 and \kappa_{1}=\omega_{0}=0 (e.g. \kappa_{t}=1-t and \omega_{t}=t ). Then u^{\theta} can be learned by regressing onto the conditional velocity field u(\cdot|x_{1}) ( 34 ) . As diffusion models ( 56 ) (DMs) admit an equivalent ODE formulation ( 35 , Ch. 10) , our contributions extend directly to DMs. Continuous-time Reinforcement Learning. We formulate finite-horizon continuous-time RL as a specific class of optimal control problems ( 61 ; 64 ) . Given a state space \mathcal{X} and an action space \mathcal{A} , we consider the transition dynamics governed by the following ODE: \frac{\mathrm{d}}{\mathrm{d}t}\psi_{t}(x)=a_{t}(\psi_{t}(x)) (3) where a_{t}\in\mathcal{A} is an action. We consider a state space \mathcal{X}\coloneqq\mathbb{R}^{d}\times[0,1] , and denote by (Markovian) deterministic policy a function \pi_{t}(X_{t})\coloneqq\pi(X_{t},t)\in\mathcal{A} mapping a state (x,t)\in\mathcal{X} to an action a\in\mathcal{A} such that a_{t}=\pi(X_{t},t) , and denote with p_{t}^{\pi} the marginal density at time t induced by policy \pi . Pre-trained Flow Models as an RL policy. A pre-trained flow model with velocity field u^{pre} can be interpreted as an action process a^{pre}_{t}\coloneqq u^{pre}(X_{t},t) , where a_{t}^{pre} is determined by a continuous-time RL policy via a_{t}^{pre}=\pi^{pre}(X_{t},t) ( 16 ) . Therefore, we can express the flow ODE induced by a pre-trained flow model by replacing a_{t} with a^{pre} in Eq. equation 3 , and denote the pre-trained model by its policy \pi^{pre} , which induces a density p^{pre}_{1}\coloneqq p^{\pi^{pre}}_{1} approximating p_{data} . 3 Reward-Guided Flow Merging via Implicit Density Operators (a) Reward-Guided Flow Merging (b) Probability-Space Opt. Viewpoint Figure 1: ( 1(a) ) Pre-trained and fine-tuned policies inducing \{p_{1}^{pre,i}\}_{i=1}^{n} and optimal density p_{1}^{*} computed via flow merging, i.e., subcase of Problem 5 where f is disregarded. ( 1(b) ) Probability-space optimization viewpoint on reward-guided flow merging, as in Problem 5 . In this section, we introduce the general problem of reward-guided flow merging via density operators . Formally, we wish to implement an operator \mathcal{O} : \Pi\times\ldots\times\Pi\to\Pi that, given pre-trained generative flow models \{\pi^{pre,i}\}_{i\in[n]} , returns a merged flow \pi^{*} inducing an ODE: \frac{\mathrm{d}}{\mathrm{d}t}\psi_{t}(x)=a^{*}_{t}(\psi_{t}(x))\quad\text{with}\quad a_{t}^{*}=\pi^{*}(x,t), (4) such that it controllably merges prior information within the n pre-trained generative models, while potentially steering its density p_{1}^{*}\coloneqq p_{1}^{\pi^{*}} towards a high-reward region according to a given scalar reward function f(x):\mathcal{X}\to\mathbb{R} . We implement such operators by fine-tuning an initial flow \pi^{init}\in\{\pi^{pre,i}\}_{i\in[n]} according to the following probability-space optimization problem (see Fig. 1(b) ). Reward-Guided FM via Density Operator \displaystyle\pi^{*}\in\argmax_{\pi:p_{0}^{*}=p_{0}^{pre}}\;\EV_{x\sim p_{1}^{\pi}}\left[f(x)\right]-\sum_{i=1}^{n}\alpha_{i}\mathcal{D}_{i}(p_{1}^{\pi}\,\|\,p_{1}^{pre,i}) (5) Here, each D_{i} is an arbitrary divergence, \alpha_{i}>0 are model-specific weights, and p_{0}^{\pi}=p_{0}^{pre} enforces that the marginal density at t=0 must match the pre-trained model marginal. This formulation recovers reward-guided fine-tuning ( 17 , e.g.,) when n=1 and \mathcal{D}_{1}=D_{KL} , and provides a formal framework for pure flow merging ( 46 ; 55 , e.g.,) with interpretable objectives, when the reward f is constant (e.g., f(x)=0\;\forall x\in\mathcal{X} ). In this case, Eq. 5 formalizes flow merging as computing a flow \pi^{*} that minimizes a weighted sum of divergences to the priors \{\pi^{pre,i}\}_{i\in[n]} . Varying the divergences \{D_{i}\}_{i\in[n]} yields different merging strategies. In-Distribution Flow Merging. Given pre-trained flow models \{\pi^{pre,i}\}_{i\in[n]} , we denote by in-distribution merging when the merged model generates samples from regions with sufficient prior density. Practically relevant instances include the intersection operator \mathcal{O}_{\land} (i.e., a logical AND), and the union operator \mathcal{O}_{\lor} (i.e., a logical OR). Formally, these operators can be defined as follows: \mathcal{O}_{\land} : Intersection ( \land ) Operator \displaystyle\pi^{*}\in\argmin_{\pi:p_{0}^{*}=p_{0}^{pre}}\;\sum_{i=1}^{n}\alpha_{i}\,D_{KL}(p_{1}^{\pi}\|p_{1}^{pre,i}) (6) \mathcal{O}_{\lor} : Union ( \lor ) Operator \displaystyle\pi^{*}\in\argmin_{\pi:p_{0}^{*}=p_{0}^{pre}}\;\sum_{i=1}^{n}\alpha_{i}\,D^{R}_{KL}(p_{1}^{\pi}\|p_{1}^{pre,i}) (7) The D_{KL} divergences in Eq. 6 heavily penalize density allocation in any region with low prior density for any model \pi^{pre,i} , leading to an optimal flow model \pi^{*} inducing p_{1}^{*}(x)\propto\prod_{i=1}^{n}p_{1}^{pre,i}(x)^{\alpha_{i}} ( 26 , cf.) . Similarly, the reverse KL divergence D^{R}_{KL}(p\|q)\coloneqq D_{KL}(q\|p) in Eq. 7 induces a mode-covering behaviour implying a flow model \pi^{*} with density p_{1}^{*}\propto\sum_{i=1}^{n}\alpha_{i}p_{1}^{pre,i}(x) ( 5 , cf.) sufficiently covering all regions with enough prior density, for any p_{1}^{pre,i},\;i\in[n] . Out-of-Distribution Flow Merging. We denote by out-of-distribution , the case where \pi^{*} samples from regions insufficiently covered by all priors. An example is the interpolation operator \mathcal{O}_{W_{p}} (see Eq. 8 ), inducing p_{1}^{*} equal to the priors Wasserstein Barycenter ( 11 ) . \mathcal{O}_{W_{p}} : Interpolation (Wasserstein- p Barycenter) Operator \displaystyle\argmin_{\pi}\;\sum_{i=1}^{n}\alpha_{i}W_{p}(p_{1}^{\pi}\,\|\,p_{1}^{pre,i}) (8) Straightforward Generalizations. While we presented a few practically relevant operators, the framework in Eqs. 5 is not tied to them: it trivially admits any new operator defined via other divergences (e.g., MMD, Rényi, Jensen–Shannon), and allows diverse D_{i} for each prior flow models \pi^{pre,i} . Moreover, sequential composition of these operators makes it possible to implement arbitrarily complex logical operations over generative models. For instance, as later shown in Sec. 7 , one can obtain \pi^{*}=(\pi^{pre,1}\lor\pi^{pre,2})\land\pi^{pre,3} by first computing \pi_{1,2}\coloneqq\mathcal{O}_{\lor}(\pi^{pre,1},\pi^{pre,2}) and then \pi^{*}\coloneqq\mathcal{O}_{\land}(\pi_{1,2},\pi^{pre,3}) . We denote such operators by generative circuits , and illustrate one in Fig. 4(d) . While being of high practical relevance, the presented framework entails optimizing non-linear distributional utilities (see Eq. 5 ) beyond the reach of standard RL or control schemes, as shown by 15 . In the next section, we show how to reduce the introduced problem to sequential fine-tuning for maximization of rewards automatically determined by the choice of operator \mathcal{O} . 4 Algorithm: Reward-Guided Flow Merging In this section, we introduce R eward-Guided F low M erging ( RFM ), see Alg. 1 , which provably solves Problem 5 . RFM implements general operators \mathcal{O} (see Sec. 3 ) by solving the following problem: Reward-Guided Flow Merging as Probability-Space Optimization \displaystyle p_{1}^{\pi^{*}}\in\argmax_{p_{1}^{\pi}}\;\mathcal{G}(p_{1}^{\pi})\quad\text{ with }\quad\mathcal{G}(p_{1}^{\pi})\coloneqq\EV_{x\sim p_{1}^{\pi}}\left[f(x)\right]-\sum_{i=1}^{n}\alpha_{i}\mathcal{D}_{i}(p_{1}^{\pi}\,\|\,p_{1}^{pre,i}) (9) Given an initial flow model \pi^{init}\in\{\pi^{pre,i}\}_{i\in[n]} , RFM follows a mirror descent (MD) scheme ( 44 ) for K iterations by sequentially fine-tuning \pi^{init} to maximize surrogate rewards g_{k} determined by the chosen operator, i.e., \mathcal{G} . To understand how RFM computes the surrogate rewards \{g_{k}\}_{k=1}^{K} guiding the optimization process in Eq. 9 , we first recall the notion of first variation of \mathcal{G} over a space of probability measures ( 30 , cf.) . A functional \mathcal{G}\in{\bm{F}}(\mathcal{X}) has a first variation at \mu\in{\bm{P}}(\mathcal{X}) if there exists a function \delta\mathcal{G}(\mu)\in{\bm{F}}(\mathcal{X}) such that: \mathcal{G}(\mu+\epsilon\mu^{\prime})=\mathcal{G}(\mu)+\epsilon\langle\mu^{\prime},\delta\mathcal{G}(\mu)\rangle+o(\epsilon). holds for all \mu^{\prime}\in{\bm{P}}(\mathcal{X}) , where the inner product is an expectation. At iteration k\in[K] , given the current generative model \pi^{k-1} , RFM fine-tunes it according to the following standard entropy-regularized control or RL problem, solvable via any established method ( 17 , e.g.,) \argmax_{\pi}\quad\langle\delta\mathcal{G}\left(p^{\pi_{k-1}}_{1}\right),p_{1}^{\pi}\rangle-\frac{1}{\gamma_{k}}D_{KL}(p_{1}^{\pi}\,\|\,p_{1}^{\pi_{k-1}}) (10) Thus, we introduce a surrogate reward function g_{k}:\mathcal{X}\to\mathbb{R} defined for all x\in\mathcal{X} such that: g_{k}(x)\coloneqq\delta\mathcal{G}\left(p^{\pi^{k-1}}_{1}\right)(x)\quad\text{ and }\quad\EV_{x\sim p_{1}^{\pi}}[g_{k}(x)]=\langle\delta\mathcal{G}\left(p^{\pi^{k-1}}_{1}\right),p_{1}^{\pi}\rangle (11) Algorithm 1 R eward-Guided F low M erging ( RFM ) 1: input: \{\pi^{pre,i}\}_{i\in[n]}: pre-trained flows, \{\mathcal{D}_{i}\}_{i\in[n]}: arbitrary divergences, f: reward, \{\alpha_{i}\}_{i\in[n]}: weighs, K: iterations number, \{\gamma_{k}\}_{k=1}^{K} stepsizes, \pi^{init}\in\{\pi^{pre,i}\}_{i\in[n]}: initial flow model 2: Init: \pi_{0}\coloneqq\pi^{init} 3: for k=1,2,\ldots,K do 4: Estimate \nabla_{x}g_{k}=\nabla_{x}\delta\mathcal{G}(p_{1}^{\pi^{k-1}}) with: \displaystyle\mathcal{G}\left(p_{1}^{\pi^{k-1}}\right) \displaystyle=\left\{\begin{aligned} &\EV_{x\sim p_{1}^{\pi^{k-1}}}\left[f(x)\right]-\sum_{i=1}^{n}\alpha_{i}\mathcal{D}_{i}(p_{1}^{\pi^{k-1}}\,\|\,p_{1}^{pre,i})&&\text{(Reward-Guided Flow Merging)}\\ &-\sum_{i=1}^{n}\alpha_{i}\mathcal{D}_{i}(p_{1}^{\pi^{k-1}}\,\|\,p_{1}^{pre,i})&&\text{(Flow Merging)}\end{aligned}\right. (12) 5: Compute \pi_{k} via standard reward-guided fine-tuning ( 17 , e.g.,) : \pi_{k}\leftarrow\textsc{{RewardGuidedFineTuningSolver}}(\nabla_{x}g_{k},\gamma_{k},\pi_{k-1}) 6: end for 7: output: policy \pi\coloneqq\pi_{K} We now present R eward-Guided F low M erging ( RFM ), see Alg. 1 . At each iteration k\in[K] , RFM estimates the gradient of the first variation at the previous policy \pi_{k-1} , i.e., \nabla_{x}\delta\mathcal{G}(p^{\pi^{k-1}}_{1}) (line 4 ). Then, it updates the flow model \pi_{k} by solving the reward-guided fine-tuning problem in Eq. 10 by employing \nabla_{x}g_{k}\coloneqq\nabla_{x}\delta\mathcal{G}(p^{\pi^{k-1}}_{1}) as reward function gradient (line 5 ). Ultimately, RFM returns a final policy \pi\coloneqq\pi_{K} . We report a detailed implementation of RewardGuidedFineTuningSolver in Apx. 3 . Implementation of Intersection, Union, and Interpolation operators. In the following, we present the specific expressions of \nabla_{x}\delta\mathcal{G}(p_{1}^{\pi}) for pure model merging with the intersection ( \mathcal{O}_{\land} ), union ( \mathcal{O}_{\lor} ), and interpolation ( \mathcal{O}_{W_{p}} ) operators introduced in Sec. 3 . \nabla_{x}\delta\mathcal{G}(p_{1}^{\pi})(x)=\begin{cases}-\sum_{i=1}^{n}\alpha_{i}s^{k-1}(x,t=1)+\sum_{i=1}^{n}\alpha_{i}s^{\pi^{pre,i}}(x,t=1)&\text{Intersection ($\mathcal{O}_{\land}$)}\\ -\sum_{i=1}^{n}\nabla_{x}\exp{(\phi_{i}^{*}(x)-1)},\phi_{i}^{*}\text{ as by Eq. \ref{eq:first_var_reverse_KL}}&\text{Union ($\mathcal{O}_{\lor}$)}\\ -\sum_{i=1}^{n}\nabla_{x}\phi_{i}^{*}(x),\phi_{i}^{*}=\argmax_{\phi:\|\nabla_{x}\phi\|\leq 1}\langle\phi,p^{\pi}-p^{pre,i}\rangle&\text{Interpol. ($\mathcal{O}_{W_{1}}$)}\end{cases} Where by s^{k-1}(x,t)\coloneqq\nabla\log p^{\pi-1}_{t}(x) we denote the score of model \pi^{k-1} at point x and time t , and s^{pre,i}\coloneqq s^{\pi^{pre,i}} . For diffusion models, a learned neural score network is typically available; for flows, the score follows from a linear transformation of \pi(X_{t},t) (e.g., 17 , Eq. 8) : s^{\pi}_{t}(x)=\frac{1}{\kappa_{t}(\tfrac{\dot{\omega}_{t}}{\omega_{t}}\kappa_{t}-\dot{\kappa}_{t})}\left(\pi(x,t)-\frac{\dot{\omega}_{t}}{\omega_{t}}x\right) (13) For the union operator, gradients are defined via critics \{\phi^{*}_{i}\}_{i=1}^{n} learned with the standard variational form of reverse KL, as in f-GAN training of neural samplers ( 45 ) . For W_{1} interpolation, each \phi^{*}_{i} plays the role of a Wasserstein-GAN discriminator with established learning procedures ( 2 ) . In both cases, each critic compares the fine-tuned density to a prior density p_{1}^{pre,i} , seemingly requiring one critic per prior. We prove that, surprisingly, this is unnecessary for the union operator, and conjecture that analogous results hold for other divergences. 1 Union operator via Pre-trained Mixture Density Representation. Given \overline{p}_{1}^{pre}=\nicefrac{{\sum_{i=1}^{n}\alpha_{i}p_{1}^{pre,i}}}{{\sum_{i=1}^{n}\alpha_{i}}} , i.e., the \alpha -weighted mixture density of pre-trained models, the following hold: \pi^{*}\in\argmin_{\pi}\;\sum_{i=1}^{n}\alpha_{i}\,D^{R}_{KL}(p_{1}^{\pi}\,\|\,p_{1}^{pre,i})=\argmin_{\pi}\;\left(\sum_{i=1}^{n}\alpha_{i}\right)D^{R}_{KL}(p_{1}^{\pi}\,\|\,\overline{p}_{1}^{pre}) (14) Prop. 1 , which is proved in Apx. D implies that the union operator in Eq. 7 over n prior models can be implemented by learning a single critic \phi^{*} , as shown in Sec. 7 . In Apx. C.2 , we report the gradient expressions above, and present a brief tutorial to derive the first variations for any new operator. Crucially, the score in Eq. 13 for the intersection gradient diverges at t=1 ( \kappa_{1}=0 ). While prior works attenuate the issue by evaluating the score at 1-\epsilon ( 16 ) , this trick hardly scales well to high-dimensional settings. In the following, we propose a principled solution to this problem by leveraging weighted score estimates along the entire noised flow process, i.e., t\in[0,1] . 5 Scalable Intersection via Flow Process Optimization To tackle the aforementioned issue, we lift the problem in Eq. 6 from the probability space of the last time-step marginal p_{1}^{\pi} , where the score diverges, to the entire flow process: Intersection Operator via Flow Process Optimization \displaystyle\argmax_{\pi:p_{0}^{\pi}=p_{0}^{pre}}\;\!\mathcal{L}_{\land}\!\left(\mathbf{Q}^{\pi}\right)\!\coloneqq\!\int_{0}^{1}\!\lambda_{t}\sum_{i=1}^{n}\alpha_{i}\,D_{KL}(p_{t}^{\pi}\,\|\,p_{t}^{pre,i})\;\mathrm{d}t (15) Here, \mathbf{Q}^{\pi}=\{p^{\pi}_{t}\}_{t\in[0,1]} denotes the entire joint flow process induced by policy \pi over \mathcal{X}^{[0,1]} . Under general regularity assumptions, an optimal policy \pi^{*} for Problem 15 is optimal also w.r.t. Eq. 6 . Interestingly, an optimal flow \pi^{*} for Problem 15 can be computed via a MD scheme acting over the space of joint flow processes \mathbf{Q}^{\pi}=\{p^{\pi}_{t}\}_{t\in[0,1]} determined by the following update rule: Reward-Guided FM (Mirror Descent) Step \mathbf{Q}^{k}\in\argmax_{q:p_{0}=p^{k-1}_{0}}\langle\delta\mathcal{L}_{\land}(\mathbf{Q}^{k-1}),\mathbf{Q}\rangle+\frac{1}{\gamma^{k}}D_{KL}\left(\mathbf{Q}\|\mathbf{Q}^{k-1}\right) (16) First, we state the following Lemma 5.1 , which allows to express the first variation of \mathcal{L}_{\land} w.r.t. the flow process \mathbf{Q^{\pi}} as an integral of first variations w.r.t. marginal densities p_{t}^{\pi} . 5.1 First Variation of Flow Process Functional. For objective \mathcal{L}_{\land} in Eq. 15 it holds the following: \langle\delta\mathcal{L}_{\land}(\mathbf{Q}^{k}),q\rangle\!=\!\int_{0}^{1}\!\lambda_{t}\!\EV_{\hskip 7.97224pt\mathbf{Q}}\!\left[\!\delta\sum_{i=1}^{n}\!\alpha_{i}\,D_{KL}(p_{t}^{\pi}\,\|\,p_{t}^{pre,i})\!\right]\!\mathrm{d}t (17) This factorization of \langle\delta\mathcal{L}_{\land}(\mathbf{Q}^{k}),q\rangle shows that a flow \pi_{k+1} inducing an optimal process \mathbf{Q}^{k} w.r.t. the update step in Eq. 16 can be computed by solving a control-affine optimal control problem via the same RewardGuidedFineTuningSolver oracle used in Alg. 1 , by introducing the running cost term: f_{t}(x)\coloneqq\ \delta\left(\sum_{i=1}^{n}\alpha_{i}\,D_{KL}(p_{t}^{\pi}\,\|\,p_{t}^{pre,i})\right)(x,t) (18) with \in[0,1) . This algorithmic idea, which allows to control the score scale at t\to 1 via \lambda_{t} , thus enhancing RFM , trivially extends to reward-guided merging, and is accompanied by a detailed pseudocode in Apx. 3 . 6 Guarantees for Reward-Guided Flow Merging In this section, we aim to provide rigorous convergence guarantees for RFM by interpreting its iterations as mirror descent on the space of measures. To this end, at each iteration it must hold that s^{k-1}(x,t):=\nabla_{x}\log p_{t}^{\pi^{k-1}}(x) , i.e., the RewardGuidedFineTuningSolver subroutine, employed by RFM at every iteration (see line 5 , Alg. 1 ), retains the score information . Our key contribution is to prove this result, which is not only essential for our convergence analysis, but also of independent interest: it provides a rigorous justification for a structural assumption that underlies several control-based fine-tuning analyses, where it is typically implicitly assumed ( 15 ; 16 ) . Score Retention via Control-based Fine-Tuning. Our main observation is that, under a standard assumption, control-based fine-tuning schemes retain score information. These include Adjoint Matching ( 17 ) , which we utilize in our experiments (Sec. 7 ). We consider the standard stochastic optimal control (SOC) framework, exposed in Apx. B , and show that the fine-tuned model via SOC necessarily encodes score information. 6.1 SOC Retains Score Information. Suppose that the prior diffusion model forward process converges to a standard Gaussian noise 1 1 1 This is a standard assumption in diffusion modeling ( 28 ; 57 , e.g.,) . . Then, the model returned by a SOC fine-tuning solver is such that: u^{\star}(x,t)\coloneqq\sigma(t)\,\nabla\log p^{k}_{t}(x) (19) where p^{k}_{t} denotes the marginal distribution of the prior forward process, initialized at p^{\pi_{k}}_{1} , and u^{\star}(x,t) the applied optimal control (see Sec. B ). In other words, RewardGuidedFineTuningSolver exactly recovers the score. Theorem 6.1 enables us to reinterpret Algorithm 1 as generating approximate mirror iterates , a framework that has proven effective for sampling and generative modeling ( 31 ; 16 ; 15 ) . Robust Convergence under Inexact Updates. Thanks to Theorem 6.1 , we can now develop a rigorous convergence theory for Algorithm 1 under the realistic condition that RewardGuidedFineTuningSolver (see Sec. 4 ) is implemented approximately . Let \mathcal{G} be the objective in Eq. 9 . Via \pi^{k} , the iterates generated by Algorithm 1 induce a sequence of stochastic processes, denoted by \mathbf{Q}^{k} , which satisfy \mathbf{Q}^{k}=p_{1}^{\pi^{k}} . Each iterate \mathbf{Q}^{k} is understood as an approximation to the idealized mirror descent step: \mathbf{Q}^{k}_{\sharp}\in\argmax_{\mathbf{Q}:p_{0}=p_{0}^{pre}}\Bigl\{\langle\delta\mathcal{G}(p^{\pi_{k}}_{1}),\mathbf{Q}\rangle-\tfrac{1}{\gamma^{k}}D_{KL}\!\left(\mathbf{Q}\,\|\,\mathbf{Q}^{k-1}\right)\Bigr\}. (20) which serves as the exact reference point for our analysis. To quantify the discrepancy between \mathbf{Q}^{k} and \mathbf{Q}_{\sharp}^{k} , let \mathcal{T}_{k} denote the history up to step k , and decompose the error as \displaystyle b_{k} \displaystyle\coloneqq\EV\!\left[\delta\mathcal{G}(p^{\pi_{k}}_{1})-\delta\mathcal{G}((\mathbf{Q}_{\sharp}^{k})_{1})\,\big|\,\mathcal{T}_{k}\right], (21) \displaystyle U_{k} \displaystyle\coloneqq\delta\mathcal{G}(p^{\pi_{k}}_{1})-\delta\mathcal{G}((\mathbf{Q}_{\sharp}^{k})_{1})-b_{k}. (22) Here, b_{k} captures systematic approximation error, and U_{k} represents a zero-mean fluctuation conditional on \mathcal{T}_{k} . Under mild assumptions over noise and bias (see Section B.2 ), the long-term behavior of the iterates can be characterized. 6.2 Asymptotic convergence under inexact updates (Informal). Assume the oracle has bounded variance and diminishing bias, and the step sizes \{\gamma^{k}\} satisfy the Robbins–Monro conditions ( \sum_{k}\gamma^{k}=\infty , \sum_{k}(\gamma^{k})^{2}<\infty ). Then the sequence \{p^{\pi_{k}}_{1}\} generated by Algorithm 1 converges almost surely to the optimum in the weak sense: p^{\pi_{k}}_{1}\rightharpoonup\tilde{p}_{1}\quad\text{a.s.}, (23) where \tilde{p}_{1} is a stationary point of \mathcal{G} . Remark. Several functionals \mathcal{G} considered in this work (e.g., forward and reverse KL, Jensen–Shannon, and their f -guided counterparts etc.) are convex over the space of measures. In these cases, Thm. 6.2 trivially strengthens to convergence to a global optimum: p^{\pi_{k}}_{1}\rightharpoonup p_{1}^{\star}=\mathbf{Q}_{1}^{\star} for some \mathbf{Q}^{\star}\in\arg\max_{\mathbf{Q}:\,\mathbf{Q}_{0}=p_{0}^{\mathrm{pre}}}\mathcal{G}(\mathbf{Q}_{1}) , as shown in Apx. B.2 . (a) Pre-trained samples (b) AND Balanced (c) AND reward up (d) AND reward up (e) Pre-trained samples (f) OR Balanced (g) OR \alpha=[0.1,0.9] (h) OR optimization (i) Pre-trained samples (j) INTR \pi^{init}=\pi^{pre,1} (k) INTR \pi^{init}=\pi^{pre,2} (l) Reward-guided INTR Figure 2: Illustrative settings with visually interpretable results. (top) Flow model balanced pure intersection ( 2(b) ), and reward-guided intersection ( 2(c) ), (mid) Flow balanced and unbalanced union, (bottom) Flow model pure and reward-guided interpolation. Crucially, RFM can correctly implement these practically relevant and diverse operators with high degree of expressivity (e.g., \alpha , reward-guidance). 7 Experimental Evaluation We evaluate RFM for the reward-guided flow merging problem (see Eq. 5 ) by tackling two types of experiments: (i) visually interpretable illustrative settings, showcasing the correctness and high expressivity of RFM , and (2) high-dimensional molecular design and conformer generation tasks. Further experimental details are reported in Apx. 8 Intersection Operator \mathcal{O}_{\land} . We consider pre-trained flow models inducing densities p_{1}^{pre,1} (green) and p_{1}^{pre,2} (violet), as shown in Fig. 2(a) . We fine-tune \pi^{init}\coloneqq\pi^{pre,1} via RFM to compute the policy \pi^{*} resulting from diverse intersection operations \pi^{*}=\mathcal{O}_{\land}(\pi^{pre,1},\pi^{pre,2}) . First, in Fig. 2(b) , we show p^{*} (black) obtained by RFM with \alpha=[0.5,0.5] , i.e., balanced (B). One can notice that the flow model p^{*} covers mostly the intersecting regions between p_{1}^{pre,1} and p_{1}^{pre,2} (see Fig. 2(a) ). In Fig. 2(c) we report an instance of reward-guided intersection (RG) for a reward function maximized upward. As one can see, RFM computes a policy \pi^{*} placing density over the highest-reward region among the intersecting ones, i.e., the top intersecting area. This reward-guided flow merging process is carried out via maximization over K=15 iterations of the objective \mathcal{G} illustrated in Fig. 2(d) . Union Operator \mathcal{O}_{\lor} . We fine-tune the pre-trained flow model \pi^{init}=\pi^{pre,1} with density illustrated in Fig. 2(e) (green) via RFM to implement balanced (i.e., \alpha=[0.5,0.5] and unbalanced (i.e., \alpha=[0.1,0.9] (UB)) versions of the union operator, namely computing \pi^{*}=\mathcal{O}_{\lor}(\pi^{pre,1},\pi^{pre,2}) . As shown in Fig. 2(f) and 2(g) RFM can successfully compute optimal policies \pi^{*} implementing both operators via optimization of the functional \mathcal{G} , corresponding to sum of weighted KL-divergences (see Eq. 7 ) evaluated for iterations k\in[K] with K=13 in Fig. 2(h) . Interpolation Operator \mathcal{O}_{W_{1}} . We use RFM to compute flows \pi^{*} inducing p_{1}^{*} corresponding to diverse interpolations between the the pre-trained models’ densities illustrated in Fig. 2(i) . Although the optimal policy to which RFM converges asymptotically is invariant w.r.t. the initial flow model \pi^{init} chosen for fine-tuning, here we show that this choice can actually be used to control the algorithm execution over few iterations (i.e., K=6 ). As one can expect, Fig. 2(j) and 2(k) show that the result density after K=6 iterations is closer to the flow model chosen as \pi^{init} , namely \pi^{pre,1} (green) in Fig. 2(j) and \pi^{pre,2} (violet) in Fig. 2(k) . We illustrate in Fig. 2(l) the density (black) obtained via reward-guided interpolation, with a reward function maximized left upwards. Complex Logical Expressions via Generative Circuits. We consider 4 flow models \{\pi_{pre,i}\}_{i=1}^{4} as in Fig. 4(a) , which we aim to merge into one flow \pi^{*} determined by the logical expression \pi^{*}=(\pi_{1}\land\pi_{2})\lor(\pi_{3}\land\pi_{4}) . We implement the generative circuit in Fig. 4(d) via sequential use of RFM . First, we compute \pi_{5}\coloneqq\mathcal{O}_{\land}(\pi^{pre,1},\pi^{pre,2}) and \pi_{6}\coloneqq\mathcal{O}_{\land}(\pi^{pre,3},\pi^{pre,4}) , shown in Fig. 4(b) , and subsequently \pi^{*}\coloneqq\mathcal{O}_{\lor}(\pi^{pre,3},\pi^{pre,4}) , as illustrated in Fig. 4(c) . This illustrative experiments confirms that RFM can implement complex logical expressions over generative models via generative circuits, as the simple one just presented. Low-Energy Molecular Design via Flow Merging Navigating chemical space to discover novel structures with desirable properties is a central goal of data-driven molecular design. A generative model must produce diverse, chemically valid structures that follow specified property profiles and constraints. We base our case study on two FlowMol models \pi^{pre,1} and \pi^{pre,2} ( 19 ) pre-trained on GEOM-Drugs ( 4 ) with different levels of single-point total energy at GFN1-xTB level of theory ( 22 ) , -14.8 and -8.1 Ha as shown in Fig. 4(e) . We aim to compute a flow model that generates molecules whose total energy is likely under both generative models. To this end, we run RFM to compute the flow \pi^{*} returned by the intersection operator (see Eq. 6 ), with parameters detailed in Apx. 8 . Figure 3: Drug-like molecules generated by \pi_{AND}^{*} flow via RFM . We report the density p_{1}^{*} (black) computed via balanced merging (i.e., \alpha_{i}=1 ) in Fig. 4(f) and the one obtained via unbalanced merging (i.e., \alpha_{1}=1,\alpha_{2}=2 ) in Fig. 4(g) . In the former case, p_{1}^{*} correctly places the majority of its density on the overlapping region between the two priors within [-20,0] Ha (see Fig. 4(f) ). The estimated mean energy of \pi^{*} (black) i.e., -10.95\pm 0.28 Ha, reported along with validity in 4(h) matches the energy value of maximal overlap between \pi^{pre,1} and \pi^{pre,2} , as one can see in 4(e) . Adding reward-guidance leads to lower energy values compared to the balanced merging model while keeping its high validity. In the unbalanced case, RFM shifts the density slightly leftwards, effectively implementing the \alpha -weighted intersection. We report energy-validity metrics resulting from balanced and unbalanced intersection in Fig. 4(h) , and compare them with their reward-guided counterpart in Table 1 . Next, we compute via RFM the union operator over two FlowMol pre-trained on the QM9 dataset ( 49 ) . We parametrize critics \phi_{i}^{*} (see Sec. 1 ) via the FlowMol latent representation with an MLP readout layer. Figure 7 shows that the estimated mean of the model \pi^{*} obtained via RFM matches the average total energy of \pi^{pre,1} and \pi^{pre,2} as predicted by the closed-form expression for union from Sec. 3 . (a) Pre-trained samples (b) \pi_{5} and \pi_{6} (c) Circuit output \pi^{*} (d) Generative circuit (e) Pre-trained samples (f) AND molecules (g) AND \alpha=[0.33,0.66] (h) Validity-Energy Figure 4: (top) RFM implements a generative circuit ( 4(d) ) describing a complex logical expressions ( \pi^{*}=(\pi_{1}\land\pi_{2})\lor(\pi_{3}\land\pi_{4}) ) by computing sequential operators ( 4(a) - 4(c) ). (bottom) RFM computes a flows intersection \pi^{*} generating drug molecules with desired energy levels. (a) Energy E (b) Dipole moment \mu (c) HOMO-LUMO gap \Delta\epsilon (d) Min energy E_{min} Figure 5: RFM can perform balanced (B), unbalanced (UB), reward-guided (RG) intersections, as well as unions (UNION) of ETFlow ( 24 ) conformer generation models. We evaluate the resulting flows in terms of median absolute errors of energy ( 5(a) ), dipole moment ( 5(b) ), HOMO–LUMO gap ( 5(c) ), and minimum energy ( 5(d) ). These results demonstrate the ability of RFM to compute new flow models whose properties predictably interpolate those of the available pre-trained flows. Reward-Guided FM of Conformer Generation Models Inferring 3D conformers from a molecule’s topology is a key prerequisite for many computational chemistry applications including molecular docking ( 37 ) , thermodynamic property prediction ( 47 ) , and modeling reaction pathways for catalyst design ( 51 ) . Given two prior ETFlow models (i.e., PRE-1 and PRE-2) ( 24 ) with different property profiles, and we aim to merge them into a conformer generator whose profiles controllably interpolate between, or slightly improve upon their initial energetic ensemble property profile. In particular, we evaluate errors in energy, dipole moment, HOMO-LUMO gap and minimum energy of the generated structure ensemble compared to the equilibrium ensemble. We run RFM initialized from PRE-2, to compute its balanced (B), unbalanced (UB), reward-guided (RG) intersection, and union variants. Figure 5(a) shows that the median absolute error (MAE) on the total energy E interpolates between PRE-1 and PRE-2, RFM -B and RFM -UNION achieve intermediate errors of \approx 0.3356 and 0.3352 kcal/mol as expected. On the other hand, the reward-guided (i.e., via energy minimization) counterpart reaches lower energy values, namely \approx 0.3193 kcal/mol, and the unbalanced variant ( \alpha_{1}=0.7,\alpha_{2}=0.3 ) remains near PRE-1 at 0.3412 kcal/mol. These results validate the ability of RFM to perform unbalanced and reward-guided intersection. We report similar results for the dipole moment \mu in Fig. 5(b) , for the HOMO–LUMO gap \Delta\epsilon 5(c) , and minimum energy E_{\min} 5(d) . Our evaluation indicates that RFM can perform (reward-guided) flow merging with conformer generation models, leading to flows with controllable interpolations or improvements over property profiles of pre-trained models. Moreover, in Apx. F , we briefly investigate the computational cost of Reward-Guided Flow Merging. Ultimately, although this work is primarily motivated by scientific discovery problems, we report in Apx. H an illustrative application of RFM to image-generation diffusion models. 8 Related Work Flow models fine-tuning via optimal control. Several works have framed fine-tuning of flow and diffusion models to maximize expected reward functions under KL regularization as an entropy-regularized optimal control problem ( 59 ; 58 ; 60 ; 17 ; 23 , e.g.,) . More recently, 15 introduced a framework for distributional fine-tuning. The problem tackled in this work (see Eq. 5 ) instead extends the orthogonal setting of expected rewards with arbitrary divergences to the case with n>1 pre-trained models. This generalization ( i ) unifies flow control and merging, ( ii ) renders possible to use of scalable control-based or RL schemes ( 17 , e.g.,) for flow merging, and ( iii ) enables reward-guided flow merging. Flow model merging and inference-time composition. Recent works in inference-time flow and diffusion model composition introduced theory-backed schemes ( 53 ; 8 ; 18 , e.g.,) . On the other hand, our work tackles the problem of (reward-guided) flow merging ( 55 , e.g.,) , a significantly less explored research problem. Crucially, while inference-time flow composition aims to compose models at sampling time, flow merging aims to combine multiple flow models into one, and then disregard the prior models. This work provides a formal probability-space viewpoint on the latter problem, introduces interpretable merging operators (see Sec. 3 ) for highly expressive compositions (e.g., via generative circuits), provably implemented by RFM , which is to our knowledge the first scheme for provable reward-guided flow merging. Moreover, to our knowledge, the theoretical guarantees in Sec. 6 are first-of-their-kind for merging of flow and diffusion models. In particular, specializing them to specific operators e.g., intersection, yields highly relevant results, such as generative models safety guarantees via intersection with prior safe models. Convex and general utilities reinforcement learning. Convex and General (Utilities) RL ( 25 ; 62 ; 63 ) generalizes RL to maximization of a concave ( 25 ; 62 ) or general ( 63 ; 6 ) functional of the state distribution induced by a policy over a dynamical system’s state space. Recent works tackled the finite samples budget setting ( 43 ; 41 ; 42 ; 14 ; 13 , e.g.,) . Similarly to previous optimization schemes for diffusion models ( 16 ; 15 ) , our framework (in Eq. 5 ) is related to Convex and General RL, with p^{\pi}_{1} being the state distribution induced by policy \pi over a subset of the flow process state space. Optimization over probability measures via mirror flows. Recently, there has been a growing interest in devising theoretical guarantees for probability-space optimization problems in diverse fields of application. These include optimal transport ( 3 ; 33 ; 31 ) , kernelized methods ( 20 ) , GANs ( 30 ) , and manifold exploration ( 16 ) among others. To our knowledge, we present the first use of this theoretical framework to establish guarantees for flow and diffusion models merging. 9 Conclusion We introduce a probability-space optimization framework for reward-guided flow merging, unifying and generalizing existing formulations. This allows to express diverse practically relevant operators over generative model densities (e.g., intersection, union, interpolation, logical expressions, and their reward-guided counterparts). We propose Reward-Guided Flow Merging, a mirror-descent scheme reducing complex merging tasks to sequential standard fine-tuning steps, solvable by established methods. Leveraging advances in mirror flows theory, we provide first-of-their kind guarantees for (reward-guided) flow merging. Empirical results on interpretable settings, molecular design, and conformer generation tasks demonstrate that our approach can steer pre-trained models to implement reward-guided merging tasks of high practical relevance. Appendix A Appendix Appendix B Proofs for Section 6 B.1 Proof of Theorem 6.1 Stochastic Optimal Control. We consider stochastic optimal control (SOC), which studies the problem of steering a stochastic dynamical system to optimize a specified performance criterion. Formally, let (X_{t}^{u})_{t\in[0,1]} be a controlled stochastic process satisfying the stochastic differential equation (SDE) \mathrm{d}X_{t}^{u}=b(X_{t}^{u},t)\,\mathrm{d}t+\sigma(t)\,u(X_{t}^{u},t)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}B_{t},\qquad X_{0}^{u}\sim p_{0}, (24) where u\in\mathcal{U} is an admissible control and B_{t} is standard Brownian motion. The objective is to select u to minimize the cost functional \EV\Bigg[\int_{0}^{1}\frac{1}{2}\|u(X_{t}^{u},t)\|^{2}\,dt-g(X_{1}^{u})\Bigg], (25) where \frac{1}{2}\|u(\cdot,t)\|^{2} represents the running cost and g is a terminal reward. A standard application of Girsanov’s theorem shows that Equation 25 is equivalent to the mirror descent iterate in Equation 20 with \delta\mathcal{G}(p_{1}^{\pi_{k}})\leftarrow g and p_{0}\leftarrow p^{pre} ( 58 ) . In addition, it is well-known that in the context of diffusion-based generative modeling, the corresponding uncontrolled dynamics \mathrm{d}X_{t}=-b(X_{t},t)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}B_{t} (26) coincide with the forward noising process used in score-based models ( 57 ; 17 ) . Proof of Theorem 6.1 . See 6.1 Proof. Step 1. Let \mathbf{Q}^{\star} denote the optimal process solving Equation 24 . A standard application of Girsanov’s theorem shows that \mathbf{Q}^{\star} also solves the Schrödinger bridge problem \min_{\begin{subarray}{c}\mathbf{Q}_{0}=p^{\text{pre}}\\ \mathbf{Q}_{1}=\mathbf{Q}^{\star}_{1}\end{subarray}}D_{\mathrm{KL}}\!\big(\mathbf{Q}\,\|\,\mathbf{P}\big), (27) where \mathbf{P} is the law of the uncontrolled dynamics \mathrm{d}X_{t}=b(X_{t},t)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}B_{t}. This equivalence holds because the SOC cost in Equation 24 penalizes control energy in the same way that Girsanov’s theorem expresses a controlled SDE as a relative entropy with respect to its uncontrolled counterpart. Step 2. Define the forward process \mathbf{P}_{\text{forward}} by \mathrm{d}X_{t}=-b(X_{t},t)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}B_{t}. (28) By assumption, this process maps any initial distribution to the standard Gaussian at t=1 . In particular, starting from X_{0}\sim\mathbf{Q}^{\star}_{1} , we obtain X_{1}\sim p^{\text{pre}}=\mathcal{N}(0,I) . Step 3. Consider the time-reversed Schrödinger bridge problem \min_{\begin{subarray}{c}\overleftarrow{\mathbf{Q}}_{0}=\mathbf{Q}^{\star}_{1}\\ \overleftarrow{\mathbf{Q}}_{1}=p^{\text{pre}}\end{subarray}}D_{\mathrm{KL}}\!\big(\overleftarrow{\mathbf{Q}}\,\|\,\mathbf{P}_{\text{forward}}\big), (29) and denote its solution by \overleftarrow{\mathbf{Q}}^{\star} . Since relative entropy is invariant under bijective mappings and time-reversal is bijective, the optimizers of Equation 27 and Equation 29 satisfy \overleftarrow{\mathbf{Q}}^{\star}\;=\;\overleftarrow{\hskip 3.41418pt\mathbf{Q}^{\star}\hskip 2.84544pt} i.e., the optimal reversed bridge is simply the time-reversal of the forward bridge. By Step 2 , the process \mathrm{d}X_{t}=-b(X_{t},t)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}B_{t},\qquad X_{0}\sim\mathbf{Q}^{\star}_{1} (30) solves Equation 29 , achieving the minimum relative entropy (zero) while satisfying the prescribed marginals. Thus, invoking the relation \overleftarrow{\mathbf{Q}}^{\star}=\overleftarrow{\mathbf{Q}^{\star}} , the solution to Equation 27 —and hence to the SOC problem Equation 24 —is given by the time-reversal of Equation 30 . Finally, applying the classical time-reversal formula ( 1 ) yields that \mathbf{Q}^{\star} is given by \mathrm{d}{X}_{t}=\Big(b(\overleftarrow{X}_{t},t)+\sigma^{2}(t)\,\nabla\log p_{t}({X}_{t})\Big)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}{B}_{t}, where p_{t} is the marginal density of Equation 30 . Hence, RewardGuidedFineTuningSolver exactly recovers the score function. ∎ B.2 Rigorous Statement and Proof of Theorem 6.2 To prepare for the convergence analysis, we impose a few auxiliary assumptions. These assumptions are standard in the study of stochastic approximation and gradient flows, and typically hold in practical situations. Our proof strategy follows ideas that have also been employed in related works ( 16 ; 15 ) . We begin with the entropy functional defined on probability measures: \mathcal{H}(p)\coloneqq\int p\log p. (31) In our analysis, \mathcal{H} serves as the mirror map or distance-generating function ( 38 ; 30 ) . The first condition addresses the behavior of the corresponding dual variables. B.1 Precompactness of Dual Iterates. The sequence of dual elements \{\delta\mathcal{H}(p_{1}^{\pi_{k}})\}_{k} is precompact in the L_{\infty} topology. This compactness property ensures that the interpolated dual trajectories remain confined to a bounded region of function space. Such a condition is crucial for invoking convergence results based on asymptotic pseudotrajectories. Variants of this assumption have appeared in the literature on stochastic approximation and continuous-time embeddings of discrete algorithms ( 7 ; 30 ; 38 ) . B.2 Noise and Bias Conditions. For the stochastic approximations used in the updates, we assume that almost surely: \displaystyle\|b_{k}\|_{\infty}\to 0, (32) \displaystyle\sum_{k}\EV\!\left[\gamma_{k}^{2}\big(\|b_{k}\|_{\infty}^{2}+\|U_{k}\|_{\infty}^{2}\big)\right]<\infty, (33) \displaystyle\sum_{k}\gamma_{k}\|b_{k}\|_{\infty}<\infty. (34) These conditions, standard in the Robbins–Monro setting ( 50 ; 7 ; 30 ) , guarantee that the stochastic bias vanishes asymptotically while the cumulative noise remains under control. Together, they ensure that random perturbations do not obstruct convergence to the optimizer of the limiting objective. With these assumptions in place, we can now state and prove the convergence guarantee. B.1 Convergence guarantee in the trajectory setting. Suppose Assumptions B.1 – B.2 hold, and the step sizes \{\gamma_{k}\} follow the Robbins–Monro conditions ( \sum_{k}\gamma_{k}=\infty , \sum_{k}\gamma_{k}^{2}<\infty ). Then the sequence \{p^{\pi_{k}}_{1}\} generated by Algorithm 1 converges almost surely, in the weak topology, to the optimum: p^{\pi_{k}}_{1}\rightharpoonup p_{1}^{*}\quad\text{a.s.}, (35) where p_{1}^{*}=\mathbf{Q}^{*}_{1} for some \mathbf{Q}^{*}\in\argmax_{\mathbf{Q}:\mathbf{Q}_{0}=p_{0}^{pre}}\mathcal{G}(\mathbf{Q}_{1}) . Proof. We analyze the continuous-time mirror flow defined by \dot{h}_{t}=\delta\mathcal{G}(p^{t}_{1}),\qquad p^{t}_{1}=\delta\mathcal{H}^{\star}(h_{t}), (36) where the Fenchel conjugate of \mathcal{H} is given by \mathcal{H}^{\star}(h)=\log\int e^{h} ( 30 ; 27 ) . To link the discrete dynamics to this continuous flow, we construct a piecewise linear interpolation of the iterates: \hat{h}_{t}=h^{(k)}+\frac{t-\tau_{k}}{\tau_{k+1}-\tau_{k}}\big(h^{(k+1)}-h^{(k)}\big),\quad h^{(k)}=\delta\mathcal{H}(p^{\pi_{k}}_{1}),\quad\tau_{k}=\sum_{r=0}^{k}\alpha_{r}, where \{\alpha_{r}\} denotes the step-size sequence. This interpolation produces a continuous path \hat{h}_{t} that tracks the discrete updates as the steps shrink. Let \Phi_{u} denote the flow map of equation 36 at time u . Standard results in stochastic approximation ( 7 ; 30 ; 38 ) imply that for any fixed horizon T>0 , there exists a constant C(T) such that \sup_{0\leq u\leq T}\|\hat{h}_{t+u}-\Phi_{u}(\hat{h}_{t})\|\leq C(T)\Big[\Delta(t-1,T+1)+b(T)+\gamma(T)\Big], where \Delta accounts for cumulative noise, b for bias, and \gamma for step-size effects. Under Assumptions B.1 – B.2 , these quantities vanish asymptotically, ensuring that \hat{h}_{t} forms a precompact asymptotic pseudotrajectory (APT) of the mirror flow. By the APT limit set theorem ( 7 , Thm. 4.2) , the limit set of a precompact APT is contained in the internally chain transitive (ICT) set of the underlying flow. In our case, Equation 36 corresponds to a gradient-like flow in the Hellinger–Kantorovich geometry ( 39 ) , with \mathcal{G} serving as a strict Lyapunov function. As \mathcal{G} decreases strictly along non-stationary trajectories, the ICT set reduces to the collection of stationary points of \mathcal{G} . Ultimately, notice that if \mathcal{G} is composed of convex divergences (e.g., forward or reverse KL terms) possibly together with a linear component (i.e., any reward function f ), its stationary points coincide with its global maximizers. Consequently, \hat{h}_{t} converges almost surely to the set of maximizers of \mathcal{G} , which establishes the claim. Moreover, notice that within Sec. 6 , we report the previously proved statement for stationary-points and then specialize it for global maximizers in the case of convex functionals. ∎ Appendix C Derivations of Gradients of First Variation C.1 A brief tutorial on first variation derivation In this work, we focus on the functionals that are Fréchet differentiable: Let V be a normed spaces. Consider a functional F:V\rightarrow\mathbb{R} . There exists a linear operator A:V\rightarrow\mathbb{R} such that the following limit holds \lim_{\|h\|_{V}\rightarrow 0}\frac{|F(f+h)-F(f)-A[h]|}{\|h\|_{V}}=0. (37) We further assume that V has enough structure such that every element of its dual (the space of bounded linear operator on V ) admits a compact representation. For example, if V is the space of bounded continuous functions with compact support, there exists a unique positive Borel measure \mu with the same support, which can be identified as the linear functional. We denote this element as \delta F[f] such that \langle\delta F[f],h\rangle=A[h] . Sometimes we also denote it as \frac{\delta F}{\delta f} . We will refer to \delta F[f] as the first-order variation of F at f . In the following, we briefly present standard strategies to derive the first-order variation of two broad classes of functionals, including a wide variety of divergence measures, which can be employ to implement novel operators by Eq. 5 . We consider: (i) those defined in closed form with respect to the density (e.g., forward KL) and, (ii) those defined via variational formulations (e.g., Wasserstein distance, reverse KL, and MMD). • Category 1: Functional defined in a closed form with respect to the density. For this class of functionals, the first-order variations can typically be computed using its definition and chain rule. Recalling the definition of first variation ( 37 ), we can calculate the first-order variation of the mean functional, as a trivial example. Given a continuous and bounded function r:\mathbb{R}^{d}\rightarrow\mathbb{R} and a probability measure \mu on \mathbb{R}^{d} , define the functional F(\mu)=\int r(x)\mu(x)dx . Then we have: |F(\mu+\delta\mu)-F(\mu)-\langle r,\delta\mu\rangle|=0. (38) Therefore we obtain that: \delta F[\mu]=r for all \mu . In the following section, we compute similarly the first variation of the KL divergence. • Category 2: Functionals defined through a variational formulation. Another fundamental subclass of functionals that plays a central role in this work is the one of functionals defined via a variational problem F[f]=\sup_{g\in\Omega}G[f,g], (39) where \Omega is a set of functions or vectors independent of the choice of f , and g is optimized over the set \Omega . We will assume that the maximizer g^{*}(f) that reaches the optimal value for G[f,\cdot] is unique (which is the case for the functionals considered in this project). It is known that one can use the Danskin’s theorem (also known as the envelope theorem) to compute \frac{\delta F[f]}{\delta f}=\partial_{f}G[f,g^{*}(f)], (40) under the assumption that F is differentiable ( 40 ) . C.2 Derivation of First Variations used in Sec. 4 In the following, we derive explicitly the first variations employed in Sec. 1 • Optimal transport and Wasserstein-p distance (Category 2) Consider the optimal transport problem \mathrm{OT}_{c}(u,v)=\inf_{\gamma}\left\{\int\int c(x,y)d\gamma(x,y):\int\gamma(x,y)dx=u(y),\int\gamma(x,y)dy=v(x)\right\} (41) where \Gamma=\left\{\gamma:\int\gamma(x,y)dx=u(y),\int\gamma(x,y)dy=v(x)\right\} It admits the following equivalent dual formulation \displaystyle\mathrm{OT}_{c}(u,v)=\sup_{f,g}\left\{\int fdu+\int gdv:f(x)+g(y)\leq c(x,y)\right\} (42) By taking c(x,y)=\|x-y\|^{p} , we recover \mathrm{OT}_{c}(u,v)=W_{p}(u,v)^{p} . Let \phi^{*} and g^{*} be the solution to the above dual optimization problem. From the Danskin’s theorem, we have \frac{\delta}{\delta u}W_{p}(u,v)^{p}=\phi^{*}. (43) In the special case of p=1 , we know that g^{*}=-\phi^{*} (note that the constraint can be equivalently written as \|\nabla\phi\|\leq 1 ), in which case \phi^{*} is typically known as the critic in the Wasserstein-GAN framework ( 2 , cf. ) . • Reverse KL divergence (Category 2) We use the variational (Fenchel–Legendre) representation of the forward KL, D_{KL}(p\|q) , as in f-GAN ( 45 ) : D_{KL}(p\|q)=\sup_{\phi:\mathcal{X}\to\mathbb{R}}\left\{\EV_{p}\phi(x)-\EV_{q}e^{\phi(x)-1}\right\} (44) which follows from the general f-divergence dual generator f(u)=u\log u-u+1 whose conjugate is f^{*}(t)=e^{t-1} . For fixed p and variable q , we define: G(q,\phi)\coloneqq\EV_{p}\phi(x)-\EV_{q}e^{\phi(x)-1} (45) Assuming uniqueness of a maximizer \phi^{*}(p,q) , Danskin’s (or envelope) theorem yields the first variation by differentiating G at \phi^{*} : \frac{\delta}{\delta q(x)}D_{KL}(p\|q)=\frac{\delta}{\delta q(x)}\left(-\int q(x)e^{\phi^{*}(x)-1}\mathrm{d}u\right)=-e^{\phi^{*}(x)-1} (46) • KL divergence (Category 1) Consider the KL functional: D_{KL}(p\|q)=-\int p\log\frac{p}{q},\mathrm{d}x (47) By the definition of the first-order variation (see Eq. 37 ), we have: \delta D_{KL}(p\|q)=\log\frac{p}{q}+1 (48) Appendix D Proof of Proposition 1 For the union operator, gradients are defined via critics \{\phi^{*}_{i}\}_{i=1}^{n} learned with the standard variational form of reverse KL, as in f-GAN training of neural samplers ( 45 ) . For W_{1} interpolation, each \phi^{*}_{i} plays the role of a Wasserstein-GAN discriminator with established learning procedures ( 2 ) . In both cases, each critic compares the fine-tuned density to a prior density p_{1}^{pre,i} , seemingly requiring one critic per prior. We prove that, surprisingly, this is unnecessary for the union operator, and conjecture that analogous results hold for other divergences. See 1 Prop. 1 , which is proved in the following, implies that the union operator in Eq. 7 over n prior models can be implemented by learning a single critic \phi^{*} , as shown in Sec. 7 . Proof. We prove the statement for n=2 , which trivially generalizes to any n . We first rewrite the LHS optimization problem as: \argmin_{\pi}\mathcal{F}(p^{\pi}) (49) where we denote p_{1}^{\pi} by p^{\pi} for notational concision and define p_{1}=p^{pre,i} and p_{2}=p^{pre,2} . Then we have: \displaystyle\mathcal{F}(p^{\pi}) \displaystyle=\alpha_{1}\EV_{p_{1}}[\log p_{1}-\log p^{\pi}]+\alpha_{2}\EV_{p_{2}}[\log p_{2}-\log p^{\pi}] (50) \displaystyle=\alpha_{1}\EV_{p_{1}}\log p_{1}+\alpha_{2}\EV_{p_{2}}\log p_{2}-\left(\alpha_{1}\EV_{p_{1}}\log p^{\pi}+\alpha_{2}\EV_{p_{2}}\log^{\pi}\right) (51) We now write the following, where \bar{p} denotes \bar{p}^{pre}_{1} : \displaystyle\EV_{\bar{p}}\log p^{\pi} \displaystyle=\int\log p^{\pi}(x)\bar{p}(x)\;\mathrm{d}x (52) \displaystyle=\int\log p^{\pi}(x)\left[\frac{\alpha_{1}p_{1}}{\alpha_{1}+\alpha_{2}}+\frac{\alpha_{2}p_{2}}{\alpha_{1}+\alpha_{2}}\right](x)\;\mathrm{d}x (53) \displaystyle=\frac{1}{\alpha_{1}+\alpha_{2}}\left(\log p^{\pi}(x)\alpha_{1}p_{1}(x)+\log p^{\pi}(x)\alpha_{2}p_{2}(x)\right) (54) \displaystyle=\frac{1}{\alpha_{1}+\alpha_{2}}\left(\alpha_{1}\EV_{p_{1}}\log p^{\pi}+\alpha_{2}\EV_{p_{2}}\log p^{\pi}\right) (55) By combining Eq. 51 and 55 , we obtain: \displaystyle\mathcal{F}(p^{\pi}) \displaystyle=\alpha_{1}\EV_{p_{1}}\log p_{1}+\alpha_{2}\EV_{p2}\log p_{2}-(\alpha_{1}+\alpha_{2})\EV_{\bar{p}}\log p^{\pi} (56) Therefore, \displaystyle\argmin_{\pi}\mathcal{F}(p^{\pi}) \displaystyle=\argmin_{\pi}\underbrace{\alpha_{1}\EV_{p_{1}}\log p_{1}+\alpha_{2}\EV_{p_{2}}\log p_{2}}_{\text{constant}}-(\alpha_{1}+\alpha_{2})\EV_{\bar{p}}\log p^{\pi} (57) \displaystyle=\argmin_{\pi}-(\alpha_{1}+\alpha_{2})\EV_{\bar{p}}\log p^{\pi} (58) \displaystyle=\argmin_{\pi}-(\alpha_{1}+\alpha_{2})\EV_{\bar{p}}\log p^{\pi}+\underbrace{(\alpha_{1}+\alpha_{2})\EV_{\bar{p}}\log\bar{p}}_{\text{constant}} (59) \displaystyle=\argmin_{\pi}(\alpha_{1}+\alpha_{2})D_{KL}(\bar{p}\|p^{\pi}) (60) Which concludes the proof. ∎ Appendix E R eward-Guided F low M erging ( RFM ) Implementation In the following, we provide an example of detailed implementations for RewardGuidedFineTuningSolver employed in Sec. 4 by Reward-Guided Flow Merging, as well as RewardGuidedFineTuningSolverRunningCosts , leveraged in Sec. 5 to scalably implement the AND operator. While the oracle implementation we report for completeness for RewardGuidedFineTuningSolver corresponds to classic Adjoint Matching (AM) ( 17 ) , the one for RewardGuidedFineTuningSolverRunningCosts trivially extends AM base implementation to account for the running cost terms introduced in Eq. 17 . E.1 Implementation of RewardGuidedFineTuningSolver Before detailing the implementations, we briefly fix notation. Both algorithms explicitly rely on the interpolant schedules \kappa_{t} and \omega_{t} from equation 1 . In the flow-model literature, these are more commonly denoted \alpha_{t} and \beta_{t} . We write u^{\textrm{pre}} for the velocity field induced by the pre-trained policy \pi^{\textrm{pre}} , and u^{\textrm{fine}} for the velocity field induced by the fine-tuned policy. In essence, each algorithm first draws trajectories and then uses them to approximate the solution of a surrogate ODE; its marginals serve as regression targets for the control policy ( 17 , Section 5) . Algorithm 2 RewardGuidedFineTuningSolverRunningCosts via AM 1: Pre-trained FM velocity field u^{\text{pre}} , step size h , number of fine-tuning iterations N , gradient of reward \nabla r , fine-tuning strength \eta_{k} 2: Initialize fine-tuned vector fields: u^{\text{finetune}}=u^{\text{pre}} with parameters \theta . 3: for n\in\{0,\ldots,N-1\} do 4: Sample m trajectories \bm{X}=(X_{t})_{t\in\{0,\ldots,1\}} with memoryless noise schedule: \sigma(t)=\sqrt{2\kappa_{t}\!\left(\tfrac{\dot{\omega}_{t}}{\omega_{t}}\kappa_{t}-\dot{\kappa}_{t}\right)} (62) 5: i.e.,: X_{t+h}=X_{t}+h\Big(2u_{\theta}^{\text{finetune}}(X_{t},t)-\tfrac{\dot{\omega}_{t}}{\omega_{t}}X_{t}\Big)+\sqrt{h}\,\sigma(t)\,\varepsilon_{t},\quad\varepsilon_{t}\sim\mathcal{N}(0,I),\quad X_{0}\sim\mathcal{N}(0,I). (51) 6: For each trajectory, solve the lean adjoint ODE backwards in time from t=1 to 0 , e.g.: \tilde{a}_{t-h}=\tilde{a}_{t}+h\,\tilde{a}_{t}^{\top}\nabla_{X_{t}}\Big(2v^{\text{base}}(X_{t},t)-\tfrac{\dot{\omega}_{t}}{\omega_{t}}X_{t}\Big),\quad\tilde{a}_{1}=\eta_{k}\nabla r(X_{1}). (52) 7: Note that X_{t} and \tilde{a}_{t} should be computed without gradients, i.e., \displaystyle X_{t} \displaystyle=\texttt{stopgrad}(X_{t}) (63) \displaystyle\tilde{a}_{t} \displaystyle=\texttt{stopgrad}(\tilde{a}_{t}) (64) 8: For each trajectory, compute the following Adjoint Matching objective: \mathcal{L}_{\text{Adj-Match}}(\theta)=\sum_{t\in\{0,\ldots,1-h\}}\left\|\tfrac{2}{\sigma(t)}\Big(v_{\theta}^{\text{finetune}}(X_{t},t)-u^{\text{base}}(X_{t},t)\Big)+\sigma(t)\,\tilde{a}_{t}\right\|^{2}. (53) 9: Compute the gradient \nabla_{\theta}\mathcal{L}(\theta) and update \theta using favorite gradient descent algorithm. 10: end for Output: Fine-tuned vector field v^{\text{finetune}} E.2 Implementation of RewardGuidedFineTuningSolverRunningCosts The following RewardGuidedFineTuningSolverRunningCosts is algorithmically identical to RewardGuidedFineTuningSolverRunningCosts , with the only difference that the lean adjoint computation now integrates a running-cost term f_{t} , defined as follows (see Sec. 5 ): f_{t}(x)\coloneqq\ \delta\left(\sum_{i=1}^{n}\alpha_{i}\,D_{KL}(p_{t}^{\pi}\,\|\,p_{t}^{pre,i})\right)(x,t),\quad t\in[0,1) (65) Algorithm 3 RewardGuidedFineTuningSolverRunningCosts via AM with running costs 1: Pre-trained FM velocity field v^{\text{base}} , step size h , number of fine-tuning iterations N , f_{t}=\nabla\delta\mathcal{G}_{t}(p^{\pi^{k}}_{t}) , weight \gamma_{k} , weight schedule \lambda 2: Initialize fine-tuned vector fields: v^{\text{finetune}}=v^{\text{base}} with parameters \theta . 3: for n\in\{0,\ldots,N-1\} do 4: Sample m trajectories \bm{X}=(X_{t})_{t\in\{0,\ldots,1\}} with memoryless noise schedule: \sigma(t)=\sqrt{2\kappa_{t}\!\left(\tfrac{\dot{\omega}_{t}}{\omega_{t}}\kappa_{t}-\dot{\kappa}_{t}\right)} (66) 5: i.e.,: X_{t+h}=X_{t}+h\Big(2v_{\theta}^{\text{finetune}}(X_{t},t)-\tfrac{\dot{\omega}_{t}}{\omega_{t}}X_{t}\Big)+\sqrt{h}\,\sigma(t)\,\varepsilon_{t},\quad\varepsilon_{t}\sim\mathcal{N}(0,I),\quad X_{0}\sim\mathcal{N}(0,I). (40) 6: For each trajectory, solve the lean adjoint ODE backwards in time from t=1 to 0 , e.g.: \displaystyle\tilde{a}_{t-h} \displaystyle=\tilde{a}_{t}+h\,\tilde{a}_{t}^{\top}\nabla_{X_{t}}\Big(2v^{\text{base}}(X_{t},t)-\tfrac{\dot{\omega}_{t}}{\omega_{t}}X_{t}\Big)-h\gamma_{k}\lambda_{t}f_{t}(X_{t}) (67) \displaystyle\tilde{a}_{1} \displaystyle=-\gamma_{k}\lambda_{1}\nabla_{X_{1}}\delta\mathcal{G}_{1}(p^{\pi^{k}}_{1})(X_{1}). (41) 7: Note that X_{t} and \tilde{a}_{t} should be computed without gradients, i.e., \displaystyle X_{t} \displaystyle=\texttt{stopgrad}(X_{t}) (68) \displaystyle\tilde{a}_{t} \displaystyle=\texttt{stopgrad}(\tilde{a}_{t}) (69) 8: For each trajectory, compute the Adjoint Matching objective: \mathcal{L}_{\text{Adj-Match}}(\theta)=\sum_{t\in\{0,\ldots,1-h\}}\left\|\tfrac{2}{\sigma(t)}\Big(v_{\theta}^{\text{finetune}}(X_{t},t)-v^{\text{base}}(X_{t},t)\Big)+\sigma(t)\,\tilde{a}_{t}\right\|^{2}. () 9: Compute the gradient \nabla_{\theta}\mathcal{L}(\theta) and update \theta using a gradient descent step 10: end for Output: Fine-tuned vector field u^{\text{finetune}} Appendix F R eward-Guided F low M erging ( RFM ): Computational Complexity, Cost, and Approximate Fine-Tuning Oracles Reward-Guided Flow Merging ( RFM , see Alg. 1) is a sequential fine-tuning scheme which, at each of the ( K ) outer iterations, calls a reward-guided fine-tuning oracle such as RewardGuidedFineTuningSolver (see Apx. 3 ). In practice, each oracle call performs ( N ) gradient steps of Adjoint Matching (see Apx. 3 ). At first sight, this suggests that the computational complexity of RFM scales linearly in K with respect to a standard fine-tuning run with ( N ) steps. However, this worst-case view does not fully capture the practical computational cost. We highlight two observations. Approximate fine-tuning oracle. First, RFM can operate reliably with a rather approximate fine-tuning oracle , i.e., with relatively small values of ( N ). We evaluate this phenomenon by replicating the objective curve of Fig. 2(d) with same parameters and setting, for three different configurations of (K,N) that keep the total budget ( K\cdot N=300 ) fixed but vary the outer (i.e., K ) and inner (i.e., N ) iteration counts: • K=10,;N=30 • K=15,;N=20 (as in Fig. 2(d) ) • K=30,;N=10 (a) K=10,N=30 (b) K=15,N=20 (c) K=30,N=10 Figure 6: (left) RFM run for reward-guided intersection with K=10,N=30 , (center) RFM run for reward-guided intersection with K=15,N=20 , (right) RFM run for reward-guided intersection with K=30,N=10 . The three corresponding curves are reported in Fig. 6 . Empirically, all three settings achieve nearly identical final objective values, indicating that a more approximate oracle (smaller (N)) can be compensated by increasing the number of outer RFM iterations ( K ), and vice versa, as long as the total optimization budget remains comparable. We observe a similar behaviour also on real-world, higher-dimensional, experiments (see Sec. 7 and Apx. 8 ), where we values of K vary from K=1 to K=37 . K/N Trade-off. Second, the runtimes of these configurations are of the same order. On our implementation, the runs with ((K,N)=(10,30),(15,20),(30,10)) require approximately 1615 s, 1643 s, and 1870 s, respectively, showing a very light increase depending on K . This further supports the view that practitioners can trade off a cheaper but less accurate inner oracle (small ( N )) against a slightly larger number of outer RFM steps (larger ( K )), and vice versa, without incurring prohibitive additional cost. Since RFM effectively solves a convex/non-convex optimization problem in probability space, we believe that classic convex optimization provides an interpretable framework for trading-off N and K , by interpreting N as the typical step-size, or learning rate, and K as the typical number of gradient steps. Clearly, higher learning rates typically require less gradient steps and vice versa. Ultimately, one should notice that increasing N does not directly imply better solution quality of the fine-tuning oracle, as it is the case for the oracle we employ within Sec. 7 (i.e., Adjoint Matching ( 17 ) ), for which performance can degrade for excessively high values of N . Appendix G Experimental Details G.1 Illustrative Examples Experimental Details Numerical values in all plots shown within Sec. 7 are means computed over diverse runs of RFM via 5 different seeds. Error bars correspond to 95\% Confidence Intervals. Shared experimental setup. For all illustrative experiments we utilize Adjoint Matching (AM) [14 ] for the entropy-regularized fine-tuning solver in Algorithm 1. Moreover, the stochastic gradient steps within the AM scheme are performed via an Adam optimizer. Intersection Operator. The balanced plot (see Fig. 2(b) is obtained by running RFM with \alpha=[0.1,0.1] , for K=80 iterations, \gamma_{k}=28 , and \lambda_{t}=0.2 for t>1-0.05 , and \lambda_{t}=0.4 otherwise. For the balanced, reward-guided case in Fig. 2(c) , we consider a reward function that is maximized by increasing the x_{2} coordinate. We run RFM with \alpha=[0.1,0.1] , for K=15 iterations, \gamma_{k}=1.2 , and \lambda_{t}=0.2 for t>1-0.05 , and \lambda_{t}=0.4 otherwise. Union Operator. In both cases, we learn a critic via standard f-GAN ( 45 ) with 300 gradient steps at each iteration k\in[K] and continually fine-tune the same critic over subsequent iterations. For critic learning, we use a learning rate of 5\exp(-5) . For the balanced case, in Fig. 2(f) , we run RFM with \alpha=[1.0,1.0] . We use K=13 iterations, \gamma_{k}=0.001 . For the unbalanced case in Fig. 2(g) , we run RFM with \alpha=[0.2,1.8] . Notice that up to normalization this is equivalent to [0.1,0.9] as reported in Fig. 2(g) for the sake of interpretability. We use K=13 iterations, \gamma_{k}=0.001 . Interpolation Operator. In both cases, we learn a critic via standard f-GAN ( 45 ) with 800 gradient steps at each iteration k\in[K] and continually fine-tune the same critic over subsequent iterations. For critic learning, we use a learning rate of 1\exp(-5) , and gradient penalty of 10.0 to enforce 1 -Lip. of the learned critic. For the case where \pi^{init}\coloneqq\pi^{pre,1} (i.e., left pre-trained model), in Fig. 2(j) , we run RFM with \alpha=[1.0,1.0] . We use K=6 iterations, \gamma_{k}=1.0 . For the case where \pi^{init}\coloneqq\pi^{pre,2} (i.e., right pre-trained model), in Fig. 2(k) , we run RFM with \alpha=[1.0,1.0] . We use K=6 iterations, \gamma_{k}=1.0 . Complex Logic Expressions via Generative Circuits. Pre-trained flows \pi_{1} and \pi_{2} , as well as \pi_{1} and \pi_{2} are intersected via RFM with \gamma_{k}=1 , for K=20 , and \lambda_{t}=0.1 . The union operator is implemented with K=30 , \gamma_{k}=0.0009 , 300 critic steps and learning rate 5\exp(-5) . G.2 Molecular Design Case Study Our base model FlowMol2 CTMC (i.e., PRE-1) ( 19 ) is pretrained on the GEOM-Drugs dataset ( 4 ) . We obtain our second model (i.e., PRE-2) by finetuning PRE-1 with AM ( 17 ) to generate poses with lower single point total energy wrt. the continuous atomic positions as calculated with dxtb at the GFN1-xTB level of theory 22 . We then run RFM with K=50 , \gamma=0.001 for the balanced flow merging, and K=20 , \gamma=0.005 to obtain the unbalanced flow merging. For reward-guided flow merging (RFM-RG), we set \gamma=0.1 and obtain the best model after K=11 . All models start from PRE-1, i.e., \pi^{init}=\pi^{pre,1} . All results for merging pre-trained models on GEOM can be found in Table 1 . Running RFM-RG with \alpha=3 and \gamma=0.001 , we obtain a model after K=35 that keeps the validity of its base models while implementing the reward-guided intersection. We note that beyond validity, a critical step towards practical application will be to integrate molecular stability and synthesizability. Our RFM formulation straightforwardly supports these extensions in the reward functional, and we leave their implementation to future work. Mean total energy Mean validity Model [Ha] [%] PRE-1 -8.09\pm 0.31 76.44\pm 1.7 RFM-B -10.95\pm 0.28 74.34\pm 0.9 RFM-RG -12.85\pm 0.16 74.02\pm 1.18 RFM-UB -13.69\pm 0.28 72.78\pm 0.4 PRE-2 -14.76\pm 0.29 68.04\pm 0.8 Table 1: Mean total energy and validity with standard deviation, averaged over 5 different seeds. Suffixes: B - balanced ; UB - unbalanced; RG - reward-guided flow merging For our second case-study - the OR operator - we use FlowMol2 CTMC pre-trained on QM9 ( 49 ) . We limit dimensionality to reduce the problem complexity by sampling 10 atoms per molecule, and run RFM with \gamma=100,K=37 . In particular Figure 7 shows that the estimated mean of the model \pi^{*} obtained via RFM matches the average total energy of \pi^{pre,1} and \pi^{pre,2} as predicted by the closed-form solution for the union operator presented in Sec. 3 . In Fig. 7 , OR denotes the final policy \pi^{*} returned by RFM . Figure 7: Union on QM9 G.3 Conformer Generation Case Study We finetune the GEOM-QM9 pre-trained ETFlow model (denoted PRE-1) with AM on the molecular system C#C[C@H](C=O)CCC to obtain PRE-2, using the same total energy objective as in the molecular design case study. This is also the molecular system we perform our evaluations on. For the subsequent merging experiments, we choose the lower-energy PRE-2 as the base model, i.e., \pi^{init}=\pi^{pre,2} . Balanced merging is performed with \alpha_{1}=\alpha_{2}=1 , \gamma=0.025 and K=6 . The unbalanced merging is run with \alpha_{1}=0.7 and \alpha_{2}=0.3 and we take the model after K=8 steps with \gamma=5e-5 . The reward-guided merging model was obtained with \gamma=0.025 after K=6 , and the union model after K=1 with \gamma=1e-3 and critics with the same GNN backbone as ETFlow. We show all results for the conformer generation case study in Tab. 2 E \mu \Delta\epsilon E_{min} Model [kcal/mol] [debye] [kcal/mol] [kcal/mol] PRE-1 0.3385\pm 0.0002 0.1679\pm 0.0002 0.5373\pm 0.0019 0.2793 RFM-UB 0.3412 0.1512 0.5173 0.2778 RFM-B 0.3356\pm 0.0001 0.1503\pm 0.0002 0.4915\pm 0.0014 0.2782 RFM-UNION 0.3352 0.1467 0.5033 0.2761 RFM-RG 0.3193\pm 0.0003 0.1141\pm 0.0002 0.4849\pm 0.0015 0.2777\pm 0.0008 PRE-2 0.3175\pm 0.0006 0.1268\pm 0.0006 0.4819\pm 0.0010 0.2761\pm 0.0027 Table 2: Median Absolute Errors for energy E , dipole moment \mu , HOMO-LUMO gap \Delta\epsilon , and minimum energy E_{min} across different models. We report mean and standard deviation over 5 different seeds. Appendix H Beyond Molecules: Reward-Guided Flow Merging of Pre-Trained Image Models We further showcase the capabilities of Reward-Guided Flow Merging on a small-scale, yet informative experiment for image generation. In the following, we consider pretrained CIFAR-10 image models ( 32 ) and use the LAION aesthetics predictor V1 ( 52 ) as a reward model. Specifically, the aesthetics predictor was trained on a subset of the SAC dataset ( 48 ) with available ratings from 1 (low preference / aesthetics) to 10 (high preference). The goal of this case study is to show that RFM can merge two models, PRE-1 and PRE-2, while optimizing the aesthetics score. We perform reward-guided flow merging with PRE-2 as the base model, obtaining the model RFM-RG after K=11 iterations with \gamma=1 and \alpha_{i}=1 . The numerical results in Tab. 3 show that RFM can successfully intersect multiple prior flow image models while maximizing the aesthetic score. In particular, the fine-tuned model achieves a score of 3.64\pm 0.53 against 3.16\pm 0.66 and 3.23\pm 0.58 of PRE-1 and PRE-2 respectively. We also report sample images of the discussed models in Fig. 8 . Model Mean aesthetic score PRE-1 3.16\pm 0.66 PRE-2 3.23\pm 0.58 RFM-RG 3.64\pm 0.53 Table 3: RFM can perform reward-guided (RG) intersections of pre-trained CIFAR-10 image models ( 32 ) . We evaluate the resulting models in terms of mean aesthetic score (i.e., the reward) over 1000 samples, and report one std. (a) PRE-1 (b) PRE-2 (c) RFM-RG Figure 8: Images generated by the two pre-trained flow models (i.e., PRE-1, PRE-2), and by the flow model obtained via reward-guided intersection (i.e., RFM-RG).