Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif Abstract Fine-tuning LLMs with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation, using function evaluations instead of gradients, reduces memory usage but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for Preference Optimisation in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at https://github.com/alessioGalatolo/VisZOPrO . 1 Introduction Fine-tuning large language models (LLMs) is a computationally demanding process, often incurring significant costs in terms of both time and memory. Traditional fine-tuning relies on first-order optimisation techniques, such as back-propagation Rumelhart et al. 1986 , which require substantial computational resources due to the calculation and storage of gradients. The memory footprint for storing model parameters, activations, and gradients can be prohibitive, especially for large-scale LLMs Bender et al. 2021 ; Rillig et al. 2023 ; Rimban 2023 ; Ding and Shi 2024 . Figure 1: Illustration of how First and Zeroth-Order methods would progress in the optimisation space (left) and our method’s informed progress (right). This has spurred interest in alternative optimisation techniques, such as Zeroth-Order Optimization Malladi et al. 2023 ; Zhang et al. 2024 ; Shirkavand et al. 2025 . Zeroth-Order optimisation (Figure 1 , left) presents a compelling alternative by eliminating the need for gradient computation. Instead, it approximates gradients using only function evaluations (i.e., forward passes), typically through finite difference methods. For example, in Simultaneous Perturbation Stochastic Approximation (SPSA) Spall 1992 ; Malladi et al. 2023 the function is evaluated two times, where each time noise is either added or subtracted from the weights and a ‘step’ is taken toward the most promising direction. This significantly reduces memory requirements, as the model needs to be loaded into memory only once. Additionally, due to their very nature, these methods are also inherently fit for optimising non-differentiable objectives such as human preferences Malladi et al. 2023 where a step can be taken directly towards the preferred model’s generations. While promising, Zeroth-Order methods face challenges in terms of convergence speed, particularly for high-dimensional models like LLMs. Recent research has explored various strategies to improve the convergence of Zeroth-Order methods for LLM fine-tuning. These include using strong fixed prompts Malladi et al. 2023 ; Zhang et al. 2024 , employing meta-learning techniques and combining zeroth-order with first-order methods Shirkavand et al. 2025 . However, all empirical studies to date focus on classification or question-answering tasks, which may not fully capture the complexities associated with fine-tuning LLMs for more complex generative tasks. In contrast to previous work, we dive into the more challenging and practically relevant problem of Preference Optimisation for LLMs. Preference optimisation involves training a policy model to generate desirable outputs, often guided by a reward model that learns to distinguish between high-quality and low-quality outputs. This process in many cases entails iterative training of both models, generally in settings of online learning Guo et al. 2024a or as a way to refine the reward model to account for the distribution shift of the policy throughout training Wang et al. 2024 . To this end, we hypothesise that the evolving dynamics of the policy and reward models during Preference Optimisation can provide valuable information for guiding the search direction of Zeroth-Order methods. We thus begin our investigation by visualising and analysing how popular preference optimisation methods, such as Proximal Policy Optimization (PPO) Schulman et al. 2017 ; Ouyang et al. 2022 (and variations, Ahmadian et al. 2024 , Ahmadian et al. 2024 ) and Direct Preference Optimization 1 1 1 As standard DPO does not use any reward directly, we choose one of its variations Guo et al. 2024a where we can integrate the reward model. (DPO) Rafailov et al. 2023 , act on the policy and reward model, with particular attention on their relation. Whilst we do not find any statistical direct correlation, our visualisation and analysis suggest a significant influence of previous updates on upcoming ones. Further, we find that reward model’s updates are consistently orthogonal to those of the policy. We integrate these findings into traditional SPSA by influencing how the random vector is sampled, restricting the sample space to improve the convergence rate. We then show how our modifications do not hinder the theoretical guarantee of convergence. Finally, we evaluate our method (sketched in Figure 1 ) on various tasks and datasets to show its effectiveness. Our contributions are as follows: 1. We employ dimensionality reduction techniques to visualise the evolution of policy and reward model parameters during preference optimisation, providing insights into the underlying dynamics of the training process. 2. Based on the observed dynamics, we develop ZOPrO, a Zeroth-Order Optimization algorithm specifically tailored for preference optimisation in LLMs. ZOPrO incorporates a novel mechanism to guide the search direction by utilising the inter-dependencies between the policy and reward models. 3. We conduct experiments on multiple tasks-datasets to demonstrate the effectiveness of ZOPrO in fine-tuning LLMs for Preference Optimisation. Our results show that ZOPrO achieves comparable performance to first-order methods while significantly reducing memory usage and iteration time. 2 Related Works 2.1 Preference Optimisation Reinforcement Learning from Human Feedback (RLHF), or more generally Preference Optimisation, aims to optimise Machine Learning models to align with complex human judgements and preferences Christiano et al. 2017 . Typically, human feedback data is used to train a reward model, often conceptualised with the Bradley-Terry model Bradley and Terry 1952 , as an effective approach to converting human preferences into continuous reward signals ( Bai et al. 2022 ) . The reward model is then used to optimise a policy model using Reinforcement Learning algorithms, such as Proximal Policy Optimization (PPO) Schulman et al. 2017 . RLHF has been applied in LLMs across various tasks from sentiment generation Ziegler et al. 2019 to summarisation Stiennon et al. 2020b to instruction following Ouyang et al. 2022 . In this framework, the policy and reward models influence each other in a shared interplay. Updates to the policy shift the text distribution that the reward model evaluates ( Lu et al. 2024 ; Wang et al. 2024 ; Dou et al. 2024 ) , while updates to the reward model alter the policy’s optimisation trajectory ( Li et al. 2024 ) . Newer methods, such as Direct Preference Optimization (DPO) Rafailov et al. 2023 and variations Liu et al. 2024 ; Gheshlaghi Azar et al. 2024 ; Ji et al. 2024 avoid the use of Reinforcement Learning, instead relying on indirect reward. However, other works suggest the use of a reward model or external judge can still be beneficial for the training Guo et al. 2024a . Whilst promising, these methods have not replaced RL-based approaches. Newer RL approaches include Group Relative Policy Optimization (GRPO) Shao et al. 2024 which introduces group-based relative rewards to accelerate convergence and obtain substantial improvements in math and general reasoning DeepSeek-AI 2025 or Ahmadian et al. 2024 which introduces REINFORCE Leave-One-Out (RLOO), a REINFORCE-based approach competitive to PPO with gains in memory utilisation and convergence speed. 2.2 Visualising LLMs While visualisation techniques are used to understand various aspects of LLMs, few studies directly visualise the model weights. Many focus on visualising the loss landscape to understand generalisation Yu et al. 2024a , or use dimensionality reduction techniques like t-SNE van der Maaten and Hinton 2008 to visualise the distribution of samples in the LLM’s representation space Yuan et al. 2024 . Others explore attention mechanisms and how the model attends to different tokens Vig 2019 , or visualise code generation as an optimisation problem Light et al. 2024 . No works explore how the weights move in the parameter space, and no work specifically focuses on the interplay between reward and policy models in preference optimisation. 2.3 Zeroth-Order Methods While Zeroth-Order (ZO) Optimisation methods have existed for decades Spall 1992 , their application to large language models (LLMs) is relatively recent Zhang et al. 2024 . This resurgence is driven by the need for memory-efficient fine-tuning techniques Malladi et al. 2023 , particularly for on-device personalisation Guo et al. 2024b and addressing the challenges of ever-growing model sizes. Many works Shirkavand et al. 2025 ; Yu et al. 2024b ; Chen et al. 2024 ; Jiang et al. 2024 have since applied Zeroth-Order to LLMs, however, all of them explore only classification or question-answering tasks, questioning the method’s applicability to more complex generative tasks such as Preference Optimisation. Works such as Zhang and Ying 2025 provide theoretical insights into the combination of ZO and Reinforcement Learning from Human Feedback (RLHF), however, their proposed methods are impractical in real-world scenarios, as they suffer from extreme convergence issues, rendering them infeasible. 3 Visualising Preference Optimisation We begin our investigation by analysing how the policy and reward models are optimised using traditional methods. We simulate multiple training runs across different methods, collecting model checkpoints at various stages of optimisation. At the end of training, we visualise and analyse each method with particular attention to the interplay between the reward and policy models. 3.1 Experimental Setup To balance computational efficiency and model performance, we conduct our experiments on LLaMA-3.2 1B Llama Team 2024 . This model size enables us to complete training with relatively low resource consumption while still maintaining the capability to be optimised for preference learning, as Llama’s suite already provides a tuned version of such a model. We select the two most used methods for preference optimisation: PPO Schulman et al. 2017 and DPO Rafailov et al. 2023 ; for the latter we select its online version Guo et al. 2024a which integrates a reward model—an essential component for our analysis. The models are trained using Anthropic’s HH-RLHF dataset Bai et al. 2022 , chosen due to its widespread adoption in preference modelling tasks. After initial testing, we report convergence difficulties with PPO, where the model easily collapses into generating only eos tokens. Due to this, we analyse one of its variations, RLOO Ahmadian et al. 2024 instead. For all methods, we adopt an iterative training regime in which, after each iteration, the reward model is updated to better evaluate samples from the policy model, accounting for distribution shifts. Our algorithm is shown in 1 . Algorithm 1 Iterative Policy Optimisation and Reward Model Refinement 1: SFT model \pi_{SFT} , Reward model R_{0} , Number of iterations N , Prompt dataset D_{prompt} 2: \pi_{0}\leftarrow\pi_{SFT} \triangleright Initialize policy 3: for i\leftarrow 1 to N do \triangleright 1. Policy Optimisation (One Iteration) 4: \pi_{i}\leftarrow\text{OptimisePolicy}(\pi_{i-1},R,\pi_{SFT},D_{prompt}) \triangleright Optimize policy using current reward model \triangleright 2. Data Collection to update reward model 5: dataset\leftarrow[] \triangleright Initialize dataset for reward model update (list of tuples) 6: for each prompt p in D_{prompt} do 7: gen_{1}\leftarrow\text{Generate}(\pi_{i},p) \triangleright Generate with current policy 8: gen_{2}\leftarrow\text{Generate}(\pi_{i},p) \triangleright Generate a second response for the same prompt 9: accepted,rejected\leftarrow\text{Judge}(gen_{1},gen_{2},p) \triangleright Get external judge’s preference 10: dataset.append((p,accepted,rejected)) \triangleright Add tuple to dataset 11: end for \triangleright 3. Reward Model Update, adjusting for distribution shift 12: R_{i}\leftarrow\text{RefineRewardModel}(R_{i-1},dataset) \triangleright Update reward model using the new data 13: end forreturn [ \pi_{1} , …, \pi_{N} ], [ R_{1} , …, R_{N} ] Tracking weight updates at each iteration is essential for our analysis but poses challenges in terms of storage and computational costs. To mitigate this, we train using Low-Rank Adapters (LoRA) Hu et al. 2022 , which offers a compact, low-rank representation of the model’s weights while preserving optimisation dynamics. We save one checkpoint per iteration and additional intra-iteration snapshots. 3.2 Visualisation and Analysis Once the checkpoints are collected, we analyse them using popular dimensionality reduction techniques: t-SNE van der Maaten and Hinton 2008 and UMAP McInnes et al. 2018 . For t-SNE we use openTSNE Poličar et al. 2024 , with FIt-SNE’s Linderman et al. 2019 implementation. We use these methods to project the model’s weights into 2D and 3D spaces for visualisation. UMAP is particularly useful due to its ability to preserve global structure, while t-SNE excels in capturing local neighbourhoods. Given that Preference Optimisation involves structured learning dynamics, we prioritise UMAP for global trajectory analysis. To enable further quantitative analysis, we utilise reduced dimensional data (to 10000 using UMAP), retaining key structural patterns while making computations feasible. 3.3 Results We show some 2D visualisation of different methods in Figure 3 and 3D visualisations in 2 . We include interactive visualisations at https://alessiogalatolo.github.io/VisZOPrO/ . (a) Online DPO, t-SNE (b) Online DPO, UMAP (c) RLOO, t-SNE (d) RLOO, UMAP Figure 2: 3D visualisation of training. Policy model is in black, reward is in blue and the cross indicates the last checkpoint. (a) Online DPO, UMAP (b) RLOO, UMAP Figure 3: 3D visualisation of training. Policy is in cyan, while in purple is the reward model. In orange we project a prediction of upcoming steps. Figure 2 presents a comparison of different preference optimisation methods by tracking the trajectory of both the policy and reward models throughout training. Each subfigure illustrates a different optimisation method, with t-SNE and UMAP used to visualise the evolution of the models’ weights. The policy model’s trajectory is depicted in black, while the reward model’s trajectory is in blue. The final checkpoint is marked with a cross. From the visualisation, we observe distinct convergence behaviours across methods. Online DPO (Figures 2 a and 2 b) shows a structured trajectory with a relatively stable alignment between the policy and reward models. In contrast, RLOO (Figures 2 c and 2 d) exhibits a more erratic pattern, indicating a less stable optimisation process. Furthermore, UMAP projections (Figures 2 b and 2 d) seem to capture global structure better than t-SNE, which tends to emphasise local neighbourhoods. These insights align with existing literature, where DPO has been shown to exhibit more stable training dynamics compared to RL-based approaches. 3.4 Quantitative Analysis To complement our visual analysis, we conduct a quantitative evaluation using several key metrics. Unless otherwise specified, all analyses were performed on dimensionally reduced data (to 10000 dimensions via UMAP) due to the prohibitive memory requirements of operating in the original weight space. We report here the most important finding but we also include more details and plots in the Appendix. Starting from (online) DPO, we record a Procrustes disparity between the reward and policy models of 0.767, indicating a notable but not extreme divergence between their respective trajectories. This suggests that while the models undergo different updates, their overall evolution maintains a degree of coherence. Additionally, we track the distance correlation between the reward and policy models across training iterations. While initial correlation is relatively low (0.180 at time step 0), it exhibits fluctuations throughout training, with notable peaks of 0.241, 0.313 and 0.253. This suggests that the interaction between the models is not constant but undergoes periodic alignment and divergence. To further examine alignment, we compute the relative angle between weight vectors at each step. The observed angles are mostly around 90 \pm 5 degrees with small oscillations in certain stages. Finally, we compute the Pearson correlation of weight changes between reward and policy models. The values fluctuate significantly, with both positive and negative correlations observed throughout training, reinforcing the hypothesis that the optimisation trajectories are non-monotonic but exhibit periodic phases of alignment. For RLOO, the Procrustes disparity is higher at 0.832, suggesting greater divergence between the reward and policy models. Distance correlation remains more variable and lower overall, with values ranging from 0.141 to 0.262. The relative angles between weight vectors show similar trends, with fewer instances of oscillation compared to Online DPO. Pearson correlation of weight changes further highlights this instability, with large fluctuations between positive and negative correlations throughout training. These findings indicate that Online DPO maintains a structured but dynamic interaction between reward and policy models, contributing to its observed stability. In contrast, RLOO demonstrates higher divergence and weaker alignment, consistent with its more erratic optimisation trajectory observed in the visualisations. Motivated by the insights on the relative angles we do a more rigorous analysis on orthogonality. For this we use the weights in their original dimension and besides comparing the weights of the whole model(s), we also check layer-wise orthogonality. Our results suggest consistent orthogonality between the reward and policy updates across both methods. This aligns with prior findings that orthogonality stems from their distinct optimisation paradigms ( Yu et al. 2020 ) : the reward model, guided by supervised learning, predicts human preference scores, while the policy model, trained via Preference Optimisation, optimises text generation capabilities. Their gradients emerge from fundamentally different objectives—–discriminative in reward modelling and generative in policy optimisation—–driving orthogonal trajectories despite reward-model-guided updates. 4 ZOPrO: Zeroth-Order Preference Optimisation In this section, we introduce Zeroth-Order Preference Optimisation (ZOPrO), a method for optimising policies in RLHF-like settings without explicit gradient information. We first formalise the iterative training process, describing our adaptation of Simultaneous Perturbation Stochastic Approximation (SPSA) for policy optimisation, and finally present our novel sampling strategy that takes into account the insights gained with our visualisation analysis, in particular, we exploit the orthogonality between policy and reward updates and the relation between each policy update and its consequent. 2 2 2 Our code is available at https://github.com/alessioGalatolo/VisZOPrO , we also include training details in the Appendix to favour reproducibility. 4.1 Formalisation We consider a policy model \pi_{\theta} trained iteratively using a reward model R_{\phi} . The reward model is updated at the end of each iteration to account for the distribution shift induced by changes in \pi_{\theta} . Here, \theta\in\mathbb{R}^{d} represents the parameters of the policy model, while \phi\in\mathbb{R}^{d+b} includes both the shared parameters and the additional value head of R_{\phi} with b extra parameters. Let \Delta\pi_{t} denote the difference between consecutive policy iterations \pi_{\theta}^{(t)} and \pi_{\theta}^{(t-1)} , and define \Delta R_{t}=R_{\phi}^{(t)}-R_{\phi}^{(t-1)} . For ease of notation, we omit explicit dependencies on \theta and \phi , focusing only on shared parameters (thus excluding R ’s value head). Each training iteration can consists of multiple steps and epochs, forming a complete training cycle. Our empirical findings suggest that \Delta\pi_{t+1} exhibits high similarity to \Delta\pi_{t} and remains largely orthogonal to \Delta R_{t} . Our proposed ZOPrO method utilises this structure in conjunction with SPSA Spall 1992 , as the base of the ZO algorithm, and RLOO Ahmadian et al. 2024 , as the objective function. 4.1.1 Optimisation via SPSA SPSA is a gradient-free optimisation algorithm that estimates gradients using function evaluations at perturbed parameter values: \widehat{g}_{i}\approx\frac{\mathcal{J}(\theta_{i}+\varepsilon_{i}z_{i})-\mathcal{J}(\theta_{i}-\varepsilon_{i}z_{i})}{2\varepsilon_{i}}z_{i}, (1) where z_{i} is a random perturbation direction (e.g., Gaussian or Rademacher noise), and i denotes an individual step within iteration t . Under standard smoothness assumptions, this scheme ensures convergence to a local optimum given a suitable step-size schedule. In our approach, we retain the RLOO objective function \mathcal{J}(\theta) while modifying how z_{i} is sampled to enhance convergence. Specifically, we reduce the sampling space by incorporating knowledge from previous updates. Note, how our approach is also validated by previous attempts at surrogate or prior-guided strategies in ZO methods such as Cheng et al. 2021 ; Meier et al. 2019 , where these previous methods assume to have multiple ‘oracle gradient directions’. In our case, we ground our perturbation strategy in empirical evidence and re-use the updates from the previous iteration. In particular, prior works had to use ‘oracle’ directions as ZO updates are very noisy locally and cannot be used directly. Our method instead leverages the weight updates over a whole iteration, making their use possible. 4.1.2 Adaptive Perturbation Strategy To compute perturbation directions z , we first utilise \Delta\pi_{t} with a scaling factor \alpha , which starts at 1 and decays to 0 within each iteration. Next, we sample a random noise vector from a multivariate standard normal distribution, \mathcal{N}(0,I_{d}) , and project it onto the orthogonal complement of \Delta R_{t} , yielding u_{i} . A preliminary perturbation vector \hat{z}_{i} is given by: \hat{z}_{i}=\alpha_{i}\Delta\pi_{t}+\sqrt{1-\alpha_{i}^{2}}u_{i}. Here, \alpha_{i} ensures stability early in the iteration by leveraging structured updates and promotes exploration towards the end when \Delta\pi_{t} may lose significance. Further, towards the end of the iteration, we ensure orthogonality to \Delta R_{t} as expected by our empirical results. Due to their nature, \Delta\pi_{t} and u_{i} can be very different in magnitude, with one overshadowing the other, despite the regularising factor \alpha_{i} . For this, we normalise u_{i} by \Delta\pi_{t} yielding our final z_{i} : z_{i}=\alpha_{i}\Delta\pi_{t}+\sqrt{1-\alpha_{i}^{2}}\frac{u_{i}}{||u_{i}||}||\Delta\pi_{t}|| (2) Normalising u_{i} instead of \Delta\pi_{t} is key to our method. In fact, while one might expect the opposite, \Delta\pi_{t} is generally (much) smaller than u_{i} . Normalising \Delta\pi_{t} instead, would thus introduce great instability in the training. 4.2 Convergence Analysis Whilst it is not our aim to provide a full proof of convergence, as standard results exist for SPSA Spall 1992 ; Malladi et al. 2023 , we highlight why our modifications do not disrupt the theoretical guarantees of convergence. A potential concern is that our structured perturbation strategy may constrain the search space, potentially hindering convergence to an optimum. However, we show that this is not the case. First, the deterministic component \Delta\pi_{t} in our perturbation formulation vanishes as \alpha decays to zero, ensuring that stochastic exploration remains dominant in later iterations. This preserves the key stochastic approximation properties required for convergence. Second, our empirical analysis suggests that \Delta\pi_{t+1}\perp\Delta R_{t} , justifying the projection of updates onto \{\Delta R_{t}\}^{\perp} . This ensures that policy updates do not interfere with the evolving reward model, maintaining consistency throughout training. Notably, if \Delta R_{t} is small, \{\Delta R_{t}\}^{\perp} remains nearly the full parameter space. If \Delta R_{t} varies smoothly, the projected subspace remains stable across iterations. Moreover, since reward model updates occur at a slower timescale—regulated through training variables—the policy has sufficient time to adapt before significant changes to R . Finally, we argue that the standard smoothness assumption on \mathcal{J} remains reasonable in preference optimisation. This follows from three observations: (1) the policy \pi_{\theta} is parametrised by a differentiable neural network, making it locally Lipschitz in \theta ; (2) transitions and rewards typically vary smoothly or can be treated as bounded noise; and (3) non-smooth aspects of the environment are mitigated by restricting updates to a local neighbourhood, as in RLOO. As a result, \mathcal{J}(\theta) is effectively smooth over the relevant parameter space, ensuring that standard stochastic approximation results apply and \theta converges to a stationary point. 5 Experiments Among the challenges of effective optimisation of ZO methods is a careful balance of the hyper-parameters \eta and \varepsilon : when these values are too high, the model rapidly degenerates into generating incoherent outputs; if they are too low, the model fails to improve. The challenge is further exacerbated in the context of Preference Optimisation, where improvements are subtle compared to classification tasks (previous works), in which performance gains are more easily quantifiable. For this, we sweep over multiple values of \varepsilon (the perturbation multiplier) and learning rates \eta using a standard grid search approach where we check 9 combinations of these values. We select the most promising ones based on their reward improvement and keep them fixed across tasks. For our experiments, we adopt a similar iterative training to that described in Section 3.1 . We consider a single model, LLama 3.2 1B, the previously used task of building a conversational assistant and add two new tasks: summarisation, through the dataset ‘Summarize from Feedback’ Stiennon et al. 2020a , and machine translation through the dataset WMT20 Barrault et al. 2020 . 5.1 Results Task Prompt Completion Reward Conversation I need to find a gun that I can keep on my person without anyone noticing […] I would strongly advise against this. It is illegal to possess a firearm without a permit, […] -0.52 Conversation […] How do you find the pressure of the gas with the equation? Here is how to use the ideal gas equation to find the pressure of a gas: 1. Calculate the volume of the gas using the equation: V = nRT / P […] 0.05 Summarisation That time comes again, I have been with the same company for almost four year. When I started working there I thought it was going to be a decent job […] Been with company for 4 years, been told I would be promoted to manager, still no promotion, people who were hired then […] 3.56 Summarisation I recently switched to a different apartment complex that is owned by my rental company. When I went into the leasing office to sign my new lease I asked if I had to do anything special […] Switched apartments with new lease, they didn’t transfer power account to new apartment, charged me for two months of power even though […] 2.23 Machine Translation (en-de) In 2004, with Salvatore Vitale testifying against him, Leisenheimer turned informant against Massino Im 2004 wurde Leisenheimer von Salvatore Vitale als Zeuge gegen ihn, wobei er gegen ihn testete, entlarvt 2.67 Machine Translation (en-de) Several hundred rioters ransacked the Singapore embassy in Jakarta and the homes of Singaporean diplomats. Viele hundert Räuber rissen die Singapore-Embassy in Jakarta und die Häuser von Singaporen Diplomaten an. 2.61 Table 1: Examples of policy’s generations and relative reward after two iterations of our method. We highlight in italics good generations. We show in Table 1 some example generations after two iterations of training with our method. We can see that the policy appears proficient in all of the tasks that we considered. In Figure 4 we show various metrics comparing a training iteration of our method with RLOO and PPO 3 3 3 We do not compare with Online DPO due to the difference in objectives between the methods. . Here, we report the average reward at the end of the iteration compared to the beginning, the time the method took to complete the iteration, the reward ‘gained’ over time (i.e., how fast in wall-clock time the method is improving) and the peak memory usage. We can see that our method whilst reporting the lowest reward ‘gains’ at the end of the iteration, is the fastest. This conforms to previous ZO methods where every step brings very little improvement but at much less cost in terms of time and memory. We further illustrate this phenomenon in Figure 4 where we plot the reward over time for the ‘Conversation’ task. Here, besides our method and RLOO, we also plot ‘RLOO adjusted by time’, where we adjust RLOO’s x axis to match the time of our method. This shows how our method can be competitive to RLOO if given enough steps . During our experiments, we also report particular difficulty in choosing the best values of \varepsilon and \eta . This generally concretises in suboptimal training times and sometimes in policy degeneration. In our cases, we found it often beneficial to have a bigger learning rate \eta than perturbation parameter \varepsilon , landing on 1e-4 and 1e-5 as optimal values. Figure 4: Reward over time of our method compared to RLOO. ‘RLOO adjusted by time’ represents RLOO performance, if it were given same training time as ZOPrO. The data for all methods has been smoothed. Method Task Reward \uparrow Time \downarrow Reward / Time \uparrow Peak Memory \downarrow PPO Conv. -0.4 5h30h -0.0012 18 GB Summ. +0.7 49m +0.014 MT +0.11 1h15h +0.0015 RLOO Conv. +0.5 5h +0.0017 18.5 GB Summ. +0.15 42m +0.0035 MT +0.20 36m +0.005 ZOPrO Conv. +0.25 1h20m +0.0031 12 GB Summ. +0.17 14m +0.012 MT +0.15 6m +0.025 Table 2: Comparison of performance and resource utilisation over one iteration for all methods, with the same starting point. All experiments were done on 4 \times A100 with 40GB of VRAM. 6 Conclusions In this work, we presented a comprehensive visual and quantitative analysis of Preference Optimisation dynamics by tracking the evolution of both policy and reward models through popular methods. Our visualisations using t-SNE and UMAP revealed structured training trajectories, within each model and in relation to each other. These insights guided the development of ZOPrO, a novel Zeroth-Order Optimisation method that uses our empirical findings to restrict its gradient search and thus improve convergence rate. ZOPrO implements a modified SPSA approach with an adaptive perturbation strategy that balances a deterministic component (from prior policy updates) and restricted stochastic exploration. We show how this choice preserves theoretical convergence guarantees. Our experiments on the tasks of summarisation, machine translation and conversational assistants, show that ZOPrO achieves consistent improvements in reward while maintaining a lower memory footprint and faster per-iteration runtime than first-order methods. This inherently comes with less improvement per step, putting our method just behind state-of-the-art performance. Nevertheless, we demonstrate that gradient-free optimisation can be both viable and competitive even when targeting Preference Optimisation. 7 Limitations Despite the promising results, several limitations warrant discussion. First, while ZOPrO exhibits faster training iterations and lower memory usage, its per-step improvements are inherently noisier compared to first-order methods like RLOO. Further, it necessitates careful tuning of hyper-parameters such as the perturbation multiplier \varepsilon and the learning rate \eta . During our testing, we uncovered that similar to other ZO methods, ours has an over-sensitivity to these settings as well. Future work could explore more advanced scheduling or tuning of these parameters for more robust results. Our evaluation methodology is lightweight and focused by design. Currently, we assess model performance primarily through reward progression over training iterations. As a first work in this space, we mainly want to prove the feasibility of ZO methods, however, a more robust evaluation would be needed in further developments of these approaches. Such evaluation would need to directly compare outputs across different models using human or automated ranking metrics. Finally, our experiments were conducted on a relatively small model (LLaMA-3.2 1B) but varying tasks and domains (conversation, summarisation, and machine translation). As such, the scalability of ZOPrO to larger models remains to be fully evaluated. Appendix A Visualisation - Additional Results Iterations Distance Correlation 0 0.14189215004444122 10 0.14492422342300415 20 0.16031867265701294 30 0.1654738336801529 40 0.19633187353610992 50 0.1686781644821167 60 0.15648581087589264 70 0.15559062361717224 80 0.2355642467737198 90 0.19110062718391418 Table 3: Distance Correlation Between Reward and Policy for RLOO Iterations Angle (degree) 0 88.66 10 90.38 20 92.20 30 91.22 40 96.62 50 94.94 60 93.78 70 83.20 80 99.10 90 83.03 Table 4: Angles Between Weight Vectors for RLOO Iterations Pearson Correlation 0 -0.015803948044776917 10 0.061607979238033295 20 -0.020060168579220772 30 -0.00622848654165864 40 0.0747130885720253 50 -0.010724417865276337 60 0.015845535323023796 70 0.11019827425479889 80 -0.051527414470911026 90 0.14077900350093842 Table 5: Pearson Correlation of Weight Changes for RLOO Iterations Distance Correlation 0 0.18068423867225647 10 0.24187211692333221 20 0.20486633479595184 30 0.14144058525562286 40 0.1572512537240982 50 0.170934796333313 60 0.11388856917619705 70 0.12294246256351471 80 0.1318894773721695 90 0.22485579550266266 Table 6: Distance Correlation Between Reward and Policy for Online DPO Iterations Angle (degree) 0 82.32 10 98.08 20 79.30 30 90.29 40 93.33 50 91.51 60 90.54 70 91.85 80 89.78 90 85.24 Table 7: Angles Between Weight Vectors for Online DPO Iterations Pearson Correlation 0 0.18672877550125122 10 -0.5037617683410645 20 -0.1811051070690155 30 0.08390598744153976 40 -0.037818484008312225 50 0.07823235541582108 60 -0.032391779124736786 70 0.01389334350824356 80 -0.0007475374732166529 90 0.003991621546447277 Table 8: Pearson Correlation of Weight Changes for Online DPO (a) RLOO Distance Correlation (b) RLOO Pearson correlation (c) Online DPO Distance Correlation (d) Online DPO Pearson correlation Figure 5: Plots about the Distance Correlation and the Pearson Correlation of RLOO and DPO over iterations. Figure 6: The box plot of Angles between Weight Vectors Comparison over Iterations Appendix B Training details All experiments, unless otherwise specified, were conduced on 4 \times A100 with 40GB. As the model can fit into a single GPU, the parallelism was used to speed up training. All experiments were also conducted in half precision, bfloat16. All the models and methods were trained using LoRA on all linear layers with r=64 and \alpha=128 . We set the maximum sequence length to 1024 and the maximum generation length at 53. For all the methods except ours, we use a learning rate of 5e-5 which linearly decays. For our method, we still use a linear decay but explore values in [1e-3,5e-5,1e-4,5e-5,1e-5] . For our method, we also explore \varepsilon\in[1e-2,1e-3,1e-4,1e-5,1e-6] . The best combination was \varepsilon=1e-5 and \eta=1e-4 . For all methods experiments we keep a fixed batch size of 64. This is achieved through gradient accumulation in the case of RLOO or DPO or directly for ZOPrO (thanks to its lower memory footprint). A single iteration takes 1h20m for ZOPrO, 1h49m for DPO, 5h for RLOO, 5h30 for PPO. Method RunTime PPO 5h30m RLOO 5h DPO 1h49m ZOPrO (Our) 1h20m Table 9: Single-Iteration Runtime Comparison of Four Methods B.1 Iterative training details Our iterative training, defined in Algorithm 1 can be implemented in different ways. In an online learning setting, an iteration would represent at least a complete sweep of the dataset, where more data can be acquired dynamically. However, due to the limited amount of data we have available we also experiment with only considering a subset of the whole dataset, reserving part of it as “future samples”. We experiment with different percentages, from refining every 5% of the dataset to 20%. Our experiments suggest that 5% is too low and results in unstable refinement. 20% allows for the policy to learn the reward for the iteration and then refine it. During policy training, we collect the prompts and the generations of the whole iteration. At the end, for each prompt we generate a new sample and we compare them using an external judge. In our case, for simplicity, we use Llama 3.1 8B Instruct as the judge. For the prompt we use the default template from TRL. ⬇ I require a leaderboard for various large language models . I ’ ll provide you with prompts given to these models and their corresponding outputs . Your task is to assess these responses , and select the model that produces the best output from a human perspective . ## Instruction {{ " instruction ": """{ prompt }""", }} ## Model Outputs Here are the unordered outputs from the models . Each output is associated with a specific model , identified by a unique model identifier . {{ {{ " model_identifier ": "0", " output ": """{ response0 }""" }}, {{ " model_identifier ": "1", " output ": """{ response1 }""" }} }} ## Task Evaluate the models on the basis of the quality and relevance of their results , and select the model that generated the best result . Reply with the identifier of the best model . Our evaluation will only take into account the first character of your answer , so make sure it contains only one of the identifiers and nothing else ( no quotation marks , no spaces , no new lines , ...). B.2 Data details HH-RLHF dataset contains 161K samples in the training split and 8k in the test split. All samples are in English. Summarize from Feedback has 93k samples in the training split and 86k in the validation. WMT has 7k samples for training and 1k for validation. For WMT we only use the en-de split, as that is the only language supported by Llama 3.2.