Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models
Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif · 2025
Read Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models by Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03460.txt.
Description
Fine-tuning a large language model the usual way, with gradients, eats memory and compute. Alessio Galatolo and co-authors study a lighter route in this 2025 paper: zeroth-order methods that judge updates by trying them, not by back-propagating errors. They first watch how a policy model and a reward model shift against each other during normal preference training.
Those patterns guide a new method, ZOPrO, which tweaks a classic sampling scheme to home in on good updates faster. On summarization, translation, and chat assistant tasks, rewards rose steadily and training speed matched gradient methods. It still trails the best systems, but it is the first zeroth-order method shown to work for preference tuning beyond simple sorting tasks.
Catalog record
| CREATOR | Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03460